Skip to content

AI-1166: xfail AI-service integtests on CI timeouts - #533

Closed
Matovidlo wants to merge 1 commit into
mainfrom
AI-1166-xfail-ai-service-integtests
Closed

AI-1166: xfail AI-service integtests on CI timeouts#533
Matovidlo wants to merge 1 commit into
mainfrom
AI-1166-xfail-ai-service-integtests

Conversation

@Matovidlo

Copy link
Copy Markdown
Contributor

Description

Linear: AI-1166

Change Type

  • Major (breaking changes, significant new features)
  • Minor (new features, enhancements, backward compatible)
  • Patch (bug fixes, small improvements, no new features)

Summary

Adds @pytest.mark.xfail(strict=False, reason='AI service may exceed read timeout in CI') to the two integtests that hit external LLM endpoints:

  • integtests/tools/test_search.py::test_find_component_idsuggest/component
  • integtests/tools/test_doc.py::test_docs_querydocs/question

Both endpoints regularly exceed the 60s read budget configured in clients/base.py, surfacing as httpx.ReadTimeout. The retry transport (httpx_retries.RetryTransport) does not retry POSTs on network exceptions by default, so a single slow upstream produces a single hard failure and blocks every unrelated PR.

xfail(strict=False) lets the suite pass when the AI service is slow while still validating correctness on healthy runs — and an unexpected pass does not flip the suite to failed.

This re-applies the exact one-line markers from commit 39233cd6 ("AI-1166: mark AI-service integration tests xfail to tolerate CI timeouts", 2026-05-12), which lived only on the still-open AI-1166-variables-support branch (PR #498) and so never reached main. Pulling the markers out into their own PR keeps that larger feature branch unrelated to a generic CI-stability fix.

Version bumped 1.61.0 → 1.61.1 (patch).

Testing

  • Tested with Cursor AI desktop (Streamable-HTTP transports)

Optional testing

  • Tested with Cursor AI desktop (all transports)
  • Tested with claude.ai web and canary-orion MCP (Streamable-HTTP)
  • Tested with In Platform Agent on canary-orion
  • Tested with RO chat on canary-orion

Behavioural test surface is the integtests themselves: when the AI service responds in time the tests still assert all their original invariants (component ID match, docs answer present, source URLs non-empty, TOON round-trip); when it times out, xfail(strict=False) records XFAIL instead of FAIL. Local tox checks pass:

  • `tox -e black` — OK
  • `tox -e isort` — OK
  • `tox -e flake8` — OK
  • `tox -e check-tools-docs` — OK (TOOLS.md unchanged)

Unit tests (tox -e python) untouched — changes are scoped to integtests/.

Checklist

  • Self-review completed
  • Unit tests added/updated (if applicable) — n/a, change is test infrastructure
  • Integration tests added/updated (if applicable) — markers added to existing tests
  • Project version bumped according to the change type
  • Documentation updated (if applicable) — n/a

🤖 Generated with Claude Code

@linear

linear Bot commented May 18, 2026

Copy link
Copy Markdown

AI-1166

@Matovidlo

Copy link
Copy Markdown
Contributor Author

@claude review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to improve CI stability by tolerating intermittent AI-service read timeouts in integration tests, plus a patch version bump.

Changes:

  • Adds non-strict xfail markers to two AI-service integration tests.
  • Bumps package version from 1.61.0 to 1.61.1.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File Description
integtests/tools/test_search.py Marks test_find_component_id as expected-failable.
integtests/tools/test_doc.py Marks test_docs_query as expected-failable.
pyproject.toml Updates project version.
uv.lock Updates locked editable package version.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread integtests/tools/test_search.py Outdated
Comment thread integtests/tools/test_doc.py Outdated
@Matovidlo
Matovidlo force-pushed the AI-1166-xfail-ai-service-integtests branch from d058b57 to 935a202 Compare May 18, 2026 12:46
`test_find_component_id` and `test_docs_query` hit external LLM endpoints
(`suggest/component`, `docs/question`) that can exceed the 60s `read`
budget under CI load, surfacing as `httpx.ReadTimeout` from a single POST
attempt — the retry transport in `clients/base.py` does not retry POSTs
on network exceptions. `@pytest.mark.xfail(strict=False)` lets the suite
pass when the service is slow while still validating correctness on
healthy runs.

Re-applies the same one-line markers from commit 39233cd, which lived
only on a feature branch that never merged.

Bumps version to 1.61.1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Matovidlo
Matovidlo force-pushed the AI-1166-xfail-ai-service-integtests branch from 935a202 to 6867307 Compare May 18, 2026 12:52
@Matovidlo
Matovidlo requested a review from Copilot May 18, 2026 12:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

Comment thread pyproject.toml
Comment thread integtests/tools/test_search.py
@Matovidlo Matovidlo closed this Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants