Skip to content

Add default HTTP timeouts to helper wrappers#132

Open
shuofengzhang wants to merge 3 commits intofacebookresearch:mainfrom
shuofengzhang:fix/http-helper-default-timeouts
Open

Add default HTTP timeouts to helper wrappers#132
shuofengzhang wants to merge 3 commits intofacebookresearch:mainfrom
shuofengzhang:fix/http-helper-default-timeouts

Conversation

@shuofengzhang
Copy link
Copy Markdown

What changed

  • Added a default timeout (DEFAULT_HTTP_TIMEOUT_SECONDS = 30) to all HTTP helper wrappers in matrix/utils/http.py:
    • post_url(..., timeout=...)
    • fetch_url(..., timeout=...)
    • fetch_url_sync(..., timeout=...)
  • Passed the timeout through to underlying network calls (aiohttp and requests).
  • Added unit tests in tests/unit/utils/test_http.py to verify timeout forwarding for async POST/GET and sync GET helpers.

Why

  • These helpers are used for control-plane calls (health checks, container endpoints, orchestration paths).
  • Without explicit timeouts, transient network stalls can block indefinitely, causing stuck jobs and delayed recovery.
  • A bounded default keeps behavior safe by default while preserving flexibility (callers can still override timeout explicitly).

Insight / Why this matters

  • Root cause: helper wrappers delegated directly to HTTP clients without setting timeouts, inheriting potentially unbounded wait behavior.
  • Why easy to miss: localhost and healthy-network tests usually return quickly, so hangs only appear under partial outages or degraded endpoints.
  • Impact: failures surface faster and retries/recovery loops can progress instead of hanging on a single blocked request.
  • Tradeoff: a default timeout may surface timeout errors where callers previously waited forever; this is intentional for reliability and can be tuned per call.

Practical gain / Why this matters

  • Improves reliability for users running Matrix in real distributed environments with intermittent endpoint issues.
  • Reduces “stuck” orchestration symptoms by making helper-level network behavior deterministic.
  • Provides regression protection via explicit timeout-forwarding tests.

Testing

  • pytest -q tests/unit/utils/test_http.py
    • Result: 8 passed
  • scripts/clone_and_test.sh facebookresearch/matrix
    • Result: 78 passed

Risk analysis

  • Scope is narrow (HTTP helper wrappers + tests only).
  • No behavior change for successful fast requests.
  • Rollback-safe: removing default timeout behavior is straightforward if maintainers prefer a different default.

@meta-cla
Copy link
Copy Markdown

meta-cla bot commented Mar 20, 2026

Hi @shuofengzhang!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Mar 20, 2026
@meta-cla
Copy link
Copy Markdown

meta-cla bot commented Mar 20, 2026

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@dongwang218
Copy link
Copy Markdown
Contributor

Thanks for the diff!

For context, the existing use of timeouts
Timeout in Container Client

There are two independent timeouts at play:

  1. HTTP Session Timeout (client-side)

Set in ContainerClient.init via aiohttp.ClientTimeout(total=self._timeout). This controls how long the HTTP round-trip can take. It's configured from exec_params.timeout_secs in the YAML config (e.g., 60s for SWE/tau2
containers, 1800s for LLM calls).

  1. Bash Command Timeout (server-side)

Passed as payload["timeout"] to the server, which uses it in subprocess.run(..., timeout=timeout) to kill long-running commands. Also configured from the same exec_params.timeout_secs.

Suggested Change for post_url etc.

Default the timeout to None so it inherits the session-level timeout, preserving existing behavior:

async def post_url(session, url, data=None, timeout=None):

This is safe because:

  • ContainerClient already sets a session-level timeout via aiohttp.ClientTimeout — all post_url calls through it are already covered.
  • Adding a per-request default (e.g., 30s) would silently override the session timeout and break callers that intentionally use longer values (e.g., 1800s for LLM calls).
  • Callers that don't set a session timeout (like metagen_proxy.py, deploy_sglang_app.py) can pass an explicit timeout if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants