Skip to content

feat: add BOBS write routing and stress streaming#191

Merged
jameshawkes merged 1 commit into
upstreamfrom
majh/bobs-routing-stress-worker
May 13, 2026
Merged

feat: add BOBS write routing and stress streaming#191
jameshawkes merged 1 commit into
upstreamfrom
majh/bobs-routing-stress-worker

Conversation

@jameshawkes
Copy link
Copy Markdown
Contributor

@jameshawkes jameshawkes commented May 13, 2026

Summary

  • update frontend to current BITS request-id/overload API and return retryable 529 responses on broker overload
  • update worker BOBS delivery to use write_url returned by /create
  • split BOBS create/body clients so /create gets fresh kube load-balancing while write/complete reuse HTTP/2 connections
  • use h2c HTTP/2 prior knowledge for worker-to-BOBS body delivery
  • return BOBS read_url unchanged, so public result URLs remain opaque /download-N/{key} links
  • pin the integration-test BOBS dependency to the merged BOBS commit that includes write routing and opaque public URLs
  • add request-driven test-worker stress behaviour with lazy chunked output
  • update worker/integration tests for routed BOBS writes

Depends on merged BOBS support in ecmwf/bobs#2 and ecmwf/bobs#3, plus chart routing in ecmwf/polytope-chart#40.

Verification

  • cargo fmt --all -- --check
  • cargo test -p polytope-worker-common bobs -- --nocapture
  • cargo test -p polytope-server-integration-tests -- --nocapture → 27 passed
  • cargo test -p polytope-server-integration-tests bobs_delivery_pipeline -- --nocapture
  • rust-analyzer diagnostics for tests/integration/src/lib.rs: none
  • built and pushed dev images during live validation: test-worker:majh-dev, mars-worker:majh-dev, fdb-worker:majh-dev, polytope-fe-worker:majh-dev
  • live tier-2 stress suite passed against bologna-dev
  • live BOBS full-range benchmark passed against the earlier /api/v1 public URL shape: 256/256 full-range downloads, 4 GiB produced/read, 211 MiB/s

@jameshawkes jameshawkes force-pushed the majh/bobs-routing-stress-worker branch 2 times, most recently from fd81c65 to cd3351b Compare May 13, 2026 20:52
@jameshawkes jameshawkes force-pushed the majh/bobs-routing-stress-worker branch from cd3351b to 3d7123b Compare May 13, 2026 21:01
@jameshawkes jameshawkes merged commit 07e515d into upstream May 13, 2026
4 checks passed
@jameshawkes jameshawkes deleted the majh/bobs-routing-stress-worker branch May 13, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant