feat(gateway): implement wake-on-traffic for paused sandboxes

[furykerry]

Ⅰ. Describe what this PR does

This PR implements wake-on-traffic for paused sandboxes in the sandbox-gateway. When a paused sandbox receives incoming traffic, the gateway automatically resumes it and forwards the request once the sandbox is running again.

Key changes:

1. Wake-on-traffic in filter (pkg/sandbox-gateway/filter/filter.go): When a paused sandbox with WakeOnTraffic=true receives traffic, the filter launches wake in a goroutine with a detached context and returns api.Running to suspend Envoy request processing. The wakeAndContinue goroutine calls Continue(api.Continue) on success or SendLocalReply(503) on failure. Thread-safe completion state machine (mutex-protected) prevents double-reply when async goroutine and Destroy race. Wake eligibility is encapsulated in shouldWakeSandbox() — a single method that checks all preconditions (Paused state, EnableWakeOnTraffic, waker initialized, valid namespace--name format) before deciding via route.WakeOnTraffic or the HasWakeAnnotation fallback. Scoped loggers (logger.With(zap.String("sandboxID", ...))) are used in DecodeHeaders and wakeAndContinue to avoid repeating the field at each call site.

2. New wake package (pkg/sandbox-gateway/wake/wake.go): Implements Waker that calls sandbox.Resume() to patch the spec and waits for the sandbox to reach Running state via the cache wait reconciler. Wake() runs under the caller's context (with timeout) — Resume itself provides first-writer-wins dedup via retryUpdate, so no singleflight is needed. Uses errors.Is() with sentinel errors for error classification (not string matching).

3. E2B pause with spec patch (pkg/servers/e2b/pause_resume.go): Adds Pause() that sets Spec.DesiredState=Paused (instead of deleting the sandbox), enabling the wake-on-traffic flow.

4. Registry route lifecycle (pkg/sandbox-gateway/server/server.go): handleRefresh retains routes for all states except dead and empty string. This ensures the filter has full visibility (including paused, creating, available) for routing and wake decisions.

5. Sentinel error classification (pkg/utils/utils.go, pkg/sandbox-manager/errors/error.go): Exports sentinel errors (ErrSandboxIsPausing, ErrSandboxIsTerminating, ErrShutdownTimeReached, ErrSandboxPhaseNotAllowed) from IsSandboxResumable. Adds Unwrap() + NewErrorWrap() to managererrors.Error to enable errors.Is() through wrapped errors.

6. Gateway cache optimization (pkg/cache/cache.go): Restricts the gateway's informer cache to only Sandbox resources via ByObject filtering, reducing memory usage and API server load.

7. Configuration: Adds ENABLE_WAKE_ON_TRAFFIC env var and wakeTimeoutSeconds config to the gateway filter.

8. autoResume API support (pkg/servers/e2b/models/sandbox.go, pkg/servers/e2b/create.go): When the E2B create sandbox API is called with "autoResume": {"enabled": true}, the sandbox is automatically annotated with agents.kruise.io/wake-on-traffic=true at creation time. This aligns with the upstream E2B API specification. The E2B SDK maps lifecycle={"auto_resume": True} to this API field. Unit tests cover enabled, disabled, and absent scenarios.

9. E2E test update (test/e2b/test_wake_on_traffic.py): Updated to use lifecycle={"on_timeout": "pause", "auto_resume": True} instead of manual kubectl annotate for the wake-on-traffic annotation. The wake-timeout-seconds annotation is still set via kubectl since the API only handles wake-on-traffic.

Ⅱ. Does this pull request fix one issue?

NONE

Ⅲ. Describe how to verify it

1. Run unit tests:

   go test ./pkg/sandbox-gateway/... ./pkg/cache/... ./pkg/servers/e2b/... ./pkg/utils/...

2. Run E2E test:

   pytest test/e2b/test_wake_on_traffic.py

3. Manual verification:
   • Create a sandbox with lifecycle={"on_timeout": "pause", "auto_resume": True} (SDK) or "autoResume": {"enabled": true} (raw API)
   • Verify the sandbox is annotated with agents.kruise.io/wake-on-traffic=true
   • Pause it via POST /sandboxes/{id}/pause
   • Send traffic to the paused sandbox
   • Verify the sandbox is automatically resumed and the request is forwarded

Ⅳ. Special notes for reviews

• Async wake: DecodeHeaders returns api.Running when wake is triggered, launching wakeAndContinue in a goroutine. The goroutine uses a detached context (not tied to the Envoy filter lifecycle). On completion, it calls Continue(api.Continue) or SendLocalReply. The Destroy() method cancels the in-flight wake context when Envoy destroys the filter (stream reset). A mutex-protected completion state machine prevents double-reply races.
• No singleflight: Waker.Wake() calls wakeInternal directly under the caller's context. Dedup is handled by Resume's retryUpdate (first-writer-wins) and refcounted wait via NewSandboxResumeTask. This avoids the footgun where a canceled caller context would cut short the shared Resume for all coalesced callers.
• handleRefresh: Only dead and empty-state routes are deleted from the registry. All other states (running, paused, creating, available) are retained for full filter visibility.
• Annotation lifecycle: Wake-on-traffic annotations can be set in two ways: (1) via the E2B create API with "autoResume": {"enabled": true} (mapped from SDK's lifecycle={"auto_resume": True}), or (2) manually via kubectl annotate for sandboxes created without autoResume. The E2E test uses the API path and verifies the annotation is present immediately after creation.
• The cache optimization (SandboxOnly mode) adds nil guards on GetSandboxController() and GetSandboxSetController() since controller handlers are not set up in that mode.

Ⅴ. Test coverage

Package	Coverage
pkg/sandbox-gateway/filter	82.8%
pkg/sandbox-gateway/wake	92.1%
pkg/servers/e2b	87.7%
pkg/proxy	90.2%

[kruise-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from furykerry by writing /assign @furykerry in a comment. For more information see:The Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

❌ Patch coverage is 70.04049% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.92%. Comparing base (dee78fb) to head (be385e8).
⚠️ Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/sandbox-gateway/filter/filter.go	66.16%	33 Missing and 12 partials ⚠️
pkg/sandbox-gateway/server/server.go	37.50%	10 Missing ⚠️
pkg/sandbox-gateway/wake/wake.go	88.63%	3 Missing and 2 partials ⚠️
...g/sandbox-gateway/controller/gateway_controller.go	0.00%	4 Missing ⚠️
pkg/cache/cache.go	40.00%	2 Missing and 1 partial ⚠️
pkg/cache/index.go	25.00%	2 Missing and 1 partial ⚠️
pkg/cache/cachetest/cachetest.go	0.00%	1 Missing ⚠️
...controller/sandboxclaim/sandboxclaim_controller.go	0.00%	1 Missing ⚠️
pkg/sandbox-manager/core.go	0.00%	1 Missing ⚠️
pkg/servers/e2b/routes.go	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #586      +/-   ##
==========================================
- Coverage   80.06%   79.92%   -0.15%     
==========================================
  Files         229      230       +1     
  Lines       17861    18063     +202     
==========================================
+ Hits        14300    14436     +136     
- Misses       2976     3027      +51     
- Partials      585      600      +15     

Flag	Coverage Δ
unittests	79.92% <70.04%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[AiRanthem]

Review notes on the wake-on-traffic changes (local go build ./..., go test for pkg/sandbox-gateway/... + pkg/utils/proxyutils/..., and -race on wake/filter all pass). Four inline notes below: one under-implemented gap, one simplification (drop singleflight), one log nit, and one RBAC least-privilege nit.

[AiRanthem]

Requesting changes. Verified against the current head by tracing the concrete code paths (not coverage inference). There are 3 blocking issues plus 3 non-blocking notes, all left as inline comments:

• Blocking 1: access-token auth is bypassed on the paused wake path.
• Blocking 2: Destroy() is not Envoy's OnDestroy, so the cancel-on-destroy never fires.
• Blocking 3: peer refresh deletes Paused routes, breaking wake-on-traffic in multi-replica topologies.

Suggested review order: filter.go (auth order, paused wake branch, destroy lifecycle) -> server.go (handleRefresh, globalPeerManager) -> api.go (Pause/Resume/syncRoute) -> wake.go (deadline, timeout annotation) -> create.go + test_wake_on_traffic.py.

[AiRanthem]

Blocking: Paused routes get deleted by peer refresh, breaking wake-on-traffic.

Wiring the wake path to peer sync via globalPeerManager surfaces a problem in handleRefresh (lines 182-192, not shown in this diff): it Updates only Running routes and Deletes every other state, including Paused.

The manager pushes Paused routes on pause (PauseSandbox -> syncRoute -> SyncRouteWithPeers, pkg/sandbox-manager/api.go:326,300-302), and manager + gateway share one memberlist mesh (same agents.kruise.io/peer=true selector + namespace; GetPeers() returns all alive members regardless of the sm- / sg- prefix). So a Paused refresh reaches the gateway and deletes the route. The local controller wrote all states (gateway_controller.go:63-65), but nothing re-adds it until the Sandbox changes again. Traffic then hits sandbox not found at the initial registry lookup and wake is never attempted.

The delete-non-Running behavior is pre-existing (this PR only adds globalPeerManager here), but wake-on-traffic now depends on Paused routes surviving peer sync, so it must be addressed. Suggested fix: in handleRefresh, Update (with resourceVersion guard) for Running / Paused / Creating / Available; Delete only for Dead or an explicit deletion marker; decide the empty-state case explicitly instead of a blanket non-Running delete.

[AiRanthem]

Non-blocking: success oracle is loose.

The loop polls until any non-502/503 response, and these final assertions only exclude 502/503, so 200/401/404/500 would pass the HTTP check. The test does additionally assert the sandbox reaches Running and the annotations persist (lines ~200, 209-213), which partially covers correctness, but the HTTP oracle should assert an explicit expected 2xx / known app response, and ideally verify that the first suspended request is actually forwarded rather than relying on later polling requests to mask a failure.

Superseded by inline comments below for readability.