feat(cubeapi): add webhook event notifications

[hyy321]

Summary

This PR adds CubeAPI webhook event notifications for sandbox lifecycle events.

Implemented features:

• Configure one or more webhook endpoints through CUBE_API_WEBHOOK_ENDPOINTS
• Subscribe endpoints by event type
• Support sandbox lifecycle events:
  • sandbox.created
  • sandbox.deleted
  • sandbox.paused
  • sandbox.resumed
• Deliver JSON webhook payloads through asynchronous HTTP POST
• Use a bounded queue and background dispatcher so webhook delivery does not block sandbox lifecycle APIs
• Support optional HMAC-SHA256 signatures
• Retry retryable delivery failures with bounded exponential backoff
• Log delivery failures without exposing endpoint secrets or payload signatures
• Provide a standard-library Python webhook receiver example
• Document configuration, payload format, signature verification, retry behavior, and adapter-based alerting integration

Design Notes

Webhook delivery is implemented as a CubeAPI logging backend.

API handlers emit structured LogEvent values. The existing logger fan-out keeps FileLogger behavior and optionally adds HttpLogger when webhook endpoints are configured.

Webhook delivery is best-effort and asynchronous. Endpoint failures, timeouts, or retries do not change sandbox lifecycle API responses.

Lifecycle webhook payloads include id, timestamp, level, event, and sandbox_id. template_id is included when available.

The payload is not a complete Sandbox object and does not include tokens, secrets, environment variables, or runtime/network details.

This PR does not add a REST API for managing webhooks, persistent outbox, disk spool, dead-letter queue, batch delivery, or exactly-once delivery.

Validation

Tested on a local CubeSandbox PVM deployment.

cargo fmt --check
cargo test logging::http::tests -- --nocapture
cargo test
python3 -m py_compile examples/webhook-receiver/receiver.py
cargo build

Results:

logging::http::tests: 9 passed, 0 failed
cargo test: 79 passed, 0 failed
receiver.py py_compile: passed
cargo build: passed

End-to-end webhook verification:

Git commit:
4c7730b fix(cubeapi): silence webhook test helper warning

Received webhook events:
event: sandbox.created
event: sandbox.paused
event: sandbox.resumed
event: sandbox.deleted

API responses:
pause:  HTTP/1.1 204 No Content
resume: HTTP/1.1 201 Created
delete: HTTP/1.1 204 No Content

Build warning check:
no endpoint_matches_event warning

Runtime log check:
no file logger permission errors

The example receiver was started with WEBHOOK_SECRET=test-secret, so receiving these events also verifies the documented HMAC-SHA256 signing path.

Refs #642

[cubesandboxbot]

Test

[hyy321]

Thanks, I pushed an update for the webhook review items.

This covers the startup/config validation path, stricter endpoint URL validation, safe endpoint debug output, and stronger payload redaction. In particular, malformed CUBE_API_WEBHOOK_ENDPOINTS now fails startup, embedded URL credentials are rejected, non-public URL targets are blocked by default unless explicitly opted in, and camelCase/PascalCase sensitive fields are now redacted consistently with the existing snake_case rules.

I also updated the local receiver docs and added tests for the validation, redaction, and debug-output cases.

Verified on the server with fmt, the webhook logger tests, the full test suite, and build.

[hyy321]

Thanks for the detailed review.

I addressed the latest webhook-related comments in this round. The changes mainly cover:

• startup DNS resolution and validation for domain-name webhook endpoints, with resolved addresses pinned into the shared reqwest client;
• a practical upper bound for max_retries;
• fixed the backoff jitter test so it actually varies the attempt value;
• a sanitization depth limit for deeply nested JSON payloads;
• explicit reqwest connection settings for connect timeout and idle pool behavior;
• simplified sign_payload to return String directly;
• derived secret-material redaction for fields like db_password_hash / client_secret_b64;
• removal of the extra cloned sanitized field map during webhook payload serialization.

I also updated the README for the new DNS validation and pinned-address behavior.

Validation on the server:

• cargo fmt --check
• cargo test logging::http::tests -- --nocapture
• cargo test
• cargo build

All tests passed locally on the validation server; existing build warnings are unrelated to this webhook change.

[hyy321]

Thanks for the detailed follow-up review.

I addressed the latest webhook review comments with focused changes:

• Replaced blocking startup hostname resolution with tokio::net::lookup_host and made webhook logger initialization async.
• Changed the serialized delivery body storage from Vec<u8> to ref-counted Bytes, so retry attempts no longer clone the full payload.
• Lowered the validated maximum webhook retry count from 20 to 6 to reduce the worst-case retry window from misconfiguration.
• Added an explicit webhook User-Agent header (CubeAPI-Webhook/1.0) so outgoing requests do not rely on the HTTP client default.
• Replaced the mock server's oneshot/yield-based readiness signal with a bounded HTTP readiness probe against a dedicated GET /__ready route.

The existing webhook behavior is otherwise preserved: request body bytes, JSON payload shape, signing input, signature format, retry/backoff flow, timeout behavior, DNS pinning behavior, endpoint validation, redaction, and sanitization behavior are unchanged.

I also reviewed the suggested top-level sanitizer pre-check, but did not implement it because it would be unsafe for nested secrets. The sanitizer intentionally removes sensitive fields recursively and enforces the nesting-depth limit, so a top-level-only bypass could allow nested sensitive fields to be delivered.

Validation completed:

• cargo fmt --check
• cargo test logging::http::tests -- --nocapture
• cargo test
• cargo build
• Manual CubeSandbox lifecycle smoke test with the example webhook receiver: sandbox.created, sandbox.paused, sandbox.resumed, and sandbox.deleted were all delivered successfully.

[hyy321]

Update after the latest webhook follow-up commits:

I addressed the lower-risk webhook review follow-ups in the latest commits:

• Documented why pool_max_idle_per_host is intentionally smaller than max_concurrency. max_concurrency bounds active in-flight requests, while the idle pool cap only limits post-burst retained idle connections.
• Rejected zero initial_backoff_ms and max_backoff_ms values during webhook config validation.
• Downgraded expected delivery task cancellations during flush-timeout shutdown from error to warn, while keeping panics and other task failures at error.
• Documented that the semaphore permit is intentionally acquired per HTTP attempt and released before retry backoff sleeps.
• Documented that endpoint enabled defaults to true, and added the explicit webhook User-Agent to the README request format.
• Added an end-to-end header correctness test that captures the actual delivered request and validates Content-Type, User-Agent, X-Cube-Webhook-* headers, delivery id consistency, timestamp parsing, and the HMAC signature over the raw wire body.

Validation completed on Linux:

• cargo fmt --check
• cargo test logging::http::tests -- --nocapture
• cargo test
• cargo build

I intentionally left the maximum webhook payload body size as a separate follow-up because it introduces a new runtime behavior decision, such as whether to reject, truncate, or drop oversized webhook deliveries.

[fslongjin]

Refs #642 (this is a multi-participation issue, please use Refs instead of Closes)

[hyy321]

Update after the latest documentation commit and validation:

The latest commit mainly improves the webhook receiver example documentation. It clarifies:

• The lifecycle check requires a working local CubeSandbox deployment.
• For PR/source-tree validation against an existing deployment, the running CubeAPI process must be built from this source tree.
• If an older systemd-managed CubeAPI service is already running, it should be temporarily stopped before running the current branch CubeAPI binary.
• The receiver is only the webhook notification target; it does not run sandbox lifecycle operations.
• The expected receiver output should include:
  • sandbox.created
  • sandbox.paused
  • sandbox.resumed
  • sandbox.deleted
• Troubleshooting notes now cover local loopback receivers, missing callbacks, signature failures, and container-localhost behavior.

Validation completed on Linux:

• cargo fmt --check
• cargo test logging::http::tests -- --nocapture
• cargo test
• cargo build

Manual lifecycle validation completed with the example receiver:

• sandbox.created
• sandbox.paused
• sandbox.resumed
• sandbox.deleted

I also manually validated the non-blocking failure path. I configured the webhook endpoint to an intentionally unreachable receiver:

http://127.0.0.1:18081/webhook

The receiver was not running. The sandbox lifecycle API calls still succeeded:

• create sandbox: success
• pause sandbox: success
• resume sandbox: success
• delete sandbox: success

CubeAPI logs showed webhook delivery failures, retries, and retry exhaustion for the lifecycle events.

Conclusion: the existing webhook delivery behavior is asynchronous and non-blocking; receiver unavailability does not prevent sandbox lifecycle API requests from succeeding.