enhancement(api): add support for dynamic API route publishing

[tobz]

Summary

This PR adds support for utilizing dynamic API routes that are published via dataspaces.

Building on #1178, this PR exposes new primitives to build API servers whose routes come solely through dynamically published API routes using dataspaces, allowing processes to expose API routes without needing to do so when the application first starts up. This allows for decoupling of the API server definitions and the things which wish to expose API routes, as well as allowing for graceful removal of API routes when the underlying component is unhealthy/unavailable.

We've introduced a number of new building blocks to support this functionality:

• DynamicAPIBuilder: a spiritual equivalent to APIBuilder that supports dynamic API routes and can be spawned directly as a Supervisable process
• DynamicRoute: the concrete type that holds the declared routes of a given component that is shared via the dataspace
• EndpointType/EndpointProtocol: helper types to define which API endpoint (privileged or unprivileged) and which protocol (HTTP or gRPC) a set of dynamic API routes should be attached to

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

Added new unit tests that ensure DynamicAPIBuilder responds to route assertions/retractions in the expected fashion.

References

AGTMETRICS-400

[tobz]

• chore(agent-data-plane): register health API dynamically #1259
• enhancement(api): add support for dynamic API route publishing #1179 👈 (View in Graphite)
• enhancement(core): add dataspaces for decoupled, cross-process state #1178
• chore(agent-data-plane): initialize TLS early on before spawning supervisor #1177
• chore(agent-data-plane): update health registry worker to allow being restarted #1176
• chore(agent-data-plane): move internal observability/control plane to supervisor scheme #1148
• enhancement(core): make initialization async in Supervisable #1146
• enhancement(core): add support for spawning supervisors on a dedicated runtime #1145
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Target: 5316221 (baseline) vs 5d48b70 (comparison) diff
Analysis Type: Stripped binaries (debug symbols excluded)
Baseline Size: 26.48 MiB
Comparison Size: 26.48 MiB
Size Change: +2.86 KiB (+0.01%)
Pass/Fail Threshold: +5%
Result: PASSED ✅

Changes by Module

Module	File Size	Symbols
hashbrown	-4.18 KiB	381
matchit	+3.89 KiB	39
core	-2.78 KiB	9098
[Unmapped]	+2.26 KiB	1
alloc	+2.22 KiB	803
axum	+1.07 KiB	224
anyhow	-720 B	1299
agent_data_plane::cli::run	+521 B	82
saluki_config::ConfigurationError	-384 B	1
saluki_config::ConfigurationLoader::from_environment	+360 B	1
http_body_util	+347 B	110
saluki_app::metrics::RuntimeMetrics	+331 B	2
agent_data_plane::internal::remote_agent	-251 B	66
futures_util	+208 B	64
saluki_components::sources::otlp	-200 B	164
agent_data_plane::internal::control_plane	+185 B	32
saluki_env::helpers::remote_agent	+160 B	24
[sections]	+149 B	7
rustls	+140 B	32
agent_data_plane::internal::create_internal_supervisor	+120 B	2

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +1.79Mi  [NEW] +1.79Mi    std::thread::local::LocalKey<T>::with::h05bb577906fe3286
  [NEW]  +120Ki  [NEW]  +119Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hd5ed6b4fd8f6c3e5
  [NEW] +64.2Ki  [NEW] +64.0Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::hd59f138ff82adaad
  [NEW] +62.1Ki  [NEW] +62.0Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h747f4ac70f1f985e
  [NEW] +58.8Ki  [NEW] +58.6Ki    _<agent_data_plane::internal::control_plane::PrivilegedApiWorker as saluki_core::runtime::supervisor::Supervisable>::initialize::_{{closure}}::h384ce618e08aac60
  [NEW] +49.5Ki  [NEW] +49.4Ki    saluki_app::bootstrap::AppBootstrapper::bootstrap::_{{closure}}::hda2df0f2beb7408a
  [NEW] +46.4Ki  [NEW] +46.3Ki    h2::proto::connection::Connection<T,P,B>::poll::h8dfb03a9b03245ec
  [NEW] +46.2Ki  [NEW] +46.0Ki    _<saluki_components::destinations::prometheus::Prometheus as saluki_core::components::destinations::Destination>::run::_{{closure}}::he1b5ae9a1cde9659
  [NEW] +46.2Ki  [NEW] +46.0Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h87d5f0a209a35da2
  [NEW] +44.1Ki  [NEW] +43.9Ki    saluki_env::workload::providers::remote_agent::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::hb1b3c74b8bdba0d5
  +0.0% +2.93Ki  +0.0% +1.81Ki    [22696 Others]
  [DEL] -44.1Ki  [DEL] -43.9Ki    saluki_env::workload::providers::remote_agent::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::hfc80d52378eff90b
  [DEL] -46.2Ki  [DEL] -46.0Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h9e6a07604104748f
  [DEL] -46.2Ki  [DEL] -46.0Ki    _<saluki_components::destinations::prometheus::Prometheus as saluki_core::components::destinations::Destination>::run::_{{closure}}::h804b0075faebe88b
  [DEL] -46.4Ki  [DEL] -46.3Ki    h2::proto::connection::Connection<T,P,B>::poll::hded4fb5f8a002638
  [DEL] -49.6Ki  [DEL] -49.4Ki    saluki_app::bootstrap::AppBootstrapper::bootstrap::_{{closure}}::hd83d25f4c7e8a1ae
  [DEL] -58.7Ki  [DEL] -58.4Ki    _<agent_data_plane::internal::control_plane::PrivilegedApiWorker as saluki_core::runtime::supervisor::Supervisable>::initialize::_{{closure}}::hf1e132bb3e8bfeaf
  [DEL] -61.9Ki  [DEL] -61.8Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hbff4cb15be99fca3
  [DEL] -64.2Ki  [DEL] -64.1Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h7847418cb5d4cb46
  [DEL]  -119Ki  [DEL]  -119Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h5c132aa500e2a433
  [DEL] -1.79Mi  [DEL] -1.79Mi    std::thread::local::LocalKey<T>::with::hddbf5b47575f6048
  +0.0% +2.86Ki  +0.0% +1.74Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

This comment was omitted because it was over 65,536 characters.Please check the Gitlab Job logs to see its output.