feat: add exploit-intelligence job polling, configuration & API endpoints by Strum355 · Pull Request #2464 · guacsec/trustify

Strum355 · 2026-07-08T16:17:18Z

Hello early reader :^) draft PR until I fill out a description

sourcery-ai

Sorry @Strum355, your pull request is larger than the review limit of 150000 diff characters

ctron · 2026-07-09T08:50:37Z

 | `UI_SCOPE`                               | Scopes to request                                                                   | `openid`                                |
+| `EXPLOIT_INTELLIGENCE_URL`               | Base URL of the Exploit Intelligence client service                                 |                                         |
+| `EXPLOIT_INTELLIGENCE_UI_URL`            | Base URL of the EI web UI for deep-linking to reports                               |                                         |
+| `EXPLOIT_INTELLIGENCE_POLL_INTERVAL_SECS`| Polling interval in seconds for EI analysis completion                              | `30`                                    |


Please don't use seconds, but humantime format, like with the others.

ctron · 2026-07-09T08:50:45Z

+| `EXPLOIT_INTELLIGENCE_URL`               | Base URL of the Exploit Intelligence client service                                 |                                         |
+| `EXPLOIT_INTELLIGENCE_UI_URL`            | Base URL of the EI web UI for deep-linking to reports                               |                                         |
+| `EXPLOIT_INTELLIGENCE_POLL_INTERVAL_SECS`| Polling interval in seconds for EI analysis completion                              | `30`                                    |
+| `EXPLOIT_INTELLIGENCE_MAX_POLL_DURATION_SECS`| Maximum duration in seconds before EI polling is considered timed out           | `1800`                                  |


ctron · 2026-07-09T08:53:11Z

Please consider that this code can run multiple times on the same database, in multiple instances.

ruromero

The functionality is there and should work as expected but all the logic ended up in the endpoints module which is an anti-pattern. Look how other functionalities are implemented. I wonder why Claude didn't check for these patterns.

Another concerning point is the fact that there are many hardcoded values that could be configurable for operational reasons. I know there should be a balance between what should be configurable vs hardcoded. It's just something to take into account.

Pay attention at the long functions (+200 lines) doing multiple steps. This goes against the defined conventions.

Finally, the whole retry/polling logic can be moved to a common module so other parts of the code could benefit from it. I haven't checked if there are other candidates for future refactoring.

ruromero · 2026-07-09T08:25:07Z

+}
+
+/// Maximum number of consecutive transient poll failures before we give up.
+const MAX_CONSECUTIVE_TRANSIENT_POLL_FAILURES: u32 = 5;


In my experience is better to have configurable values for values like this.

ruromero · 2026-07-09T08:25:31Z

+        // SPDX multi-component flow: upload to /api/v1/products/upload-spdx
+        let upload_url = format!("{}/api/v1/products/upload-spdx", config.url);
+
+        const UPLOAD_MAX_ATTEMPTS: u32 = 3;


I think this should be configurable as well

ruromero · 2026-07-09T08:37:53Z

+///
+/// Updates both the component record (with finding/advisory) and the parent
+/// job record (status only) upon completion.
+pub(crate) async fn poll_for_result(


Adding a distinction between RetryableFailure vs PermanentFailure would help end users distinguish between something that can be reattempted or something that requires an action before retry.

ruromero · 2026-07-09T08:38:58Z

+/// This is the SPDX counterpart of `poll_for_result`. It polls the product-level
+/// endpoint until all components are complete, then fetches each component's
+/// report individually.
+pub(crate) async fn poll_for_product_result(


I see a lot of duplication between this method and poll_for_result. I sure this can be refactored

ruromero · 2026-07-09T08:41:49Z

+    /// exchange requests.
+    pub fn oidc(token_url: String, client_id: String, client_secret: String) -> Self {
+        let client = reqwest::Client::builder()
+            .connect_timeout(Duration::from_secs(10))


I'm in favour of configuring this kind of values with reasonable defaults

ruromero · 2026-07-09T08:42:14Z

+    pub fn oidc(token_url: String, client_id: String, client_secret: String) -> Self {
+        let client = reqwest::Client::builder()
+            .connect_timeout(Duration::from_secs(10))
+            .timeout(Duration::from_secs(30))


Same for configurable values with default instead of magic numbers

ruromero · 2026-07-09T09:29:14Z

+}
+
+/// Background task: submit SBOM to EI client, poll for results, ingest VEX.
+async fn run_analysis(


Business logic should not live in the endpoints layer.
The endpoints layer should only hold simple HTTP adapters. The CRUD operations should be in the services layer.
The background processing like run_analysis, poll_for_result, poll_for_result_product and recover_and_resume belongs to the runner module.

Add a new exploit_intelligence module under modules/fundamental that provides API endpoints for triggering Exploit Intelligence analysis, listing jobs, and retrieving job details. The module includes: - POST /v3/exploit-intelligence/analyze: Creates an analysis job and spawns a background task that submits the SBOM to the EI client service, polls for completion, and ingests returned VEX documents with source: exploit-intelligence labels. - GET /v3/exploit-intelligence/jobs: Paginated list of analysis jobs with filtering by sbom_id and status. - GET /v3/exploit-intelligence/jobs/{id}: Job details including status, finding, report_url, advisory_id, and error details. Configuration is via environment variables: EXPLOIT_INTELLIGENCE_URL, EXPLOIT_INTELLIGENCE_POLL_INTERVAL_SECS, EXPLOIT_INTELLIGENCE_AUTH_TOKEN. Implements TC-4677 Assisted-by: Claude Code

…d integration tests - Add max_poll_duration_secs field to ExploitIntelligenceConfig (default 1800s) with EXPLOIT_INTELLIGENCE_MAX_POLL_DURATION_SECS env var - Add polling timeout logic in run_analysis: calculates max iterations from config and breaks with Failed status when exceeded - Add find_active_job method for deduplication: queries for Pending/Running jobs matching the same sbom_id + vulnerability_id - In analyze endpoint, check for active jobs before creating duplicates - Add three integration tests: deduplication, timeout failure, finding-only flow - Document new env var in docs/env-vars.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The entity layer already has a proper Finding enum (Vulnerable, NotVulnerable, Uncertain), but the API DTOs were converting it to Option<String>. This exposes a JobFinding enum in the API model, matching the pattern used by JobStatus. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…typed structs The EI response was parsed via serde_json::Value with hardcoded JSON Pointer paths, making the code silently break on any upstream schema change. Introduces typed Rust structs (UploadResponse, PollResponse, etc.) so contract mismatches surface as deserialization errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…nnection pooling reqwest::Client was created bare with no timeouts or pooling config, and a new instance was constructed per background task. Now a shared client is built once with connect_timeout(10s), request timeout(60s), pool_max_idle_per_host(4), and pool_idle_timeout(90s). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…n refresh Replaces the static bearer token with a TokenProvider abstraction supporting both static tokens (backward compatible via EXPLOIT_INTELLIGENCE_AUTH_TOKEN) and OIDC client credentials flow with automatic token refresh before expiry. New env vars: EXPLOIT_INTELLIGENCE_OIDC_TOKEN_URL, EXPLOIT_INTELLIGENCE_OIDC_CLIENT_ID, EXPLOIT_INTELLIGENCE_OIDC_CLIENT_SECRET. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Upload requests retry up to 3 times with exponential backoff (1s, 2s, 4s) on transient HTTP errors (429, 502, 503, 504) or network failures. Poll loop classifies errors as transient (retryable) or permanent (immediate failure). Up to 5 consecutive transient poll failures are tolerated before the job is marked failed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

After a server restart, EI jobs that were Pending or Running become orphaned because their background tasks no longer exist. - Pending jobs (no scan_id) are marked Failed with a descriptive message telling the user to re-trigger the analysis. - Running jobs (scan_id present) get their poll loops re-spawned so the EI backend result is picked up transparently. Recovery runs inside the EI endpoints `configure()` where the actix runtime is available, using `actix_web::rt::spawn` (not `tokio::spawn`) because IngestorService contains non-Send types. The poll loop is extracted into a standalone `poll_for_result()` function shared by `run_analysis` (normal path) and `recover_and_resume` (restart recovery path). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…p-links The report_url was incorrectly built from the API base URL (config.url), which points to the agent-morpheus-client backend. The EI web UI may be hosted at a different URL. Add EXPLOIT_INTELLIGENCE_UI_URL as a separate optional config — when set, completed jobs include clickable deep-links to the EI web UI; when unset, report_url is null. Also fix pre-existing test compilation issues (missing entity fields, wrong assertion method on JobFinding enum). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Regenerate OpenAPI spec with missing JobFinding, ComponentResult schemas and fields - Fix TOCTOU race in analyze endpoint by using single RW transaction for dedup check-and-insert - Add test authorizer to all read endpoint tests (get_job, list_jobs) - Fix advisory_id parsing by stripping urn:uuid: prefix before Uuid::from_str - Return 200 OK for deduplicated jobs instead of 201 Created - Enforce minimum 5s poll interval via clap validation and defense-in-depth .max(5) - Replace magic integer literals with named Finding enum constants in aggregate query - Make total count conditional on paginated.total to skip unnecessary count queries - Make component initialization idempotent for orphaned jobs by checking existing records - Use read-only transaction for SBOM retrieval in run_analysis - URL-encode path segments in all EI client API and report URL construction - Add tests: happy-path analyze, running status, status filter, 404 SBOM not found, 503 EI unconfigured - Document EXPLOIT_INTELLIGENCE_UI_URL environment variable - Warn when OIDC args are partially configured instead of silently falling back - Increase default OIDC expires_in from 300s to 3600s - Add err.is_request() to transient network error detection for DNS failures - Log warning when SBOM format label is missing and defaulting to CycloneDX - Add doc comments to all ComponentResult fields - Introduce EiRuntime wrapper struct for config/client coupling Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… fix SubmissionFailure deserialization Rename component_purl → component_ref in the API model and service layer to reflect that the field stores both purls and container image references. Fix SubmissionFailure.purl → .image to match what morpheus-client actually sends, resolving silent deserialization failures that stored "unknown" for failed component references. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The EI service returns finding status as uppercase ("TRUE"/"FALSE") but the mapping matched lowercase only, causing all findings to fall through to "uncertain". Use to_ascii_lowercase() before matching. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rework the EI orchestration layer for the unified schema where all per-analysis results (scan_id, finding, advisory_id) live on the component table — even CycloneDX single-component jobs now create exactly one component row. Key changes: - CycloneDX flow creates a component record before upload, stores scan_id on the component via update_component_running - Delete update_job_running (scan_id no longer on job) - Simplify update_job_completed to status-only (no finding/advisory) - Recovery loads components from DB instead of reading job.scan_id - SPDX flow uses GET /reports?productId= to discover component scan IDs and findings, since the product endpoint returns empty components - poll_for_result takes component_id and updates component on completion - report_url computed at API response time from ui_url + identifiers - Aggregate finding computed from component data in model layer - Remove component_name (dropped from entity) - Remove advisory_id from job details (lives on components) - Add ProductReportEntry/ReportVuln structs for reports-list endpoint - Fix ProductData deserialization (alias "id", default components) Depends on TC-4675 entity/migration changes (will not compile alone). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace custom common::retry module with the backon crate, which is already a transitive dependency via walker-common. Upload retries use backon's ExponentialBuilder directly, and polling uses ConstantBuilder with a Cell-based consecutive failure counter in the .when() predicate. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

github-project-automation Bot added this to Trustify Jul 8, 2026

sourcery-ai Bot reviewed Jul 8, 2026

View reviewed changes

Strum355 changed the title ~~Tc 4677~~ feat: add exploit-intelligence job polling, configuration & API endpoints Jul 8, 2026

ctron reviewed Jul 9, 2026

View reviewed changes

ruromero requested changes Jul 9, 2026

View reviewed changes

Strum355 and others added 14 commits July 9, 2026 15:47

feat(exploit-intelligence): add multi-component SPDX product flow

a86e530

Strum355 force-pushed the TC-4677 branch from ce90a1c to 3b028cb Compare July 9, 2026 14:48

ptomanRH requested a review from jcrossley3 July 9, 2026 14:49

jcrossley3 reviewed Jul 9, 2026

View reviewed changes

Comment thread common/src/retry.rs Outdated

Strum355 force-pushed the TC-4677 branch from 3b028cb to caabea1 Compare July 10, 2026 14:06

Uh oh!

Conversation

Strum355 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sourcery-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ctron commented Jul 9, 2026

Uh oh!

ruromero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Strum355 commented Jul 8, 2026 •

edited

Loading