feat: add log and assessment retention sweepers#8
Merged
Conversation
Bound disk growth from run-log JSONL files and whole assessments, which previously accumulated without limit and filled the shared volume. - AppConfig gains assessment_log_retention_* (log file only, default on/7d) and assessment_retention_* (whole assessment, default off/30d), backfilled by migration 011 and served by GET/PUT /api/config - PUT /api/config rejects retention day fields < 1 with HTTP 400 - shared deleteRunWithArtifacts removes the run row, cascaded results, JSONL log, and collected .ndjson files; HandleDeleteRun now uses it, fixing the collected-log leak on manual delete - RunStore.ListExpired lists non-running runs older than a cutoff - two hourly sweeper goroutines in main.go re-read AppConfig each tick - assessments page gains an Assessment retention dialog (toggles + day counts)
There was a problem hiding this comment.
Pull request overview
This PR adds configurable retention for run-log JSONL files and whole assessments (runs + results + collected .ndjson artifacts) to prevent unbounded disk growth, with backend sweepers and a frontend dialog to manage the settings.
Changes:
- Extend
AppConfig+ persistence/migrations to include assessment log retention and whole-assessment retention settings, with validation onPUT /api/config. - Add two hourly background sweepers: one for aged run-log JSONL files and one to purge expired non-running runs and their artifacts via a shared deletion helper.
- Add an assessments-page “Retention” dialog wired to
GET/PUT /api/config.
Reviewed changes
Copilot reviewed 24 out of 25 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| web/frontend/src/routes/assessments/+page.svelte | Adds “Retention” button + lazy-loads config for the dialog. |
| web/frontend/src/lib/components/RetentionDialog.svelte | New UI dialog for toggling retention settings and persisting via per-key config writes. |
| openspec/changes/assessment-retention/tasks.md | Task checklist for the feature implementation and verification. |
| openspec/changes/assessment-retention/specs/runs/spec.md | Updates run-delete requirement to include collected .ndjson cleanup. |
| openspec/changes/assessment-retention/specs/assessment-retention/spec.md | New requirements/spec for retention settings, sweepers, and UI. |
| openspec/changes/assessment-retention/proposal.md | Proposal describing motivation, changes, and impact. |
| openspec/changes/assessment-retention/design.md | Design decisions and trade-offs for retention behavior. |
| openspec/changes/assessment-retention/.openspec.yaml | Openspec metadata for the change. |
| internal/web/runlog.go | Adds SweepRunLogs to delete aged JSONL run logs by mtime. |
| internal/web/run_deletion.go | Adds shared deleteRunWithArtifacts and SweepAssessments purge logic. |
| internal/web/retention_test.go | Unit tests for both sweepers using the fake stores and temp dirs. |
| internal/web/handlers.go | Uses shared deletion helper for DELETE /api/runs/{id} and adds retention-day validation to PUT /api/config. |
| internal/web/frontend/.gitkeep | Keeps frontend directory in tree. |
| internal/web/api_runs_test.go | Adds API tests ensuring delete run removes JSONL + collected .ndjson and tolerates missing artifacts. |
| internal/web/api_config_test.go | Adds API tests for retention-day validation and persistence. |
| internal/testutil/fakes/fakes.go | Extends fake RunStore and fake ConfigStore for retention keys and ListExpired. |
| internal/testutil/fakes/fakes_test.go | Adds contract test for fake ListExpired. |
| internal/db/runs.go | Adds RunStore.ListExpired SQL query for non-running runs older than cutoff. |
| internal/db/migrations/011_assessment_retention_appconfig.up.sql | Backfills default retention keys into app_config. |
| internal/db/migrations/011_assessment_retention_appconfig.down.sql | Removes retention keys on migration rollback. |
| internal/db/config.go | Parses/serializes new retention keys in typed AppConfig. |
| internal/db/config_test.go | Adds tests for parsing defaults/round-trips including retention keys. |
| internal/config/appconfig.go | Adds retention fields + defaults to AppConfig. |
| internal/config/appconfig_test.go | Updates default config test to include retention defaults. |
| cmd/simrun/main.go | Starts hourly retention sweeper goroutine at server startup. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+124
to
+132
| sweep := func() { | ||
| cfg, err := configStore.GetAppConfig(context.Background()) | ||
| if err != nil { | ||
| log.Warnf("Retention sweep: failed to load config: %v", err) | ||
| return | ||
| } | ||
| web.SweepRunLogs(bootstrap.DataDir, cfg.AssessmentLogRetentionEnabled, cfg.AssessmentLogRetentionDays) | ||
| web.SweepAssessments(context.Background(), runStore, bootstrap.DataDir, cfg.AssessmentRetentionEnabled, cfg.AssessmentRetentionDays) | ||
| } |
| assessment_retention_days: Math.trunc(Number(assessmentDays)) | ||
| }; | ||
|
|
||
| const changed = Object.entries(next).filter(([key, value]) => value !== config[key]); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bound disk growth from run-log JSONL files and whole assessments, which previously accumulated without limit and filled the shared volume.