Skip to content

fix(eval): cancel queued provider-eval jobs when a standalone evaluation is deleted#247

Open
tryeverything24 wants to merge 1 commit into
mini-router:mainfrom
tryeverything24:fix-cancel-deleted-eval-job
Open

fix(eval): cancel queued provider-eval jobs when a standalone evaluation is deleted#247
tryeverything24 wants to merge 1 commit into
mini-router:mainfrom
tryeverything24:fix-cancel-deleted-eval-job

Conversation

@tryeverything24

Copy link
Copy Markdown

What

Deleting a standalone provider evaluation from the admin UI now also cancels its
still-queued job, so a "deleted" eval can no longer be picked up and executed by a
worker.

Why

DELETE /api/admin/evaluations/{id} soft-deletes a standalone provider evaluation by
setting deleted_at, but it left the evaluation's provider_eval row in job_queues
untouched. The worker's poll query (worker.process_once) only filters on
JobQueue.status == "queued" and never looks at EvaluationRun.deleted_at, so a
provider eval that was deleted before it ran would still be claimed, executed via
evaluate_provider_route(...), and charged real provider spend — the exact outcome an
operator is trying to avoid by deleting it. The run then completes while remaining
hidden from /api/provider-benchmarks (which filters on deleted_at), so the wasted
cost is also invisible.

The submission delete path already guards against this: DELETE /api/admin/submissions/{id}
calls cancel_submission_jobs(...). Standalone-evaluation delete was simply missing the
equivalent step.

Change

  • Add cancel_evaluation_jobs(session, evaluation_id, reason=...) in
    services/queue.py, mirroring cancel_submission_jobs: it flips any queued
    provider_eval job for that evaluation id to cancelled (a running job is left
    alone, matching the submission path).
  • Call it from admin_delete_evaluation right after setting deleted_at, before the
    commit, so the soft-delete and the cancellation land in one transaction.

Revoke behaviour is unchanged; only the queued job is affected, and only on delete.

Tests

Adds validator/tests/test_delete_evaluation.py:

  • deleting a standalone eval cancels its queued provider_eval job (RED before the fix:
    the job stayed queued),
  • an unrelated evaluation's queued job is untouched,
  • deleting a submission-backed evaluation is still rejected with 400.

Verification

  • python -m pytest -q (validator suite) green against Postgres 16.
  • ruff check . clean in validator/.

DELETE /api/admin/evaluations/{id} soft-deleted a standalone provider

evaluation but left its provider_eval row in job_queues untouched. The

worker only filters on JobQueue.status == "queued" and never checks

EvaluationRun.deleted_at, so an eval deleted before it ran would still

be claimed, executed, and charged real provider spend -- then complete

while hidden from /api/provider-benchmarks. The submission delete path

already guards this via cancel_submission_jobs; add the equivalent

cancel_evaluation_jobs and call it from admin_delete_evaluation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant