Current bencher github action errors with API error

[Narfinger]

This is an example run that errors with:
"Failed to check API version: Failed to get API server version: Invalid response payload: EOF while parsing a value at line 1 column 0".
https://github.com/servo/servo/actions/runs/18558415230/job/52901840555
The status page also does not say anything is wrong.

[epompeii]

@Narfinger I'm sorry for the inconvenience.

There was an outage that was fixed this morning. It looks like that workflow run caught the tail end of it.

I have a root cause on the issue, and I'm working on a fix to prevent this from happening again.

[typfel]

Hi, even though I can see on the status page that bencher.api should be operational again we are still getting errors when trying to create new reports

Failed to create new report: Error processing request:
Status: 404 Not Found
Headers: {"access-control-allow-origin": "*", "access-control-allow-methods": "GET, POST, PUT, PATCH, DELETE, OPTIONS", "access-control-allow-headers": "Content-Type, Authorization", "access-control-expose-headers": "X-Total-Count", "content-type": "application/json", "x-request-id": "7bc2f9fe-ca47-40f3-a474-3ea62641a9c9", "content-length": "686", "date": "Fri, 17 Oct 2025 07:25:42 GMT", "server": "Fly/31563e821d (2025-10-15)", "via": "1.1 fly.io, 1.1 fly.io", "fly-request-id": "01K7RGYR960Y9V0561C4Y2X860-lax"}
Request ID: 7bc2f9fe-ca47-40f3-a474-3ea62641a9c9
Message: Head Version ((QueryBranch { id: BranchId(2230), uuid: BranchUuid(250e6bce-0bc1-4835-8b8e-b175777da959), project_id: ProjectId(227), name: BranchName("main"), slug: BranchSlug(Slug("main")), head_id: Some(HeadId(2230)), created: DateTime(2024-07-17T15:00:22Z), modified: DateTime(2024-07-17T15:00:22Z), archived: None }, Some(GitHash("206451e71a3fa91440967a29d366225cd57bcfb2")))) not found: Record not found
Head Version may be private and require authentication or it may not exist.
Expected format is `Authorization: *** Where `<bencher.api.token>` is your Bencher API token.

Was there some data loss and do I need to reset something?

[epompeii]

@typfel I'm sorry you're still running into issues.

I think what happened was:

1. Bencher was down
2. You all merged into main with hash 206451e71a3fa91440967a29d366225cd57bcfb2
3. Bencher is now back up
4. You are now trying to use main with hash 206451e71a3fa91440967a29d366225cd57bcfb2 as your Start Point

Possible Fixes:
A. Kick off a benchmark run for main with hash 206451e71a3fa91440967a29d366225cd57bcfb2
B. Remove this line from your workflow setup: --branch-start-point-hash '${{ github.event.pull_request.base.sha }}' \

[epompeii]

I'm going to close this issues out. The outage was fixed within hours of this issue being opened.

I refrained from closing this issue as I've continued to dig in. I believe the root cause was:

1. There was a long running transaction on the server
2. Another request was sent and then cancelled
3. The database access mutex was still locked

With all that said, I was not able to create a satisfying repro. My efforts will be focused here:

• Encapsulate write endpoints in a transaction #576
• Improving cancellation safety