agent: fail the job if process spawning fails

[emilyalbini]

Note: This PR is stacked on top of #101.

---

While working on post tasks, I discovered that the agent doesn't mark the job as failed if spawning tasks or diagnostics fail. We have never hit this in practice as we spawn /bin/bash for both of them, and the chances of that path not existing on the worker are negligible. This is far more likely to happen for post tasks, as we spawn the command requested by the user directly, and the command might not exist.

To test the agent didn't mark the job as failed, I patched the agent to try to spawn a missing binary rather than /bin/bash:

job 01KPN9E2F6K0FCXC6P1JKETF98 submitted
polling for job output...
STATE CHANGE:  -> queued
STATE CHANGE: queued -> running
|=| job assigned to worker 01KPN9E30QVKRF09BA9YV30BEK [factory aws-GotGCZDz, i-03ce969a962af0c5a] (queued for 32 s)
|T| starting task 0: "default"
JobEvent { payload: "ERROR: exec: No such file or directory (os error 2)", seq: 3, stream: "agent", task: None, time: 2026-04-20T11:12:32.081619121Z, time_remote: Some(2026-04-20T11:12:31.991713448Z) }
|W| found 0 output files
STATE CHANGE: running -> completed
|=| task 0 was incomplete, marked failed

Note how the state is changed to completed rather than failed. Task 0 being marked as failed doesn't change the state of the job as a whole.

[emilyalbini]

I assume this behavior was wrong, and that we shouldn't mark the worker as ready if the pre-diagnostics script fails to execute.

[jclulow]

It was, I believe, a deliberate choice. The pre-diagnostic scripts were something I added while debugging various bits of AWS-induced malaise, but I was generally leaning to failing open in this case if we weren't able to get it started.

It seems fine to tighten it up, though; we'll just need to be careful not to misconfigure the pre-diag stuff in such a way that we hit this condition.

[jclulow]

It was, I believe, a deliberate choice. The pre-diagnostic scripts were something I added while debugging various bits of AWS-induced malaise, but I was generally leaning to failing open in this case if we weren't able to get it started.

It seems fine to tighten it up, though; we'll just need to be careful not to misconfigure the pre-diag stuff in such a way that we hit this condition.

[jclulow]

I think not failing here if the post diagnostics couldn't get underway was definitely more of a deliberate choice, FWIW, as it means a job that had succeeded might then fail for what didn't seem like great reasons, etc. It's probably fine to tighten up, though, and we'll see how it goes!