Bug
When the implementation node encounters a persistent error (e.g., trying to checkout a non-existent fork branch), it retries in a tight loop without:
- Cleaning up failed workspace directories — each retry creates a new
/tmp/forge-{TICKET}-{random}/ directory
- Backoff or delay — retries happen as fast as the loop can run
- Retry cap — the loop continues until the disk fills up or an external limit is hit
Observed behavior
Setting fork_owner/fork_repo in the checkpoint to values where the branch doesn't exist on the fork causes:
git checkout -b forge/aisos-2119 fork/forge/aisos-2119
fatal: 'fork/forge/aisos-2119' is not a commit and a branch 'forge/aisos-2119' cannot be created from it
The error is non-recoverable (the branch will never appear), but the node retries immediately. In ~30 seconds this created 50,230 workspace directories in /tmp, consuming ~13GB and filling the tmpfs.
Root cause
In src/forge/workflow/nodes/implementation.py, the error handler returns:
return {
**state,
"last_error": str(exc),
"current_node": implementation_node,
}
This routes back to the same node, which the graph re-enters immediately. The prepare_workspace() call creates a new directory on each attempt but never cleans up on failure.
Expected behavior
- Failed workspace directories should be cleaned up before retry
- Exponential backoff or a retry limit should prevent tight loops
- After N consecutive failures (e.g., 3), the node should escalate to
blocked status instead of retrying indefinitely
Reproduction
- Set
fork_owner and fork_repo in a ticket's checkpoint to valid values
- Delete the corresponding branch from the fork
- Trigger the implementation node
- Watch
/tmp fill with workspace directories
Environment
- Forge commit: main branch (observed on
forge/aisos-741 feature branch)
- OS: Fedora 43, tmpfs at
/tmp
Bug
When the implementation node encounters a persistent error (e.g., trying to checkout a non-existent fork branch), it retries in a tight loop without:
/tmp/forge-{TICKET}-{random}/directoryObserved behavior
Setting
fork_owner/fork_repoin the checkpoint to values where the branch doesn't exist on the fork causes:The error is non-recoverable (the branch will never appear), but the node retries immediately. In ~30 seconds this created 50,230 workspace directories in
/tmp, consuming ~13GB and filling the tmpfs.Root cause
In
src/forge/workflow/nodes/implementation.py, the error handler returns:This routes back to the same node, which the graph re-enters immediately. The
prepare_workspace()call creates a new directory on each attempt but never cleans up on failure.Expected behavior
blockedstatus instead of retrying indefinitelyReproduction
fork_ownerandfork_repoin a ticket's checkpoint to valid values/tmpfill with workspace directoriesEnvironment
forge/aisos-741feature branch)/tmp