Skip to content

Bug: Implementation node retry loop fills /tmp with workspace directories on persistent failure #191

Description

@JoshSalomon

Bug

When the implementation node encounters a persistent error (e.g., trying to checkout a non-existent fork branch), it retries in a tight loop without:

  1. Cleaning up failed workspace directories — each retry creates a new /tmp/forge-{TICKET}-{random}/ directory
  2. Backoff or delay — retries happen as fast as the loop can run
  3. Retry cap — the loop continues until the disk fills up or an external limit is hit

Observed behavior

Setting fork_owner/fork_repo in the checkpoint to values where the branch doesn't exist on the fork causes:

git checkout -b forge/aisos-2119 fork/forge/aisos-2119
fatal: 'fork/forge/aisos-2119' is not a commit and a branch 'forge/aisos-2119' cannot be created from it

The error is non-recoverable (the branch will never appear), but the node retries immediately. In ~30 seconds this created 50,230 workspace directories in /tmp, consuming ~13GB and filling the tmpfs.

Root cause

In src/forge/workflow/nodes/implementation.py, the error handler returns:

return {
    **state,
    "last_error": str(exc),
    "current_node": implementation_node,
}

This routes back to the same node, which the graph re-enters immediately. The prepare_workspace() call creates a new directory on each attempt but never cleans up on failure.

Expected behavior

  • Failed workspace directories should be cleaned up before retry
  • Exponential backoff or a retry limit should prevent tight loops
  • After N consecutive failures (e.g., 3), the node should escalate to blocked status instead of retrying indefinitely

Reproduction

  1. Set fork_owner and fork_repo in a ticket's checkpoint to valid values
  2. Delete the corresponding branch from the fork
  3. Trigger the implementation node
  4. Watch /tmp fill with workspace directories

Environment

  • Forge commit: main branch (observed on forge/aisos-741 feature branch)
  • OS: Fedora 43, tmpfs at /tmp

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions