Pathological single-statement input can hang indexing for tens of minutes — tree-sitter progress callback doesn't fire during error recovery

[murari316]

Problem

A single large source file — specifically one whose grammar produces a very small number of statements relative to file size (e.g. a SQL file with only a handful of top-level statements but each spanning megabytes, such as one INSERT ... VALUES (...), (...), ... with tens of thousands of value-tuples) — can cause infigraph index to hang for tens of minutes on that one file, without finishing.

Live profiling (macOS sample, repeated across multiple runs) shows the parsing thread stuck almost entirely inside tree-sitter's internal error-recovery path:

extract::extract_file
  → ts_parser_parse_with_options
    → ts_parser_parse
      → ts_parser__recover
        → ts_subtree_summarize_children
        → stack_node_release / ts_subtree_release

This matches tree-sitter's GLR error-recovery machinery thrashing on a single large, ambiguous statement rather than making forward progress.

Root cause: progress callback does not reliably fire during recovery

infigraph-core's extract_file (crates/infigraph-core/src/extract/mod.rs) uses tree-sitter's parse_with_options with a progress_callback (the mechanism tree-sitter 0.25+ recommends for parse cancellation, replacing the older set_timeout_micros, which was removed entirely by 0.26). The callback tracks ParseState::current_byte_offset() and requests cancellation (ControlFlow::Break) once byte-offset progress has stalled for a grace window.

This was implemented and tested against a real reproduction of the hang:

• On one run, the callback correctly detected the stall and cancelled the parse within the expected window.
• On repeated identical runs against the exact same input and binary, the callback did not fire at all — profiling during the hang showed zero samples anywhere near the callback/checkpoint, only deep inside ts_parser__recover's reduction/release internals, for the entire duration (multiple minutes, would otherwise have continued far longer based on the original observed ~40 minute hang).

This indicates tree-sitter's error-recovery loop does not consistently return to the checkpoint where the progress callback is invoked — the callback-based cancellation mechanism appears to be unreliable specifically during recovery, as opposed to normal parsing progress. Cancellation therefore cannot be counted on to bound worst-case parse time when a file triggers recovery-heavy behavior.

What's needed

A real fix would require either:

• A tree-sitter-level change so recovery genuinely checks/respects the progress callback (or an equivalent cancellation point) on a bounded cadence regardless of how deep or how long the recovery loop runs, or
• Confirmation/guidance from tree-sitter's side on whether there's a supported way to bound recovery-loop duration that we're missing.

We deliberately have not shipped a workaround (e.g. skipping oversized files) as the fix for this — we'd like to understand whether a real, dependency-level fix is feasible before falling back to a heuristic in our own extraction pipeline.

Environment

• tree-sitter crate: 0.26 (workspace pins tree-sitter = "0.26"; confirmed same behavior against 0.26.11, the current latest release)
• Platform observed: macOS (arm64)
• Rust bindings: parse_with_options / ParseOptions::progress_callback / ParseState::current_byte_offset