Skip to content

Enhance apply_diff tool description to improve success rate for Gemini models #611

Description

@awschmeder

Description

Gemini Flash models (Flash and Flash Lite) and other smaller or fast models frequently struggle with the apply_diff tool, resulting in failed edit operations. Analysis suggests these failures stem from imprecise adherence to the expected diff format, leading to mismatches.

The current system instructions for apply_diff (defined in src/core/prompts/tools/native-tools/apply_diff.ts) are insufficient for these models.

Related Issues and PRs

Proposed Fix

Update the documentation and formatting instructions for the apply_diff tool to include stricter, clearer requirements that emphasize 100% string matches and the exact formatting of the :start_line: header.

Specifically, update the instructions in src/core/prompts/tools/native-tools/apply_diff.ts to include:

  1. Strict :start_line: Formatting: Emphasize that the syntax must be exactly :start_line:[integer] (e.g., :start_line:220) and prohibit shorthand forms like :220.
  2. Emphasis on Exact Matching: Explicitly instruct the model to copy exact lines from the source file for a 100% string match.

The following addition to AGENTS.md has been found to resolve the issue and should be integrated into the core Zoo prompts:

# Diff-Based Editing & apply_diff Tool Usage
When targeting text for a diff using the `apply_diff` tool:
- The `:start_line:[line_number]` header is required and must follow the exact syntax `:start_line:[integer]` (for example, `:start_line:220`).
- Do not write headers with shorthand forms like `:220`.
- Copy the exact lines from the source file for a 100% string match.

Test Procedure

  1. Verify existing apply_diff behavior with apps/vscode-e2e/src/suite/tools/apply-diff.test.ts.
  2. Apply the documentation update.
  3. Observe if Gemini models show an improved success rate in apply_diff operations in subsequent interactions.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions