feat: additive include filters, external prompts, and XML output format (#20)

* fix: make --include-dir and --include-files work additively

Previously, when both --include-dir and --include-files were specified,
includeFiles would clear the includeDirs patterns, making them mutually
exclusive. Now they combine additively - files from both filters are
included in the output.

* test: add E2E tests for combined --include-dir and --include-files

- Add integration tests verifying additive behavior of includeDirs + includeFiles
- Add sample-rust-project in playground/ for manual testing
- Tests verify: tree output, prompt, file inclusion, ignores working correctly

* fix: support external prompt file paths and prepend prompts without placeholder

- External paths (containing / or \) are now used directly instead of
  being nested under codefetch/prompts/
- Prompts without {{CURRENT_CODEBASE}} placeholder are prepended to
  the codebase content instead of replacing it

* feat: add XML tags for structured output sections

- Wrap prompts in <task>...</task> tags
- Wrap file tree in <filetree>...</filetree> tags
- Wrap source code in <source_code>...</source_code> tags

This provides better structure for AI models to understand the different
sections of the codebase output.

* docs: update README and changelogs for v2.2.0

- Document XML-structured output format with <task>, <filetree>, <source_code> tags
- Document additive --include-dir and --include-files behavior
- Document external prompt file path support
- Update all changelogs (root, cli, sdk) for v2.2.0

* chore: update pnpm-lock.yaml

* docs: add pnpm lockfile troubleshooting to HOW-TO-RELEASE

* fix: use workspace:* for codefetch-sdk dependency

Author: regenrek

CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 2.2.0
+
+### Added
+- **XML-structured output format** - Output now uses semantic XML tags for better AI parsing:
+  - `<task>...</task>` - Wraps the prompt/instructions
+  - `<filetree>...</filetree>` - Wraps the project tree structure
+  - `<source_code>...</source_code>` - Wraps all source code files
+- **Additive `--include-dir` and `--include-files`** - These options now work together additively instead of being mutually exclusive. Use both to include specific directories PLUS specific files.
+- **External prompt file support** - Prompt files with paths (e.g., `-p docs/arch/prompt.md`) are now correctly resolved from the project root instead of requiring them to be in `codefetch/prompts/`
+
+### Fixed
+- Fixed `--include-dir` and `--include-files` being mutually exclusive - now they combine additively
+- Fixed external prompt file paths not being found when containing directory separators
+- Fixed prompts without `{{CURRENT_CODEBASE}}` placeholder not including the codebase content
+
 ## 2.1.2
 
 ### Fixed

HOW-TO-RELEASE.md

@@ -160,6 +160,13 @@ Delete the GitHub Release if needed: `gh release delete vX.Y.Z`
 - `npm ERR! code E403` or auth failures: run `npm login` and retry
 - `gh` failures: `gh auth status`; ensure `repo` scope exists
 - Tag push rejected: pull/rebase or fast-forward `main`, then rerun
+- **CI fails with `ERR_PNPM_OUTDATED_LOCKFILE`**: The lockfile is out of sync with `package.json`. This happens when dependencies change (e.g., `workspace:*` → `^2.1.0`). Fix it locally:
+  ```bash
+  pnpm install --no-frozen-lockfile
+  git add pnpm-lock.yaml
+  git commit -m "chore: update pnpm-lock.yaml"
+  git push
+  ```
 
 ## Release Frequency Suggestions
 

README.md

@@ -129,6 +129,10 @@ npx codefetch --include-files "src/components/AgentPanel.tsx,src/lib/llm/**/*" -
 
 # Include src directory, exclude test files
 npx codefetch --include-dir src --exclude-files "*.test.ts" -o src-no-tests.md
+
+# Combine --include-dir and --include-files (additive!)
+# This includes ALL files from crates/core/src PLUS the specific lib.rs file
+npx codefetch --include-dir crates/core/src --include-files "crates/engine/src/lib.rs" -o combined.md
 ```
 
 Dry run (only output to console)
@@ -289,10 +293,20 @@ Inline prompts are automatically appended with the codebase content.
 
 #### Custom Prompt Files
 
-Create custom prompts in `codefetch/prompts/` directory:
+You can use custom prompt files in two ways:
 
-1. Create a markdown file (e.g., `codefetch/prompts/my-prompt.md`)
-2. Use it with `--prompt my-prompt.md`
+**1. External prompt files (anywhere in your project):**
+```bash
+# Use a prompt file from anywhere in your project
+npx codefetch -p docs/arch/review-prompt.md
+npx codefetch --prompt ./prompts/security-audit.txt
+```
+
+**2. Prompt files in `codefetch/prompts/` directory:**
+```bash
+# Create codefetch/prompts/my-prompt.md, then use:
+npx codefetch --prompt my-prompt.md
+```
 
 You can also set a default prompt in your `codefetch.config.mjs`:
 
@@ -344,6 +358,41 @@ Codefetch uses a set of default ignore patterns to exclude common files and dire
 
 You can view the complete list of default patterns in [default-ignore.ts](packages/sdk/src/default-ignore.ts).
 
+## Output Format
+
+Codefetch generates structured output using semantic XML tags to help AI models better understand the different sections:
+
+```xml
+<task>
+Your prompt or instructions here...
+</task>
+
+<filetree>
+Project Structure:
+└── src
+    ├── index.ts
+    └── utils
+        └── helpers.ts
+</filetree>
+
+<source_code>
+src/index.ts
+```typescript
+// Your code here
+```
+
+src/utils/helpers.ts
+```typescript
+// More code here
+```
+</source_code>
+```
+
+The XML structure provides:
+- `<task>` - Contains your prompt/instructions (from `-p` flag)
+- `<filetree>` - Contains the project tree visualization (from `-t` flag)
+- `<source_code>` - Contains all the source code files with their paths
+
 ## Token Counting
 
 Codefetch supports different token counting methods to match various AI models:

packages/cli/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 2.2.0
+
+### Added
+- **XML-structured output format** - Output now uses semantic XML tags for better AI parsing:
+  - `<task>...</task>` - Wraps the prompt/instructions
+  - `<filetree>...</filetree>` - Wraps the project tree structure
+  - `<source_code>...</source_code>` - Wraps all source code files
+- **External prompt file support** - Prompt files with paths (e.g., `-p docs/arch/prompt.md`) are now correctly resolved from the project root
+
+### Fixed
+- Fixed `getPromptFile` not resolving external file paths correctly when they contain directory separators
+
 ## 2.1.2
 
 ### Fixed

packages/cli/package.json

@@ -41,7 +41,7 @@
   "dependencies": {
     "@clack/prompts": "^0.11.0",
     "c12": "^2.0.1",
-    "codefetch-sdk": "^2.1.0",
+    "codefetch-sdk": "workspace:*",
     "consola": "^3.3.3",
     "ignore": "^7.0.0",
     "mri": "^1.2.0",

packages/cli/src/commands/default.ts

@@ -29,6 +29,15 @@ function getPromptFile(
   if (VALID_PROMPTS.has(config.defaultPromptFile)) {
     return config.defaultPromptFile;
   }
+  // Check if it's an external file path (contains path separator or is absolute)
+  // External paths should be used as-is, not nested under codefetch/prompts/
+  if (
+    config.defaultPromptFile.includes("/") ||
+    config.defaultPromptFile.includes("\\") ||
+    config.defaultPromptFile.startsWith(".")
+  ) {
+    return resolve(config.defaultPromptFile);
+  }
   return resolve(config.outputPath, "prompts", config.defaultPromptFile);
 }
 

packages/cli/src/commands/open.ts

@@ -35,6 +35,15 @@ function getPromptFile(
   if (VALID_PROMPTS.has(config.defaultPromptFile)) {
     return config.defaultPromptFile;
   }
+  // Check if it's an external file path (contains path separator or is absolute)
+  // External paths should be used as-is, not nested under codefetch/prompts/
+  if (
+    config.defaultPromptFile.includes("/") ||
+    config.defaultPromptFile.includes("\\") ||
+    config.defaultPromptFile.startsWith(".")
+  ) {
+    return resolve(config.defaultPromptFile);
+  }
   return resolve(config.outputPath, "prompts", config.defaultPromptFile);
 }
 

packages/cli/test/integration/codebase-fixture.test.ts

@@ -267,4 +267,87 @@ describe("Integration: codebase-test fixture", () => {
     expect(content).toContain("button.js");
     expect(content).toContain("utils");
   });
+
+  it("combines --include-dir and --include-files additively", () => {
+    const result = spawnSync(
+      "node",
+      [
+        cliPath,
+        "-o",
+        "combined-include.md",
+        "--include-dir",
+        "src/utils",
+        "--include-files",
+        "src/components/button.js",
+        "-t",
+        "3",
+      ],
+      {
+        cwd: FIXTURE_DIR,
+        encoding: "utf8",
+        stdio: ["inherit", "pipe", "pipe"],
+      }
+    );
+
+    expect(result.stderr).toBe("");
+    expect(result.stdout).toContain("Output written to");
+
+    const outPath = join(CODEFETCH_DIR, "combined-include.md");
+    expect(fs.existsSync(outPath)).toBe(true);
+
+    const content = fs.readFileSync(outPath, "utf8");
+    // Should include files from utils directory (via --include-dir)
+    expect(content).toContain("test1.ts");
+    expect(content).toContain("test2.js");
+    // Should include specific file (via --include-files)
+    expect(content).toContain("button.js");
+    // Should NOT include other files not matching the patterns
+    expect(content).not.toContain("app.js");
+    expect(content).not.toContain("header.js");
+    expect(content).not.toContain("container.js");
+    // Project tree should be present
+    expect(content).toMatch(/Project Structure:/);
+  });
+
+  it("combines multiple --include-dir directories with --include-files", () => {
+    const result = spawnSync(
+      "node",
+      [
+        cliPath,
+        "-o",
+        "multi-dir-include.md",
+        "--include-dir",
+        "src/utils,src/components/base",
+        "--include-files",
+        "src/app.js",
+        "-t",
+        "3",
+      ],
+      {
+        cwd: FIXTURE_DIR,
+        encoding: "utf8",
+        stdio: ["inherit", "pipe", "pipe"],
+      }
+    );
+
+    expect(result.stderr).toBe("");
+    expect(result.stdout).toContain("Output written to");
+
+    const outPath = join(CODEFETCH_DIR, "multi-dir-include.md");
+    expect(fs.existsSync(outPath)).toBe(true);
+
+    const content = fs.readFileSync(outPath, "utf8");
+    // Should include files from utils directory
+    expect(content).toContain("test1.ts");
+    expect(content).toContain("test2.js");
+    // Should include files from components/base directory
+    expect(content).toContain("container.js");
+    // Should include the specific file app.js
+    expect(content).toContain("app.js");
+    // Should NOT include files from other directories
+    expect(content).not.toContain("button.js");
+    expect(content).not.toContain("header.js");
+    // Project tree should be present
+    expect(content).toMatch(/Project Structure:/);
+  });
 });

packages/cli/test/unit/markdown.test.ts

@@ -8,7 +8,8 @@ const UTILS_DIR = join(FIXTURE_DIR, "src/utils");
 
 describe("generateMarkdown with chunk-based token limit", () => {
   it("enforces maxTokens by chunk-based reading", async () => {
-    const MAX_TOKENS = 50;
+    // Note: XML tags (<source_code>, </source_code>) add ~4 tokens overhead
+    const MAX_TOKENS = 55;
     const files = [join(UTILS_DIR, "test1.ts"), join(UTILS_DIR, "test2.js")];
 
     const result = await generateMarkdown(files, {
@@ -72,6 +73,12 @@ describe("generateMarkdown with chunk-based token limit", () => {
       disableLineNumbers: false,
     });
 
+    // Check for XML tags
+    expect(markdown).toContain("<filetree>");
+    expect(markdown).toContain("</filetree>");
+    expect(markdown).toContain("<source_code>");
+    expect(markdown).toContain("</source_code>");
+    // Check content
     expect(markdown).toContain("Project Structure:");
     expect(markdown).toMatch(/└── /);
     expect(markdown).toContain("test1.ts");
@@ -81,16 +88,17 @@ describe("generateMarkdown with chunk-based token limit", () => {
   it("respects token limits with project tree", async () => {
     const files = [join(UTILS_DIR, "test1.ts")];
 
+    // Note: XML tags (<filetree>, <source_code>) add overhead
     const markdown = await generateMarkdown(files, {
-      maxTokens: 20,
+      maxTokens: 40,
       verbose: 0,
       projectTree: 2,
       tokenEncoder: "simple",
       disableLineNumbers: false,
     });
 
     const tokens = await countTokens(markdown, "simple");
-    expect(tokens).toBeLessThanOrEqual(20);
+    expect(tokens).toBeLessThanOrEqual(40);
   });
 });
 

packages/sdk/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 2.2.0
+
+### Added
+- **XML-structured output format** - Output now uses semantic XML tags for better AI parsing:
+  - `<task>...</task>` - Wraps the prompt/instructions
+  - `<filetree>...</filetree>` - Wraps the project tree structure
+  - `<source_code>...</source_code>` - Wraps all source code files
+- **Additive `includeDirs` and `includeFiles`** - These options now work together additively instead of being mutually exclusive
+- Added `hasCodebasePlaceholder` helper function to `template-parser.ts`
+
+### Fixed
+- Fixed `collectFiles` treating `includeDirs` and `includeFiles` as mutually exclusive - now they combine additively
+- Fixed `processPromptTemplate` to prepend prompts without `{{CURRENT_CODEBASE}}` placeholder to the codebase content instead of replacing it
+
 ## 2.0.4
 
 ### Added