Skip to content

Add PromptEvent metric for tracking prompt events#852

Open
svarlamov wants to merge 6 commits intomainfrom
feat/prompt-event-metric
Open

Add PromptEvent metric for tracking prompt events#852
svarlamov wants to merge 6 commits intomainfrom
feat/prompt-event-metric

Conversation

@svarlamov
Copy link
Copy Markdown
Member

@svarlamov svarlamov commented Mar 28, 2026

Summary

Add PromptEvent metric (event ID 5) for tracking individual events within AI prompt sessions. Events are emitted as a side-effect of the existing checkpoint command — no separate command or hook configuration needed.

Event kinds

  • HumanMessage — user prompts
  • AiMessage — assistant text responses
  • ThinkingMessage — assistant thinking blocks (Claude transcript only)
  • ToolCall — non-file-editing tool invocations
  • FileWrite — file-editing tool invocations (Edit, Write, etc.)

Design

  • Single command: Prompt events emit from within the checkpoint handler, keeping hooks simple (one command per event)
  • Content-based stable IDs: SHA256 of prompt_id:kind:content:index, prefixed with prompt_id
  • Session state persistence: Tracks emitted events for idempotency and rollback detection
  • Claude Code: Full transcript parsing with exact parent IDs
  • All other agents: Generic handler using hook input data with estimated parent IDs

Agent support

All agents with hook mechanisms emit prompt events via their existing checkpoint hooks:

  • Claude Code, Cursor, GitHub Copilot, Windsurf, Droid, Gemini, Amp, OpenCode

Key files

  • src/commands/prompt_event.rs — core logic (transcript parsing, event emission, state management)
  • src/commands/git_ai_handlers.rs — checkpoint handler calls emit_prompt_events_from_checkpoint
  • src/metrics/events.rs — PromptEventValues metric definition
  • agent-support/opencode/git-ai.ts — OpenCode plugin updated
  • agent-support/amp/git-ai.ts — Amp plugin (prompt events via checkpoint)

Test plan

  • 5 integration tests (transcript parsing, idempotency, rollback, growth, async_mode guard)
  • 12 unit tests (event ID stability, state roundtrip, metric values, transcript parsing)
  • All agent install_hooks tests pass (no duplicate hooks)
  • cargo clippy clean, cargo fmt clean
  • All Ubuntu CI checks passing

🤖 Generated with Claude Code

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

View 5 additional findings in Devin Review.

Open in Devin Review

Comment on lines +183 to +184
if s.len() > 256 {
s[..256].to_string()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 truncate_for_hash panics on multi-byte UTF-8 characters due to byte-based slicing

truncate_for_hash uses s[..256] which indexes by bytes, not characters. In Rust, slicing a &str at a non-character-boundary panics at runtime. If user messages or AI responses contain multi-byte UTF-8 characters (emojis, non-ASCII text, etc.) and byte position 256 falls in the middle of a multi-byte character, this will panic and crash the process.

Codebase already handles this correctly elsewhere

Other files in the same codebase use safe approaches:

  • src/commands/continue_session.rs:155-156: stdout.floor_char_boundary(MAX_DIFF_BYTES) with explicit comment "Use floor_char_boundary to avoid panicking on multi-byte UTF-8"
  • src/commands/debug.rs:691: trimmed.chars().take(200).collect()
  • src/commands/search.rs:591: text.chars().take(80).collect()
Suggested change
if s.len() > 256 {
s[..256].to_string()
if s.len() > 256 {
let safe_end = s.floor_char_boundary(256);
s[..safe_end].to_string()
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9c8a087 — now uses s.floor_char_boundary(256) to avoid panicking on multi-byte UTF-8.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

svarlamov and others added 6 commits March 29, 2026 21:54
Adds a new PromptEvent (event ID 5) metric that tracks individual events
within a prompt session — human messages, AI messages, thinking, tool
calls, and file writes. Each event has a content-based stable ID prefixed
with the prompt ID, and a parent ID for building event chains. Parent
resolution reads the agent transcript; when that fails, falls back to
the most recent event with an estimated flag.

- New metric type: PromptEventValues (kind, event_id, parent_id, parent_id_estimated)
- New command: `git-ai prompt-event <agent> --hook-input stdin`
- Claude Code hooks: UserPromptSubmit, PostToolUse (all tools), Stop
- OpenCode plugin: prompt-event hooks on tool.execute.after
- Only active in async_mode with daemon running
- State tracking per session for idempotency and rollback detection

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…f, Droid, Gemini, Amp)

Generalize prompt-event handler to support any agent without transcript
access using estimated parent IDs. Add prompt-event PostToolUse hooks to
all agent installers alongside existing checkpoint hooks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix non-deterministic event IDs in generic handler (was using SystemTime, now uses tool content)
- Fix Amp plugin session_id (was using per-tool toolUseID, now uses stable plugin-scoped UUID)
- Fix PreToolUse incorrectly mapped to HumanMessage in generic handler
- Extract apply_parent_id helper to deduplicate parent ID assignment pattern

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the separate `prompt-event` command. Prompt events are now emitted
as a side-effect of the checkpoint command, which already receives all
needed context via hook input. This simplifies the hook configuration
(one command per hook event instead of two) and keeps all logic in one place.

- Emit prompt events early in handle_checkpoint before agent preset runs
- Remove PROMPT_EVENT_CMD constants and hook installation from all agents
- Remove is_git_ai_prompt_event_command utility
- Remove runPromptEvent from Amp plugin
- Update tests to invoke checkpoint instead of prompt-event
- Net: -391 lines

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix OpenCode plugin calling non-existent prompt-event command (use checkpoint)
- Remove redundant prompt-event call for file-edit tools in OpenCode plugin
- Fix truncate_for_hash panic on multi-byte UTF-8 (use floor_char_boundary)
- Fix event dedup for identical tool calls in generic and fallback handlers
  (use state.emitted_event_ids.len() as index instead of hardcoded 0)
- Update module docstring to reflect prompt-event is no longer a command

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@svarlamov svarlamov force-pushed the feat/prompt-event-metric branch from 9c8a087 to 19c579c Compare March 29, 2026 21:56
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

View 10 additional findings in Devin Review.

Open in Devin Review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 OpenCode file-edit tool events misclassified as ToolCall instead of FileWrite in prompt events

When process_generic_prompt_events processes hook data from OpenCode's file-edit tool checkpoints (PreToolUse/PostToolUse for edit, write, patch, multiedit), the tool_name is extracted at src/commands/prompt_event.rs:402-406 by looking for toolName in tool_input or tool_name at the top level. However, the OpenCode plugin's hook input for file-edit tools only includes filePath in tool_input (agent-support/opencode/git-ai.ts:92 and agent-support/opencode/git-ai.ts:138), never toolName. This causes tool_name to default to "unknown", and the event kind falls through to ToolCall instead of FileWrite at line 427. The new non-file-edit tool path at agent-support/opencode/git-ai.ts:111 correctly includes toolName, demonstrating the fix pattern — the file-edit hooks should also include toolName: input.tool in tool_input.

(Refers to lines 134-139)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant