Add compare command for transcript diff against gold-standard references#71
Merged
Add compare command for transcript diff against gold-standard references#71
Conversation
Introduces a `compare` CLI command that takes main (.cha) and gold (.gold.cha) transcripts, aligns them word-by-word using the existing Hirschberg DP algorithm with conform/match_fn normalization, and annotates each utterance with %xsrep (diff tokens with +/- markers) and %xsmor (POS tags) tiers. An analysis engine computes WER metrics and writes them to a .compare.csv file. Changes: - document.py: Add CompareToken model, Utterance.comparison field, Task.COMPARE and Task.COMPARE_ANALYSIS enum entries - formats/chat/parser.py: Parse %xsrep/%xsmor lines into CompareToken list - formats/chat/generator.py: Emit %xsrep/%xsmor lines from comparison data - pipelines/analysis/compare.py: New CompareEngine (processing) and CompareAnalysisEngine (analysis) with self-contained conform/match_fn - pipelines/dispatch.py: Register new engines in DEFAULT_PACKAGES - cli/dispatch.py: Add compare command handling with gold file pairing - cli/cli.py: Add compare Click command with --lang and --merge-abbrev - cli/bench.py: Add compare to benchmark command choices - tests: Add pytest.importorskip() for optional deps (torch, numpy, num2words, filelock, praatio) so tests skip gracefully https://claude.ai/code/session_01QGhmSCjjWdLn43o9PvB85b
- Parse main doc without special_mor_ so existing %mor/%gra are read into Form objects normally (Stanza overwrites, generator emits once) - Gold still uses special_mor_=True for lenient parsing - Track form_idx for each comparison token and interleave punctuation at original positions instead of appending all punct at end https://claude.ai/code/session_01QGhmSCjjWdLn43o9PvB85b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces a
compareCLI command that takes main (.cha) and gold (.gold.cha)transcripts, aligns them word-by-word using the existing Hirschberg DP algorithm
with conform/match_fn normalization, and annotates each utterance with %xsrep
(diff tokens with +/- markers) and %xsmor (POS tags) tiers. An analysis engine
computes WER metrics and writes them to a .compare.csv file.
Changes:
and Task.COMPARE_ANALYSIS enum entries
CompareAnalysisEngine (analysis) with self-contained conform/match_fn
num2words, filelock, praatio) so tests skip gracefully
https://claude.ai/code/session_01QGhmSCjjWdLn43o9PvB85b