fix(extraction): treat the en-dash as a clause boundary too, and pin hyphen non-boundary#65
Open
RobertSigmundsson wants to merge 1 commit into
Conversation
…hyphen non-boundary The clause-boundary set had the em-dash (—) but not the en-dash (–, U+2013), which editors and operating systems emit just as often for ranges/asides. Without it the same cross-clause junk bigram the em-dash fix targets slips through when the separator happens to be an en-dash. Add the en-dash to _CLAUSE_BOUNDARY and add two regression tests: one that no bigram bridges an en-dash, and one that a plain ASCII hyphen is NOT a clause boundary (compound halves like "cache-aside" must still pair).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Toni!
The two non-blockers you flagged on #57, folded into one small change.
En-dash as a clause boundary
_CLAUSE_BOUNDARYhad the em-dash (—) but not the en-dash (–, U+2013). Editors and operating systems emit the en-dash just as often for ranges and asides, so the same cross-clause junk bigram the em-dash fix targets (a filename fragment glued onto the next clause) still slipped through when the separator was an en-dash. Added it to the set.Hyphen non-boundary — now pinned
You noted the hyphen-non-boundary behaviour had no dedicated test. Added one: a plain ASCII hyphen must NOT split a clause, so compound halves like
cache-asidestill pair as a bigram — kept as an explicit contrast to the en-/em-dash, which do split.Type of Change
Testing
test_no_bigram_across_en_dashandtest_hyphen_is_not_a_clause_boundaryadded toTestClauseBoundaryBigrams.main): 6,082 passed, 33 skipped in ~30 s (-n0).ruff+mypyclean. (The literal en-dash in the regex and one test string carry a# noqa: RUF001— ruff flags a bare en-dash as confusable with a hyphen, but here we want it literal.)Note on the ma/na/co/sa question (separate)
I ran the collision audit you suggested for the four Polish stop-words; it points at a different fix than removal (the
N/A/S.A.collisions don't actually reproduce), so I opened #64 for that discussion rather than bundling it here.Robert