feat(analyzer): add Czech PII recognizers (CZ_*)#2155
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds first-class Czech (cs) analyzer support by introducing new pattern-based CZ_* recognizers (plus Czech DATE_TIME notation support), wiring them into the predefined registry/configuration, and documenting how to run a bilingual EN+CS pipeline.
Changes:
- Added 5 Czech PII recognizers (
CZ_BIRTH_NUMBER,CZ_BANK_ACCOUNT,CZ_ID_CARD,CZ_PASSPORT,CZ_DRIVER_LICENSE) and a CzechDATE_TIMErecognizer (CzDateRecognizer). - Registered the new recognizers in
predefined_recognizers/__init__.pyandconf/default_recognizers.yaml(disabled by default,country_code: cz). - Added documentation and recipes for Czech language support, plus comprehensive unit and end-to-end tests.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| presidio-analyzer/tests/test_czech_language_support.py | End-to-end Czech pipeline test + PHONE_NUMBER collision regression coverage |
| presidio-analyzer/tests/test_cz_passport_recognizer.py | Unit tests for CzPassportRecognizer patterns/validation |
| presidio-analyzer/tests/test_cz_id_card_recognizer.py | Unit tests for CzIdCardRecognizer patterns/validation |
| presidio-analyzer/tests/test_cz_driver_license_recognizer.py | Unit tests for CzDriverLicenseRecognizer patterns/validation |
| presidio-analyzer/tests/test_cz_date_recognizer.py | Unit tests for CzDateRecognizer Czech date notation coverage |
| presidio-analyzer/tests/test_cz_birth_number_recognizer.py | Unit tests for CzBirthNumberRecognizer (date semantics + mod-11 / exception) |
| presidio-analyzer/tests/test_cz_bank_account_recognizer.py | Unit tests for CzBankAccountRecognizer (weighted mod-11 + invalidations) |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_passport_recognizer.py | New Czech passport recognizer implementation |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_id_card_recognizer.py | New Czech ID card recognizer implementation |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_driver_license_recognizer.py | New Czech driver license recognizer implementation |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_date_recognizer.py | New Czech date notation recognizer emitting DATE_TIME |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_birth_number_recognizer.py | New rodné číslo recognizer with checksum/exception handling |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_bank_account_recognizer.py | New Czech bank account recognizer with validation + date-fragment invalidation |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/init.py | Czechia country-specific recognizers package marker |
| presidio-analyzer/presidio_analyzer/predefined_recognizers/init.py | Exposed new Cz*Recognizer classes via public predefined recognizers API |
| presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml | Added Czech recognizers to default registry config (disabled by default) |
| docs/supported_entities.md | Documented Czechia entity types and Czech DATE_TIME coverage |
| docs/recipes/czech-language-support/spacy_en_cs.yaml | Bilingual EN+CS spaCy engine config (en_core_web_lg + xx_ent_wiki_sm) |
| docs/recipes/czech-language-support/README.md | Usage recipe and guidance for Czech-language pipelines |
| CHANGELOG.md | Added unreleased entry describing Czech recognizers + recipe |
fix(analyzer): use lookbehind anchors in CZ_PASSPORT and CZ_DRIVER_LICENSE patterns Replace \b with (?<![\w/-]) in the letter-prefixed patterns so they no longer match inside slash/hyphen-delimited identifiers (e.g. AB123456 inside "123-AB123456"), aligning them with the anchoring used by the other Czech recognizers. Adds regression tests for the blocked contexts. Addresses Copilot review feedback.m>
| found_entities = {result.entity_type for result in results} | ||
| assert found_entities <= {"PHONE_NUMBER", "EMAIL_ADDRESS", "IBAN_CODE", "DATE_TIME"} | ||
| assert "CZ_BIRTH_NUMBER" not in found_entities | ||
| assert "PERSON" not in found_entities |
| - `spacy_en_cs.yaml` — a ready-to-use NLP engine configuration that loads both | ||
| `en_core_web_lg` and the multilingual `xx_ent_wiki_sm` (spaCy publishes no | ||
| Czech-specific pretrained pipeline; the multilingual WikiNER model is the | ||
| official artifact that covers Czech PERSON/LOCATION/ORG detection) |
There was a problem hiding this comment.
Worth mentioning that xx_ent_wiki_sm was not trained on Czech-specific, so one should not assume very high accuracy specifically on Czech entities
| @@ -0,0 +1,179 @@ | |||
| # Czech Language Support | |||
There was a problem hiding this comment.
Very well written! Thanks!
|
|
||
| return None | ||
|
|
||
| def invalidate_result(self, pattern_text: str) -> Optional[bool]: |
There was a problem hiding this comment.
Bank account pattern might flag Czech legal citations as accounts (often at score 1.0)
The NNN/YYYY form of Czech legal references collides with the account pattern (?:\d{2,6}-)?\d{2,10}/\d{4}, and this invalidate_result guard only filters them when the numerator is 1–2 digits and <= 31 (the date case). Real law references are 3+ digits and pass straight through:
262/2006 (the actual Zákoník práce / Labour Code) passes the _mod11 checksum, so validate_result returns True and it's promoted to MAX_SCORE (1.0).
115/2010 — same, and the test ("115/2010", None) documents that it isn't invalidated today.
169/2011, 133/2000, 50/2019 — fail the checksum but still surface at the base 0.4.
This matters because the accompanying recipe targets legal / public-administration text, where these citations are extremely common — so it's a high-frequency false positive for the intended audience, not an edge case.
Suggested fix: reject a slash match when there's no - prefix and the 4-digit bank_code is in the 1900–2099 range, regardless of numerator length (drop the len(account_part) <= 2 / <= 31 restriction for that case). Better still, validate bank_code against the finite set of real ČNB bank codes — a year like 2006/2010 is not an issued code. Please also add positive assert no match tests for 262/2006 and 115/2010.
|
Hi @santiagotri, thanks for this PR! great work. There are a few minor comments to address and we can merge it |
Summary
Adds 5 new pattern-based recognizers for Czech-language PII plus Czech
DATE_TIMEdate coverage, covering the most relevant entity types under GDPR / zákon č. 110/2019 Sb.,
and fixes the collision where Czech identifiers (rodné číslo, bank accounts) were
mis-detected as
PHONE_NUMBER:[prefix-]number/bank code)CzDateRecognizerAlso ships a Czech language support recipe (
docs/recipes/czech-language-support/)with a ready-to-use bilingual EN+CS NLP configuration (
spacy_en_cs.yaml).Design decisions
PatternRecognizerpattern and are placed inpredefined_recognizers/country_specific/czechia/, mirroring the Germanimplementation from feat(analyzer): add German PII recognizers (DE_*) #1909
enabled: false) following the project conventionfor country-specific recognizers; registered for
supported_languages: [cs]only,with
country_code: czpass promotes to
MAX_SCORE, but a checksum failure on an otherwise date-plausiblenumber returns
None(keep base score, let context words decide) rather thanFalse. Rationale: legitimate rodná čísla exist that fail strict mod 11 — 9-digitpre-1954 numbers carry no check digit at all, and ~1000 numbers issued 1954–1985
fall under the historical mod-10 exception — so a hard reject would drop real PII.
Falseis reserved for structurally impossible values (all-zero suffix/account,bank code
0000, wrong length)PhoneRecognizerscores a flat 0.4 and previouslywon on
850412/0003and on part of19-123456789/0800. Base scores plus contextboosting put the Czech entities at 0.55–0.85 (1.0 on valid checksums), so
score-based conflict resolution (as applied by presidio-anonymizer) picks the
Czech entity; the genuine phone number
+420 601 234 567is unaffected"rodného", …) because the
LemmaContextAwareEnhancersubstring-matches lemmas andno Czech lemmatizer is available in the recommended pipeline
(?<![\w/-])…) instead of bare\bto preventmatches inside bank accounts, IBANs, and longer digit runs
CZ_PASSPORT/CZ_DRIVER_LICENSEenforce an uppercase prefix invalidate_result(): patterns are matched withIGNORECASE(global flags), soordinary two-letter words before a number ("je 123456") would otherwise match
CzDateRecognizeremits the standardDATE_TIMEentity because no available CzechNER model emits DATE labels;
CZ_BANK_ACCOUNTinvalidates date-likedd/yyyyfragments (e.g.
04/2023)uses the official multilingual
xx_ent_wiki_smfor PERSON/LOCATION/ORG anddocuments a HuggingFace transformers model as the higher-quality upgrade path
global_regex_flags: 26(IGNORECASE | MULTILINE | DOTALL)consistent with the rest of Presidio
Test plan
presidio-analyzer/tests/test_cz_*.py) plusan end-to-end scenario (
test_czech_language_support.py)validate_result()/invalidate_result()tests for thechecksum-based recognizers (valid, historical-exception, failing, and
structurally impossible inputs)
address, phone, e-mail, ID card, passport, driving licence, bank account,
IBAN), asserts every span/entity, includes a before-state regression test for
the PHONE_NUMBER collision, and verifies the exact anonymized output — using
precomputed NLP artifacts so CI needs no model download
python -m pytest presidio-analyzer/tests/test_cz*.pymainbaseline (remainingfailures are pre-existing optional-dependency/model gaps, verified via stash
comparison)
ruff checkpasses with no issuesdefault_recognizers.yamlentries follow the existing country-specificpattern; default en-only registry verified unaffected via
RecognizerRegistryProviderload testdocs/supported_entities.mdupdated with Czechia sectiondocs/recipes/czech-language-support/-
CHANGELOG.mdentry added under[unreleased]Checklist
🤖 Generated with Claude Code