diff --git a/CHANGELOG.md b/CHANGELOG.md index c353759932..8538dedbb4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,10 @@ All notable changes to this project will be documented in this file. ## [unreleased] ### Analyzer + +#### Added +- Czech PII recognizers for `CZ_BIRTH_NUMBER`, `CZ_BANK_ACCOUNT`, `CZ_ID_CARD`, `CZ_PASSPORT`, and `CZ_DRIVER_LICENSE`, plus Czech `DATE_TIME` date coverage (`CzDateRecognizer`); all are disabled by default. Includes a Czech language support recipe (`docs/recipes/czech-language-support`) + #### Fixed - Language model recognizers (`BasicLangExtractRecognizer`, `AzureOpenAILangExtractRecognizer`) configured in a recognizer registry YAML now honour `config_path` (and other recognizer-specific kwargs). Previously these entries were validated by the strict `PredefinedRecognizerConfig` schema, which has no `config_path` field and does not allow extra keys, so `config_path` was silently dropped and the recognizer fell back to its bundled default model configuration. Added a `LangExtractRecognizerConfig` model (`extra="allow"`) and registered both recognizer class names in `CONFIG_MODEL_MAP`. - `BasicLangExtractRecognizer` now honours values under `langextract.model.provider.language_model_params` (including `timeout` and `num_ctx`). Previously these were silently dropped because `langextract.extract()` ignores its `language_model_params` argument when a pre-built `ModelConfig` is passed via `config=`, causing Ollama-backed recognizers to fall back to langextract's 120s default regardless of the configured timeout. The recognizer now merges `language_model_params` into `ModelConfig.provider_kwargs`, which is the path that reaches the provider constructor. Explicit entries under `provider.kwargs:` still take precedence. Also fixed a `TypeError` when `kwargs:` or `language_model_params:` is `null` in the YAML. (#1943, Thanks @lsternlicht) diff --git a/docs/recipes/czech-language-support/README.md b/docs/recipes/czech-language-support/README.md new file mode 100644 index 0000000000..29972db9fa --- /dev/null +++ b/docs/recipes/czech-language-support/README.md @@ -0,0 +1,183 @@ +# Czech Language Support + +> **Domain**: Finance, Legal, Public Administration, General +> **Data Type**: Czech-language free text (contracts, bank correspondence, ID documents, customer communication) +> **Goal**: Detect Czech PII and sensitive identifiers using Presidio's built-in Czech recognizers with a bilingual spaCy NLP engine + +## Overview + +**Domain**: Finance / Legal / Public Administration +**Data Type**: Czech-language documents +**Goal**: Detect Czech PII — including the rodné číslo (birth number), domestic +bank account numbers, official document numbers and Czech-notation dates — +alongside the English model so that bilingual (EN + CS) documents are +handled correctly. + +This recipe provides: + +- `spacy_en_cs.yaml` — a ready-to-use NLP engine configuration that loads both + `en_core_web_lg` and the multilingual `xx_ent_wiki_sm` (spaCy publishes no + Czech-specific pretrained pipeline) +- An overview of all Czech-specific recognizers available in Presidio + +## Quick Start + +### Prerequisites + +```bash +pip install presidio-analyzer presidio-anonymizer +python -m spacy download en_core_web_lg +python -m spacy download xx_ent_wiki_sm +``` + +### Sample Data + +```python +sample_text = ( + "Jmenuji se Jan Novák a narodil jsem se dne 12. dubna 1985. " + "Moje rodné číslo je 850412/0003. " + "Kontaktovat mě můžete na telefonním čísle +420 601 234 567. " + "Moje číslo občanského průkazu je 123456789 " + "a číslo cestovního pasu je AB123456. " + "Bankovní účet mám vedený pod číslem 19-123456789/0800. " + "IBAN účtu je CZ6508000000192000145399." +) +``` + +### Basic Configuration + +```python +from presidio_analyzer import AnalyzerEngine +from presidio_analyzer.nlp_engine import NlpEngineProvider +from presidio_analyzer.predefined_recognizers import ( + CzBankAccountRecognizer, + CzBirthNumberRecognizer, + CzDateRecognizer, + CzDriverLicenseRecognizer, + CzIdCardRecognizer, + CzPassportRecognizer, +) +from presidio_anonymizer import AnonymizerEngine + +# Load the bilingual EN + CS spaCy configuration +provider = NlpEngineProvider(conf_file="spacy_en_cs.yaml") +nlp_engine = provider.create_engine() + +analyzer = AnalyzerEngine( + nlp_engine=nlp_engine, + supported_languages=["en", "cs"], +) + +# The Czech recognizers ship disabled by default (upstream convention for +# non-default languages); register them for the Czech pipeline explicitly. +for recognizer in ( + CzBirthNumberRecognizer(), + CzBankAccountRecognizer(), + CzIdCardRecognizer(), + CzPassportRecognizer(), + CzDriverLicenseRecognizer(), + CzDateRecognizer(), +): + analyzer.registry.add_recognizer(recognizer) + +anonymizer = AnonymizerEngine() + +# Analyze Czech text +results = analyzer.analyze(text=sample_text, language="cs") +anonymized = anonymizer.anonymize(text=sample_text, analyzer_results=results) + +print(anonymized.text) +``` + +Alternatively, enable the recognizers declaratively by copying +`default_recognizers.yaml` and flipping `enabled: true` on the `Cz*` entries. + +## Approach + +Presidio ships pattern-based recognizers for 5 Czech entity types plus Czech +`DATE_TIME` coverage (see table below). Each recognizer targets a single +entity, uses lookaround-anchored regex patterns with base confidence between +0.1 and 0.6, and relies on: + +1. **Context words** (inflected Czech terminology near the match — the + enhancer matches surface forms, so declensions like "účtu", "průkazu" + are listed explicitly) to boost confidence +2. **Check digit validation** where the official specification defines one + (rodné číslo mod 11, bank account weighted mod 11) + +Identifiers that regularly collide with `PHONE_NUMBER` (rodné číslo such as +`850412/0003`, bank accounts such as `19-123456789/0800`) rely on the +combination of a higher base score and context boosting to win score-based +conflict resolution against the phone recognizer's flat 0.4 score. + +The multilingual `xx_ent_wiki_sm` model adds named-entity recognition for +PERSON, LOCATION and ORGANIZATION on top of the pattern recognizers. Czech +dates ("12. dubna 1985") are covered by `CzDateRecognizer` because no Czech +NER model emitting DATE labels is available. + +### Supported Czech Entities + +| Entity | Name | Check digit | +|--------|------|-------------| +| `CZ_BIRTH_NUMBER` | Rodné číslo | ✅ mod 11 (§ 13 z. č. 133/2000 Sb.) | +| `CZ_BANK_ACCOUNT` | Číslo bankovního účtu | ✅ weighted mod 11 (vyhláška ČNB č. 169/2011 Sb.) | +| `CZ_ID_CARD` | Číslo občanského průkazu | – | +| `CZ_PASSPORT` | Číslo cestovního pasu | – | +| `CZ_DRIVER_LICENSE` | Číslo řidičského průkazu | – | +| `DATE_TIME` | Czech date notation (via `CzDateRecognizer`) | – | + +## Results + +Formal evaluation against a labelled Czech dataset has not yet been performed. +To benchmark this recipe follow the [Presidio Research evaluation workflow](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb): + +1. Generate synthetic Czech text with the [data generator](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) +2. Configure the analyzer with `spacy_en_cs.yaml` +3. Run the evaluator and report precision / recall / F₂ / latency + +**Precision**: TBD +**Recall**: TBD +**F₂ Score**: TBD +**Latency**: TBD + +### Key Findings + +- Recognizers with check digit validation (rodné číslo, bank account) achieve + very low false-positive rates on checksum-valid inputs, and deliberately + keep checksum-failing but date-plausible matches at the base pattern score: + many circulating rodná čísla predate the strict mod 11 rule, so context + words decide those cases instead of a hard reject. +- Recognizers without a checksum (ID card, passport, driving licence) + rely heavily on context words; setting `score_threshold=0.4` filters + context-free digit strings effectively. +- Czech legal citations (`262/2006 Sb.`, `89/2012 Sb.`) share the `NNN/YYYY` + shape with prefix-less bank accounts and are ubiquitous in legal text; the + bank account recognizer invalidates prefix-less matches with a numerator of + up to 3 digits and a year-like 1900–2099 "bank code", and never promotes a + checksum pass over a year-like code the ČNB did not issue. +- `xx_ent_wiki_sm` ships without a lemmatizer, so the context enhancer + compares surface forms. The bundled context word lists therefore contain + the inflected forms that actually occur next to each identifier. + +## Tips for Others + +- **Set `score_threshold`** to 0.4–0.5 for production use to filter out + low-confidence pattern-only matches from context-free digit strings. +- **Use the `entities` parameter** to limit detection to the entity types + relevant to your domain (e.g. only `CZ_BIRTH_NUMBER` and `CZ_BANK_ACCOUNT` + in banking correspondence). +- **Higher-quality Czech NER**: for better PERSON/LOCATION recall, switch to + the `TransformersNlpEngine` with a Czech NER model from the HuggingFace hub + (e.g. [`richielo/small-e-czech-finetuned-ner-wikiann`](https://huggingface.co/richielo/small-e-czech-finetuned-ner-wikiann), + a Czech ELECTRA model fine-tuned on WikiANN) and keep the same + `model_to_presidio_entity_mapping`. Stanza currently provides no Czech NER + package either, so transformers is the recommended upgrade path. +- **Full street addresses** ("Václavské náměstí 123/45, 110 00 Praha 1") are + only as good as the NER model's LOCATION spans; generic models often split + them into street and city fragments. Add a custom address recognizer if + your domain requires whole-address redaction. + +--- + +**Author**: Data Privacy Stack contributors +**Date**: 2026-07-11 diff --git a/docs/recipes/czech-language-support/spacy_en_cs.yaml b/docs/recipes/czech-language-support/spacy_en_cs.yaml new file mode 100644 index 0000000000..3f8e031075 --- /dev/null +++ b/docs/recipes/czech-language-support/spacy_en_cs.yaml @@ -0,0 +1,30 @@ +nlp_engine_name: spacy +models: + - + lang_code: en + model_name: en_core_web_lg + - + # spaCy publishes no Czech-specific pretrained pipeline; the official + # multilingual WikiNER model provides PER/LOC/ORG NER for Czech text. + # See the recipe README for a higher-quality transformers alternative. + lang_code: cs + model_name: xx_ent_wiki_sm + +ner_model_configuration: + model_to_presidio_entity_mapping: + PER: PERSON + PERSON: PERSON + NORP: NRP + FAC: LOCATION + LOC: LOCATION + LOCATION: LOCATION + GPE: LOCATION + ORG: ORGANIZATION + ORGANIZATION: ORGANIZATION + DATE: DATE_TIME + TIME: DATE_TIME + + low_confidence_score_multiplier: 0.4 + low_score_entity_names: + - ORG + - ORGANIZATION diff --git a/docs/supported_entities.md b/docs/supported_entities.md index 4273461a41..cb7b2facc7 100644 --- a/docs/supported_entities.md +++ b/docs/supported_entities.md @@ -168,6 +168,19 @@ For more information, refer to the [adding new recognizers documentation](analyz | DE_HANDELSREGISTER | German Handelsregisternummer: commercial register number with HRA (sole traders / partnerships) or HRB (corporations) prefix followed by 1–6 digits. HRA entries directly identify natural persons (sole traders). Legal basis: §§ 9, 14 HGB, HRV. | Pattern match and context | | DE_PLZ | German Postleitzahl (postal code): 5-digit code in the range 01001–99998. Constitutes personal data in combination with other address fields (DSGVO Art. 4 Nr. 1). **High false-positive risk** – only reliable with address-context words present; base confidence is 0.05. Legal basis: DSGVO Art. 4 Nr. 1. | Pattern match and context (context required for actionable results) | +### Czechia + +| Entity Type | Description | Detection Method | +| --- | --- | --- | +| CZ_BIRTH_NUMBER | Czech rodné číslo (birth number): the national identification number encoding date of birth and sex, formatted YYMMDD/SSS (until 1953) or YYMMDD/SSSS (since 1954), with or without the slash. Legal basis: § 13 zákona č. 133/2000 Sb. | Pattern match, context and checksum (mod 11, incl. the 1954–1985 historical exception) | +| CZ_BANK_ACCOUNT | Czech domestic bank account number in the national format `[prefix-]number/bank code` (e.g. 19-2000145399/0800). Legal basis: vyhláška ČNB č. 169/2011 Sb. | Pattern match, context and checksum (weighted mod 11 on prefix and account number) | +| CZ_ID_CARD | Czech občanský průkaz (identity card) number: 9 digits on machine readable cards and the current eOP. **Context words required for actionable results** — base confidence is 0.2. Legal basis: zákon č. 269/2021 Sb. | Pattern match and context | +| CZ_PASSPORT | Czech cestovní pas (passport) number: 2 letters + 6–7 digits (older series) or 7–8 digits (biometric series). Legal basis: zákon č. 329/1999 Sb. | Pattern match and context | +| CZ_DRIVER_LICENSE | Czech řidičský průkaz (driving licence) number: 2 letters + 6–9 digits. Legal basis: § 103 zákona č. 361/2000 Sb. | Pattern match and context | + +Czech-language pipelines additionally get `DATE_TIME` coverage for Czech date +notation ("12. dubna 1985", "12. 4. 1985") through the `CzDateRecognizer`. + ### Medical / Clinical Detected using the `MedicalNERRecognizer` (requires the `transformers` extra). Uses the [blaze999/Medical-NER](https://huggingface.co/blaze999/Medical-NER) model by default. diff --git a/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml b/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml index 0ad50b1603..3c1b481ca6 100644 --- a/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml +++ b/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml @@ -481,6 +481,49 @@ recognizers: enabled: false country_code: de + # Czechia recognizers (disabled by default, enable for Czech-language pipelines) + - name: CzBirthNumberRecognizer + supported_languages: + - cs + type: predefined + enabled: false + country_code: cz + + - name: CzBankAccountRecognizer + supported_languages: + - cs + type: predefined + enabled: false + country_code: cz + + - name: CzIdCardRecognizer + supported_languages: + - cs + type: predefined + enabled: false + country_code: cz + + - name: CzPassportRecognizer + supported_languages: + - cs + type: predefined + enabled: false + country_code: cz + + - name: CzDriverLicenseRecognizer + supported_languages: + - cs + type: predefined + enabled: false + country_code: cz + + - name: CzDateRecognizer + supported_languages: + - cs + type: predefined + enabled: false + country_code: cz + - name: BasicLangExtractRecognizer supported_languages: - en diff --git a/presidio-analyzer/presidio_analyzer/predefined_recognizers/__init__.py b/presidio-analyzer/presidio_analyzer/predefined_recognizers/__init__.py index 2823ff035d..f614c9dbdf 100644 --- a/presidio-analyzer/presidio_analyzer/predefined_recognizers/__init__.py +++ b/presidio-analyzer/presidio_analyzer/predefined_recognizers/__init__.py @@ -13,6 +13,20 @@ # Canada recognizers from .country_specific.canada.ca_sin_recognizer import CaSinRecognizer +# Czechia recognizers +from .country_specific.czechia.cz_bank_account_recognizer import ( + CzBankAccountRecognizer, +) +from .country_specific.czechia.cz_birth_number_recognizer import ( + CzBirthNumberRecognizer, +) +from .country_specific.czechia.cz_date_recognizer import CzDateRecognizer +from .country_specific.czechia.cz_driver_license_recognizer import ( + CzDriverLicenseRecognizer, +) +from .country_specific.czechia.cz_id_card_recognizer import CzIdCardRecognizer +from .country_specific.czechia.cz_passport_recognizer import CzPassportRecognizer + # Finland recognizers from .country_specific.finland.fi_personal_identity_code_recognizer import ( FiPersonalIdentityCodeRecognizer, @@ -256,6 +270,13 @@ "NgNinRecognizer", "NgVehicleRegistrationRecognizer", "MedicalNERRecognizer", + # Czechia recognizers + "CzBirthNumberRecognizer", + "CzBankAccountRecognizer", + "CzDateRecognizer", + "CzDriverLicenseRecognizer", + "CzIdCardRecognizer", + "CzPassportRecognizer", # Germany recognizers "DeTaxIdRecognizer", "DeTaxNumberRecognizer", diff --git a/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/__init__.py b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/__init__.py new file mode 100644 index 0000000000..e3a74571e7 --- /dev/null +++ b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/__init__.py @@ -0,0 +1 @@ +"""Czechia-specific recognizers package.""" diff --git a/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_bank_account_recognizer.py b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_bank_account_recognizer.py new file mode 100644 index 0000000000..b0896cfbec --- /dev/null +++ b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_bank_account_recognizer.py @@ -0,0 +1,190 @@ +from typing import List, Optional + +from presidio_analyzer import Pattern, PatternRecognizer + + +class CzBankAccountRecognizer(PatternRecognizer): + """ + Recognizes Czech domestic bank account numbers. + + Czech account numbers use the national format + ``[prefix-]number/bank_code``: an optional prefix of up to 6 digits, + an account number of 2-10 digits and a mandatory 4-digit bank code + (e.g. 0800 Česká spořitelna, 0100 Komerční banka). + + Legal basis: vyhláška ČNB č. 169/2011 Sb., o pravidlech tvorby čísla + účtu v platebním styku. Data protection: GDPR Art. 4 Nr. 1, + zákon č. 110/2019 Sb. + + Format: + - Prefix (optional): 2-6 digits followed by a hyphen + - Account number: 2-10 digits + - Slash and a 4-digit bank code + + Checksum: both the prefix and the account number carry a weighted + mod 11 check (weights 1, 2, 4, 8, 5, 10, 9, 7, 3, 6 applied from the + rightmost digit). A full checksum pass promotes the match to + MAX_SCORE; a failure keeps the base pattern score so that context + words ("účet", "účtu", ...) drive the final confidence. + + Czech legal citations (zákon č. 262/2006 Sb., 115/2010 Sb., ...) + share the ``NNN/YYYY`` shape with prefix-less accounts and are very + common in legal and public-administration text, so two guards apply: + + - ``invalidate_result`` drops prefix-less matches whose numerator + has at most 3 digits (Czech laws are numbered up to ~500 per + year) and whose "bank code" is a year-like 1900-2099 value — + even when the numerator passes the checksum (262 does). + - A checksum pass with a year-like bank code that the ČNB never + issued is not promoted to MAX_SCORE; it keeps the pattern score. + Codes 2010 (Fio banka), 2060 (Citfin) and + 2070 (TRINITY BANK) fall inside the year range but are real, + so accounts at those banks still validate fully. + + Examples (fictitious): 19-2000145399/0800, 2000145399/0800 + + :param patterns: List of patterns to be used by this recognizer + :param context: List of context words to increase confidence in detection + :param supported_language: Language this recognizer supports + :param supported_entity: The entity this recognizer can detect + """ + + COUNTRY_CODE = "cz" + + # Weights from vyhláška č. 169/2011 Sb., applied right-to-left. + _WEIGHTS = (1, 2, 4, 8, 5, 10, 9, 7, 3, 6) + + # Bank codes issued by the ČNB ("Číselník kódů platebního + # styku") that fall inside the year-like 1900-2099 range: Fio banka, + # Citfin and TRINITY BANK. Any other year-like code makes + # a legal-citation or date reading at least as likely as an account. + _YEAR_LIKE_ISSUED_BANK_CODES = frozenset({"2010", "2060", "2070"}) + + PATTERNS = [ + Pattern( + "Czech bank account (prefix-number/bank code)", + r"(? bool: + total = sum( + int(digit) * weight + for digit, weight in zip(reversed(digits), self._WEIGHTS) + ) + return total % 11 == 0 + + def validate_result(self, pattern_text: str) -> Optional[bool]: + """ + Validate the account number using the ČNB weighted mod 11 check. + + Both the optional prefix and the account number must pass the + weighted checksum for the match to be promoted to MAX_SCORE. A + checksum failure returns ``None`` (keep the pattern score and let + context words decide) because typos and formatted fragments are + common in free text; only structurally impossible values + (all-zero account, bank code 0000) return ``False``. + + :param pattern_text: the text to validate (prefix-number/bank code) + :return: True if all checksums pass, False if structurally + invalid, None to fall back to the pattern score + """ + pattern_text = pattern_text.strip() + + account_part, _, bank_code = pattern_text.partition("/") + if len(bank_code) != 4 or not bank_code.isdigit(): + return False + if bank_code == "0000": + return False + + prefix, _, number = account_part.rpartition("-") + if not number.isdigit() or (prefix and not prefix.isdigit()): + return False + + # An account number of all zeros is never assigned. + if number.lstrip("0") == "": + return False + + if self._mod11(number) and (not prefix or self._mod11(prefix)): + if self._is_unissued_year_like_code(bank_code): + # A checksum-passing numerator over a year-like bank code + # that the ČNB never issued (e.g. 12345/2006) is at least + # as likely a citation or date as an account; keep the + # pattern score and let context words decide instead of + # promoting to MAX_SCORE. + return None + return True + + return None + + def _is_unissued_year_like_code(self, bank_code: str) -> bool: + return ( + 1900 <= int(bank_code) <= 2099 + and bank_code not in self._YEAR_LIKE_ISSUED_BANK_CODES + ) + + def invalidate_result(self, pattern_text: str) -> Optional[bool]: + """ + Invalidate matches that are far more likely dates or legal citations. + + Czech legal references (``262/2006 Sb.``, ``115/2010 Sb.``) and + date fragments (``04/2023``) share the ``N{1,3}/YYYY`` shape with + prefix-less account numbers, and citations are ubiquitous in the + legal/public-administration text this recognizer targets. Czech + laws are numbered with at most three digits per year, so a + prefix-less match whose numerator has up to 3 digits and whose + "bank code" is a year-like 1900-2099 value is treated as a + citation or date — even when the numerator happens to pass the + mod 11 checksum (``262/2006``, the Labour Code, does). Accounts + at banks whose real code falls in that range (e.g. Fio banka, + 2010) are in practice written with longer account numbers, so + they are unaffected. + + :param pattern_text: the text to check + :return: True if the match should be invalidated, None otherwise + """ + account_part, _, bank_code = pattern_text.partition("/") + if ( + "-" not in account_part + and len(account_part) <= 3 + and account_part.isdigit() + and bank_code.isdigit() + and 1900 <= int(bank_code) <= 2099 + ): + return True + return None diff --git a/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_birth_number_recognizer.py b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_birth_number_recognizer.py new file mode 100644 index 0000000000..be0c3f65e0 --- /dev/null +++ b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_birth_number_recognizer.py @@ -0,0 +1,129 @@ +from typing import List, Optional + +from presidio_analyzer import Pattern, PatternRecognizer + + +class CzBirthNumberRecognizer(PatternRecognizer): + """ + Recognizes the Czech birth number (rodné číslo). + + The rodné číslo is the national identification number assigned to every + person registered in the Czech Republic. It encodes the date of birth and + sex of the holder and is the most sensitive Czech personal identifier, + appearing on birth certificates, identity cards and health documents. + + Legal basis: § 13 zákona č. 133/2000 Sb., o evidenci obyvatel a rodných + číslech. Data protection: GDPR Art. 87 (national identification numbers), + zákon č. 110/2019 Sb. + + Format: YYMMDD/SSS (9 digits, issued until 1953) or YYMMDD/SSSS + (10 digits, issued since 1954), optionally written without the slash. + + - YY: year of birth (2 digits) + - MM: month of birth; +50 for women. Since 2004 also +20 (men) and + +70 (women) when the serial pool for a day is exhausted + (valid ranges: 01-12, 21-32, 51-62, 71-82) + - DD: day of birth (01-31) + - SSS(S): serial suffix; 10-digit numbers carry a check digit + + Checksum (10-digit numbers only): the whole number must be divisible + by 11. Numbers issued between 1954 and 1985 may instead satisfy the + historical exception where the first 9 digits mod 11 equal 10 and the + check digit is 0. + + Examples (fictitious): 780123/0107, 855612/0111 + + :param patterns: List of patterns to be used by this recognizer + :param context: List of context words to increase confidence in detection + :param supported_language: Language this recognizer supports + :param supported_entity: The entity this recognizer can detect + """ + + COUNTRY_CODE = "cz" + + # Month field: 01-12 (men), 21-32 (men, post-2004), 51-62 (women), + # 71-82 (women, post-2004). Day field: 01-31. + _MM = r"(?:0[1-9]|1[0-2]|2[1-9]|3[0-2]|5[1-9]|6[0-2]|7[1-9]|8[0-2])" + _DD = r"(?:0[1-9]|[12]\d|3[01])" + + PATTERNS = [ + Pattern( + "Rodné číslo (YYMMDD/SSSS)", + rf"(? Optional[bool]: + """ + Validate the rodné číslo using the official mod 11 rule. + + 10-digit numbers (issued since 1954) must be divisible by 11, or + satisfy the 1954-1985 historical exception (first 9 digits mod 11 + equal 10 with a 0 check digit). A checksum pass promotes the match + to MAX_SCORE. + + A checksum failure returns ``None`` instead of ``False``: the date + part is already enforced by the pattern, 9-digit numbers carry no + check digit at all, and real-world data contains legacy numbers, so + the match keeps its base pattern score and the ContextAwareEnhancer + drives the final confidence via context words ("rodné číslo", ...). + + :param pattern_text: the text to validate (with or without slash) + :return: True if the checksum is valid, False if structurally + impossible, None to fall back to the pattern score + """ + digits = pattern_text.replace("/", "").strip() + + if not digits.isdigit() or len(digits) not in (9, 10): + return False + + # The serial suffix 000 was never assigned. + if digits[6:].lstrip("0") == "": + return False + + if len(digits) == 9: + # Pre-1954 numbers have no check digit. + return None + + if int(digits) % 11 == 0: + return True + + # Historical exception for ~1000 numbers issued 1954-1985. + if int(digits[:9]) % 11 == 10 and digits[9] == "0": + return True + + return None diff --git a/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_date_recognizer.py b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_date_recognizer.py new file mode 100644 index 0000000000..12957d69b4 --- /dev/null +++ b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_date_recognizer.py @@ -0,0 +1,86 @@ +from typing import List, Optional + +from presidio_analyzer import Pattern, PatternRecognizer + + +class CzDateRecognizer(PatternRecognizer): + """ + Recognizes dates written in Czech notation. + + Czech dates are written day-first with an ordinal dot after the day + (and month), either fully numeric ("12. 4. 1985", "12.4.1985") or + with the month name in genitive or nominative form + ("12. dubna 1985"). The generic ``DateRecognizer`` only covers + English/ISO formats, and no Czech NER model emitting DATE labels is + available, so this pattern recognizer provides DATE_TIME coverage + for Czech-language pipelines. + + Emits the standard ``DATE_TIME`` entity so downstream configuration + (anonymizer operators, allow-lists) treats Czech dates like any + other date. + + :param patterns: List of patterns to be used by this recognizer + :param context: List of context words to increase confidence in detection + :param supported_language: Language this recognizer supports + :param supported_entity: The entity this recognizer can detect + """ + + COUNTRY_CODE = "cz" + + # Month names in genitive (as used in dates) and nominative form. + # Longer alternatives come first so e.g. "července" is preferred + # over its prefix "červen". + _MONTHS = ( + "ledna|leden|února|únor|března|březen|dubna|duben|" + "května|květen|července|červenec|června|červen|" + "srpna|srpen|září|října|říjen|listopadu|listopad|" + "prosince|prosinec" + ) + + PATTERNS = [ + Pattern( + "Czech date (d. month yyyy)", + rf"\b(?:0?[1-9]|[12]\d|3[01])\.?\s(?:{_MONTHS})\s(?:\d{{4}})\b", + 0.6, + ), + Pattern( + "Czech date (d. m. yyyy)", + r"\b(?:0?[1-9]|[12]\d|3[01])\.\s?(?:0?[1-9]|1[0-2])\.\s?\d{4}\b", + 0.6, + ), + Pattern( + "Czech date (d. month, no year)", + rf"\b(?:0?[1-9]|[12]\d|3[01])\.?\s(?:{_MONTHS})\b", + 0.2, + ), + ] + + CONTEXT = [ + "datum", + "data", + "dne", + "den", + "narození", + "narozen", + "narozena", + "narodil", + "narodila", + ] + + def __init__( + self, + patterns: Optional[List[Pattern]] = None, + context: Optional[List[str]] = None, + supported_language: str = "cs", + supported_entity: str = "DATE_TIME", + name: Optional[str] = None, + ): + patterns = patterns if patterns else self.PATTERNS + context = context if context else self.CONTEXT + super().__init__( + supported_entity=supported_entity, + patterns=patterns, + context=context, + supported_language=supported_language, + name=name, + ) diff --git a/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_driver_license_recognizer.py b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_driver_license_recognizer.py new file mode 100644 index 0000000000..4381c566bc --- /dev/null +++ b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_driver_license_recognizer.py @@ -0,0 +1,99 @@ +from typing import List, Optional + +from presidio_analyzer import Pattern, PatternRecognizer + + +class CzDriverLicenseRecognizer(PatternRecognizer): + """ + Recognizes Czech driving licence numbers (číslo řidičského průkazu). + + The řidičský průkaz is the Czech driving licence, issued in the + EU-harmonised card format. The document number consists of 2 + uppercase letters followed by 6-9 digits (e.g. ED 994127 on cards, + CZ-prefixed variants appear in registers and exports). + + Legal basis: § 103 zákona č. 361/2000 Sb., o provozu na pozemních + komunikacích; vyhláška č. 31/2001 Sb. EU Directive 2006/126/EC. + Data protection: GDPR Art. 4 Nr. 1, zákon č. 110/2019 Sb. + + Format: 2 uppercase letters + 6-9 digits, optionally separated by a + space. No public check digit algorithm exists, so ``validate_result`` + only drops clearly invalid inputs (lowercase prefix such as ordinary + words before a number, all-zero digits). Final confidence is driven + by context words ("řidičský průkaz", "ŘP", ...) via the + ContextAwareEnhancer. + + Examples (fictitious): ED994127, CZ123456789 + + :param patterns: List of patterns to be used by this recognizer + :param context: List of context words to increase confidence in detection + :param supported_language: Language this recognizer supports + :param supported_entity: The entity this recognizer can detect + """ + + COUNTRY_CODE = "cz" + + PATTERNS = [ + Pattern( + "Řidičský průkaz (2 letters + digits)", + r"(? Optional[bool]: + """ + Validate the driving licence number structurally. + + Patterns are matched case-insensitively (global regex flags), but + real licence numbers are printed in uppercase, so a lowercase + prefix (e.g. the words "na 123456" would otherwise match) is + rejected here. All-zero digit blocks are rejected as well. + + :param pattern_text: the text to validate + :return: False if malformed, None otherwise (keep pattern score, + let context drive confidence) + """ + pattern_text = pattern_text.strip() + + letters = pattern_text[:2] + digits = pattern_text[2:].strip() + + if not letters.isalpha() or not letters.isupper(): + return False + + if not digits.isdigit() or digits.lstrip("0") == "": + return False + + return None diff --git a/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_id_card_recognizer.py b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_id_card_recognizer.py new file mode 100644 index 0000000000..b1cfafe549 --- /dev/null +++ b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_id_card_recognizer.py @@ -0,0 +1,101 @@ +from typing import List, Optional + +from presidio_analyzer import Pattern, PatternRecognizer + + +class CzIdCardRecognizer(PatternRecognizer): + r""" + Recognizes Czech identity card numbers (číslo občanského průkazu). + + The občanský průkaz is the mandatory Czech identity card. Machine + readable cards issued since the early 2000s (including the current + eOP) carry a 9-digit document number printed on the front side and + in the MRZ. + + Legal basis: zákon č. 269/2021 Sb., o občanských průkazech. + Data protection: GDPR Art. 4 Nr. 1, zákon č. 110/2019 Sb. + + Format (9 digits): no publicly documented check digit exists, so + ``validate_result`` cannot give positive evidence of a real document + number; it only drops clearly invalid inputs (all-zero). All + structurally plausible 9-digit inputs return ``None``: the match + keeps its base pattern score (0.2) and the ContextAwareEnhancer + drives the final confidence via context words ("občanský průkaz", + "číslo OP", ...). + + Legacy booklet-style identity cards used letter-prefixed serial + formats that are no longer issued; they are explicitly out of scope. + + Examples (fictitious): 123456789, 987654321 + + :param patterns: List of patterns to be used by this recognizer + :param context: List of context words to increase confidence in detection + :param supported_language: Language this recognizer supports + :param supported_entity: The entity this recognizer can detect + """ + + COUNTRY_CODE = "cz" + + PATTERNS = [ + Pattern( + "Občanský průkaz (9 digits)", + r"(? Optional[bool]: + """ + Validate the identity card number structurally. + + The občanský průkaz number has no publicly documented check digit, + so this method only drops clearly invalid inputs. Final confidence + on valid-shaped numbers is driven by context words via the + ContextAwareEnhancer. + + :param pattern_text: the text to validate (9 digits) + :return: False if malformed or all-zero, None otherwise (keep + pattern score, let context drive confidence) + """ + pattern_text = pattern_text.strip() + + if len(pattern_text) != 9 or not pattern_text.isdigit(): + return False + + if pattern_text == "000000000": + return False + + return None diff --git a/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_passport_recognizer.py b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_passport_recognizer.py new file mode 100644 index 0000000000..58cf3be655 --- /dev/null +++ b/presidio-analyzer/presidio_analyzer/predefined_recognizers/country_specific/czechia/cz_passport_recognizer.py @@ -0,0 +1,105 @@ +from typing import List, Optional + +from presidio_analyzer import Pattern, PatternRecognizer + + +class CzPassportRecognizer(PatternRecognizer): + """ + Recognizes Czech passport numbers (číslo cestovního pasu). + + Czech passports (cestovní pas) are issued by the Ministry of the + Interior. Biometric passports issued since 2006 carry an 8-digit + document number; older series used 7 digits or a 2-letter prefix + followed by 6-7 digits. + + Legal basis: zákon č. 329/1999 Sb., o cestovních dokladech. + Data protection: GDPR Art. 4 Nr. 1, zákon č. 110/2019 Sb. + + Format: + - 2 uppercase letters followed by 6-7 digits (older series), or + - 7-8 digits (current biometric series) + + No public check digit algorithm exists for the printed document + number, so ``validate_result`` only drops clearly invalid inputs + (lowercase letter prefix, all-zero digits). Final confidence is + driven by context words ("cestovní pas", "pas č.", ...) via the + ContextAwareEnhancer. + + Examples (fictitious): AB123456, 39182634 + + :param patterns: List of patterns to be used by this recognizer + :param context: List of context words to increase confidence in detection + :param supported_language: Language this recognizer supports + :param supported_entity: The entity this recognizer can detect + """ + + COUNTRY_CODE = "cz" + + PATTERNS = [ + Pattern( + "Cestovní pas (letter prefix)", + r"(? Optional[bool]: + """ + Validate the passport number structurally. + + Patterns are matched case-insensitively (global regex flags), but + real document numbers are printed in uppercase, so a lowercase + letter prefix (e.g. the words "je 123456" would otherwise match) + is rejected here. All-zero digit blocks are rejected as well. + + :param pattern_text: the text to validate + :return: False if malformed, None otherwise (keep pattern score, + let context drive confidence) + """ + pattern_text = pattern_text.strip() + + letters = "".join(c for c in pattern_text if c.isalpha()) + digits = "".join(c for c in pattern_text if c.isdigit()) + + if letters and not letters.isupper(): + return False + + if not digits or digits.lstrip("0") == "": + return False + + return None diff --git a/presidio-analyzer/tests/test_cz_bank_account_recognizer.py b/presidio-analyzer/tests/test_cz_bank_account_recognizer.py new file mode 100644 index 0000000000..718e6b7a54 --- /dev/null +++ b/presidio-analyzer/tests/test_cz_bank_account_recognizer.py @@ -0,0 +1,124 @@ +""" +Tests for CzBankAccountRecognizer (Czech domestic bank account numbers). + +All test numbers are fictitious. Valid numbers satisfy the weighted mod 11 +check of vyhláška ČNB č. 169/2011 Sb. (weights 1, 2, 4, 8, 5, 10, 9, 7, 3, 6 +applied from the rightmost digit, on both the prefix and the account number). +""" +import pytest + +from tests import assert_result +from presidio_analyzer.predefined_recognizers import CzBankAccountRecognizer + +# Pattern score from CzBankAccountRecognizer.PATTERNS. A failed checksum +# returns None from validate_result, keeping this score so that context +# words decide. +_PATTERN_SCORE = 0.4 + + +@pytest.fixture(scope="module") +def recognizer(): + return CzBankAccountRecognizer() + + +@pytest.fixture(scope="module") +def entities(): + return ["CZ_BANK_ACCOUNT"] + + +@pytest.mark.parametrize( + "text, expected_len, expected_positions, expected_score_fn", + [ + # fmt: off + # Valid prefix and account checksum -> MAX_SCORE + ("19-2000145399/0800", 1, ((0, 18),), lambda max_score: max_score), + ("2000145399/0800", 1, ((0, 15),), lambda max_score: max_score), + ("123456788/0100", 1, ((0, 14),), lambda max_score: max_score), + ("č. ú. 19-2000145399/0800", 1, ((6, 24),), lambda max_score: max_score), + # Account number fails the weighted mod 11 -> pattern score + ("19-123456789/0800", 1, ((0, 17),), lambda max_score: _PATTERN_SCORE), + # Prefix fails the weighted mod 11 -> pattern score + ("18-2000145399/0800", 1, ((0, 18),), lambda max_score: _PATTERN_SCORE), + # Bank code 0000 does not exist -> dropped by validation + ("2000145399/0000", 0, (), None), + # All-zero account number -> dropped by validation + ("000000/0800", 0, (), None), + # Date-like fragments are invalidated (dd/yyyy with 19xx/20xx) + ("04/2023", 0, (), None), + ("31/1999", 0, (), None), + # Czech legal citations (zákon č. NNN/YYYY Sb.) are invalidated, + # even when the numerator passes the mod 11 checksum (262 does) + ("262/2006", 0, (), None), + ("115/2010", 0, (), None), + ("zákon č. 262/2006 Sb., zákoník práce", 0, (), None), + ("89/2012", 0, (), None), + # Accounts at banks whose real ČNB code is year-like (Fio banka, + # 2010) still validate fully: numerators exceed law numbers + ("2000145399/2010", 1, ((0, 15),), lambda max_score: max_score), + # Embedded in a longer digit run -> no match + ("119-123456789/08001", 0, (), None), + # fmt: on + ], +) +def test_when_all_cz_bank_accounts_then_succeed( + text, expected_len, expected_positions, expected_score_fn, + recognizer, entities, max_score, +): + results = recognizer.analyze(text, entities) + assert len(results) == expected_len + for res, (st_pos, fn_pos) in zip(results, expected_positions): + assert_result( + res, entities[0], st_pos, fn_pos, expected_score_fn(max_score) + ) + + +@pytest.mark.parametrize( + "number, expected", + [ + # Valid prefix + account combinations + ("19-2000145399/0800", True), + ("2000145399/0800", True), + ("35-123456788/0300", True), + # Failing account or prefix checksum -> None (context decides) + ("19-123456789/0800", None), + ("18-2000145399/0800", None), + ("850412/0003", None), + # Checksum passes but the year-like "bank code" was never issued + # by the ČNB -> None (no MAX_SCORE promotion) + ("262/2006", None), + ("123456788/2006", None), + # Year-like code actually issued by the ČNB (Fio banka) -> True + ("2000145399/2010", True), + # Structurally impossible -> False + ("2000145399/0000", False), + ("000000/0800", False), + ("2000145399/800", False), + ("2000145399/08000", False), + ], +) +def test_when_cz_bank_account_validated_then_checksum_result_is_correct( + number, expected, recognizer +): + assert recognizer.validate_result(number) == expected + + +@pytest.mark.parametrize( + "number, expected", + [ + # Date-like fragments and legal citations are invalidated + ("04/2023", True), + ("31/1999", True), + ("262/2006", True), + ("115/2010", True), + ("89/2012", True), + ("169/2011", True), + # Real account shapes are not + ("19-2000145399/0800", None), + ("2000145399/0800", None), + ("2345678/2010", None), # long numerator at Fio banka (2010) + ], +) +def test_when_cz_bank_account_invalidated_then_result_is_correct( + number, expected, recognizer +): + assert recognizer.invalidate_result(number) == expected diff --git a/presidio-analyzer/tests/test_cz_birth_number_recognizer.py b/presidio-analyzer/tests/test_cz_birth_number_recognizer.py new file mode 100644 index 0000000000..05097094ca --- /dev/null +++ b/presidio-analyzer/tests/test_cz_birth_number_recognizer.py @@ -0,0 +1,100 @@ +""" +Tests for CzBirthNumberRecognizer (rodné číslo). + +All test numbers are fictitious/generated and do not represent real persons. +Valid numbers satisfy the official mod 11 rule of § 13 zákona č. 133/2000 Sb. +(10-digit numbers divisible by 11, or the 1954-1985 historical exception +where the first 9 digits mod 11 equal 10 and the check digit is 0). +""" +import pytest + +from tests import assert_result +from presidio_analyzer.predefined_recognizers import CzBirthNumberRecognizer + +# Pattern scores from CzBirthNumberRecognizer.PATTERNS. A failed checksum +# returns None from validate_result (see docstring there), so those +# matches keep the base pattern score and context words decide. +_SLASH_SCORE = 0.5 +_NO_SLASH_SCORE = 0.3 + + +@pytest.fixture(scope="module") +def recognizer(): + return CzBirthNumberRecognizer() + + +@pytest.fixture(scope="module") +def entities(): + return ["CZ_BIRTH_NUMBER"] + + +@pytest.mark.parametrize( + "text, expected_len, expected_positions, expected_score_fn", + [ + # fmt: off + # Valid checksum -> MAX_SCORE + ("780123/0107", 1, ((0, 11),), lambda max_score: max_score), + ("855612/0111", 1, ((0, 11),), lambda max_score: max_score), # female month (+50) + ("990131/0111", 1, ((0, 11),), lambda max_score: max_score), + ("Moje rodné číslo je 780123/0107.", 1, ((20, 31),), lambda max_score: max_score), + # Valid checksum without separator -> MAX_SCORE + ("RČ 8556120111 bylo ověřeno.", 1, ((3, 13),), lambda max_score: max_score), + # Historical exception (first 9 digits mod 11 == 10, check digit 0) + ("780123/1010", 1, ((0, 11),), lambda max_score: max_score), + # Date part is plausible but the checksum fails -> pattern score + ("850412/0003", 1, ((0, 11),), lambda max_score: _SLASH_SCORE), + ("8504120003", 1, ((0, 10),), lambda max_score: _NO_SLASH_SCORE), + # 9-digit pre-1954 number (no check digit defined) -> pattern score + ("530101/123", 1, ((0, 10),), lambda max_score: _SLASH_SCORE), + # Invalid month (13) -> no match + ("781323/0107", 0, (), None), + # Invalid day (00) -> no match + ("780100/0107", 0, (), None), + # Serial suffix 000 was never assigned -> dropped by validation + ("780123/000", 0, (), None), + # Bank account shaped strings must not match (month/day impossible) + ("19-123456789/0800", 0, (), None), + # Embedded in a longer digit run -> no match + ("77805041200031", 0, (), None), + # fmt: on + ], +) +def test_when_all_cz_birth_numbers_then_succeed( + text, expected_len, expected_positions, expected_score_fn, + recognizer, entities, max_score, +): + results = recognizer.analyze(text, entities) + assert len(results) == expected_len + for res, (st_pos, fn_pos) in zip(results, expected_positions): + assert_result( + res, entities[0], st_pos, fn_pos, expected_score_fn(max_score) + ) + + +@pytest.mark.parametrize( + "number, expected", + [ + # Valid 10-digit numbers (divisible by 11) + ("7801230107", True), + ("780123/0107", True), + ("8556120111", True), + # Historical exception (1954-1985) + ("7801231010", True), + # Plausible but failing checksum -> None (context decides) + ("8504120003", None), + ("850412/0003", None), + # 9-digit pre-1954 numbers carry no check digit -> None + ("530101123", None), + # Serial suffix 000 -> False + ("780123000", False), + ("780123/000", False), + # Wrong length / non-numeric -> False + ("78012301", False), + ("78012301070", False), + ("78O123O107", False), + ], +) +def test_when_cz_birth_number_validated_then_checksum_result_is_correct( + number, expected, recognizer +): + assert recognizer.validate_result(number) == expected diff --git a/presidio-analyzer/tests/test_cz_date_recognizer.py b/presidio-analyzer/tests/test_cz_date_recognizer.py new file mode 100644 index 0000000000..3fc49f8e35 --- /dev/null +++ b/presidio-analyzer/tests/test_cz_date_recognizer.py @@ -0,0 +1,59 @@ +""" +Tests for CzDateRecognizer (Czech date notation, DATE_TIME entity). + +Covers numeric day-first dates with ordinal dots ("12. 4. 1985", +"12.4.1985") and dates with Czech month names in genitive or nominative +form ("12. dubna 1985"). Emits the standard DATE_TIME entity. +""" +import pytest + +from tests import assert_result +from presidio_analyzer.predefined_recognizers import CzDateRecognizer + +# Pattern scores from CzDateRecognizer.PATTERNS. +_FULL_DATE_SCORE = 0.6 +_NO_YEAR_SCORE = 0.2 + + +@pytest.fixture(scope="module") +def recognizer(): + return CzDateRecognizer() + + +@pytest.fixture(scope="module") +def entities(): + return ["DATE_TIME"] + + +@pytest.mark.parametrize( + "text, expected_len, expected_positions, expected_score", + [ + # fmt: off + # Textual month (genitive, as used in running text) + ("12. dubna 1985", 1, ((0, 14),), _FULL_DATE_SCORE), + ("Narodil se 1. ledna 2000.", 1, ((11, 24),), _FULL_DATE_SCORE), + ("5. července 2020", 1, ((0, 16),), _FULL_DATE_SCORE), + ("15. září 2021", 1, ((0, 13),), _FULL_DATE_SCORE), + ("31. prosince 1999", 1, ((0, 17),), _FULL_DATE_SCORE), + # Numeric day-first with ordinal dots + ("12. 4. 1985", 1, ((0, 11),), _FULL_DATE_SCORE), + ("12.4.1985", 1, ((0, 9),), _FULL_DATE_SCORE), + ("3. 12. 2023", 1, ((0, 11),), _FULL_DATE_SCORE), + # Day + month without a year (weak evidence) + ("12. dubna", 1, ((0, 9),), _NO_YEAR_SCORE), + # Impossible day or month -> no match + ("32. dubna 1985", 0, (), None), + ("12. 13. 1985", 0, (), None), + # English/ISO formats are left to the generic DateRecognizer + ("2023-04-12", 0, (), None), + # fmt: on + ], +) +def test_when_all_cz_dates_then_succeed( + text, expected_len, expected_positions, expected_score, + recognizer, entities, +): + results = recognizer.analyze(text, entities) + assert len(results) == expected_len + for res, (st_pos, fn_pos) in zip(results, expected_positions): + assert_result(res, entities[0], st_pos, fn_pos, expected_score) diff --git a/presidio-analyzer/tests/test_cz_driver_license_recognizer.py b/presidio-analyzer/tests/test_cz_driver_license_recognizer.py new file mode 100644 index 0000000000..475a58214a --- /dev/null +++ b/presidio-analyzer/tests/test_cz_driver_license_recognizer.py @@ -0,0 +1,78 @@ +""" +Tests for CzDriverLicenseRecognizer (číslo řidičského průkazu). + +Format: 2 uppercase letters + 6-9 digits, optionally space-separated. +No public check digit algorithm exists, so validate_result only rejects +clearly-invalid inputs (lowercase prefix such as ordinary words before a +number, all-zero digits). Context words drive final confidence. + +All test numbers are fictitious. +""" +import pytest + +from tests import assert_result +from presidio_analyzer.predefined_recognizers import CzDriverLicenseRecognizer + +# Pattern score from CzDriverLicenseRecognizer.PATTERNS. +_PATTERN_SCORE = 0.3 + + +@pytest.fixture(scope="module") +def recognizer(): + return CzDriverLicenseRecognizer() + + +@pytest.fixture(scope="module") +def entities(): + return ["CZ_DRIVER_LICENSE"] + + +@pytest.mark.parametrize( + "text, expected_len, expected_positions", + [ + # fmt: off + ("ED994127", 1, ((0, 8),)), + ("ED 994127", 1, ((0, 9),)), + ("CZ123456789", 1, ((0, 11),)), + ("Řidičský průkaz č. CZ123456789", 1, ((19, 30),)), + # Lowercase prefix (ordinary Czech words) -> dropped by validation + ("je 123456", 0, ()), + ("na 1234567", 0, ()), + ("cz123456789", 0, ()), + # All-zero digits -> dropped by validation + ("CZ000000", 0, ()), + # Too few digits -> no match + ("ED 99412", 0, ()), + # Embedded in a longer run -> no match + ("CZ1234567890", 0, ()), + # Glued to a slash/hyphen-delimited identifier -> no match + ("19-CZ123456789", 0, ()), + ("spis/CZ123456789", 0, ()), + # fmt: on + ], +) +def test_when_all_cz_driver_licenses_then_succeed( + text, expected_len, expected_positions, recognizer, entities +): + results = recognizer.analyze(text, entities) + assert len(results) == expected_len + for res, (st_pos, fn_pos) in zip(results, expected_positions): + assert_result(res, entities[0], st_pos, fn_pos, _PATTERN_SCORE) + + +@pytest.mark.parametrize( + "number, expected", + [ + ("ED994127", None), + ("ED 994127", None), + ("CZ123456789", None), + ("je 123456", False), + ("ed994127", False), + ("CZ000000", False), + ("E1994127", False), + ], +) +def test_when_cz_driver_license_validated_then_result_is_correct( + number, expected, recognizer +): + assert recognizer.validate_result(number) == expected diff --git a/presidio-analyzer/tests/test_cz_id_card_recognizer.py b/presidio-analyzer/tests/test_cz_id_card_recognizer.py new file mode 100644 index 0000000000..e04ce65651 --- /dev/null +++ b/presidio-analyzer/tests/test_cz_id_card_recognizer.py @@ -0,0 +1,77 @@ +""" +Tests for CzIdCardRecognizer (číslo občanského průkazu). + +Format: 9 digits (machine readable cards / eOP). No publicly documented +check digit exists, so validate_result only rejects clearly-invalid inputs +(wrong length, non-digit, all-zero) and returns None for every other +9-digit input. Context words drive final confidence via the enhancer. + +All test numbers are fictitious. +""" +import pytest + +from tests import assert_result +from presidio_analyzer.predefined_recognizers import CzIdCardRecognizer + +# Pattern score from CzIdCardRecognizer.PATTERNS. validate_result returns +# None on all structurally-valid inputs, so matches keep this score. +_PATTERN_SCORE = 0.2 + + +@pytest.fixture(scope="module") +def recognizer(): + return CzIdCardRecognizer() + + +@pytest.fixture(scope="module") +def entities(): + return ["CZ_ID_CARD"] + + +@pytest.mark.parametrize( + "text, expected_len, expected_positions", + [ + # fmt: off + # Structurally valid -> pattern score (context boosts in pipeline) + ("123456789", 1, ((0, 9),)), + ("987654321", 1, ((0, 9),)), + ("Číslo OP: 123456789", 1, ((10, 19),)), + ("číslo občanského průkazu je 123456789.", 1, ((28, 37),)), + # Wrong length -> no match + ("12345678", 0, ()), + ("1234567890", 0, ()), + # All-zero -> dropped by validation + ("000000000", 0, ()), + # Part of a bank account or letter-prefixed identifier -> no match + ("19-123456789", 0, ()), + ("123456789/0800", 0, ()), + ("CZ123456789", 0, ()), + # Decimal fraction -> no match + ("3.123456789", 0, ()), + # fmt: on + ], +) +def test_when_all_cz_id_cards_then_succeed( + text, expected_len, expected_positions, recognizer, entities +): + results = recognizer.analyze(text, entities) + assert len(results) == expected_len + for res, (st_pos, fn_pos) in zip(results, expected_positions): + assert_result(res, entities[0], st_pos, fn_pos, _PATTERN_SCORE) + + +@pytest.mark.parametrize( + "number, expected", + [ + ("123456789", None), + ("987654321", None), + ("000000000", False), + ("12345678", False), + ("1234567890", False), + ("12345678A", False), + ], +) +def test_when_cz_id_card_validated_then_result_is_correct( + number, expected, recognizer +): + assert recognizer.validate_result(number) == expected diff --git a/presidio-analyzer/tests/test_cz_passport_recognizer.py b/presidio-analyzer/tests/test_cz_passport_recognizer.py new file mode 100644 index 0000000000..02b43c3a59 --- /dev/null +++ b/presidio-analyzer/tests/test_cz_passport_recognizer.py @@ -0,0 +1,73 @@ +""" +Tests for CzPassportRecognizer (číslo cestovního pasu). + +Format: 2 uppercase letters + 6-7 digits (older series) or 7-8 digits +(current biometric series). No public check digit algorithm exists, so +validate_result only rejects clearly-invalid inputs (lowercase letter +prefix, all-zero digits). Context words drive final confidence. + +All test numbers are fictitious. +""" +import pytest + +from tests import assert_result +from presidio_analyzer.predefined_recognizers import CzPassportRecognizer + +# Pattern scores from CzPassportRecognizer.PATTERNS. +_LETTER_SCORE = 0.3 +_DIGITS_SCORE = 0.1 + + +@pytest.mark.parametrize( + "text, expected_len, expected_positions, expected_score", + [ + # fmt: off + # Letter-prefixed series + ("AB123456", 1, ((0, 8),), _LETTER_SCORE), + ("AB 123456", 1, ((0, 9),), _LETTER_SCORE), + ("AB1234567", 1, ((0, 9),), _LETTER_SCORE), + ("číslo cestovního pasu je AB123456.", 1, ((25, 33),), _LETTER_SCORE), + # Digit-only biometric series (low base score, context-driven) + ("39182634", 1, ((0, 8),), _DIGITS_SCORE), + ("cestovní pas č. 39182634", 1, ((16, 24),), _DIGITS_SCORE), + # Lowercase prefix (ordinary words) -> dropped by validation + ("ab123456", 0, (), None), + ("je 123456", 0, (), None), + # All-zero digits -> dropped by validation + ("AB000000", 0, (), None), + # Too many digits for the letter-prefixed series -> no match + ("AB123456789", 0, (), None), + # Embedded in a longer digit run -> no match + ("391826341234", 0, (), None), + # Glued to a slash/hyphen-delimited identifier -> no match + ("č. j. 123-AB123456", 0, (), None), + ("spis/AB123456", 0, (), None), + # fmt: on + ], +) +def test_when_all_cz_passports_then_succeed( + text, expected_len, expected_positions, expected_score +): + recognizer = CzPassportRecognizer() + entities = ["CZ_PASSPORT"] + results = recognizer.analyze(text, entities) + assert len(results) == expected_len + for res, (st_pos, fn_pos) in zip(results, expected_positions): + assert_result(res, entities[0], st_pos, fn_pos, expected_score) + + +@pytest.mark.parametrize( + "number, expected", + [ + ("AB123456", None), + ("AB 1234567", None), + ("39182634", None), + ("ab123456", False), + ("Ab123456", False), + ("AB000000", False), + ("0000000", False), + ], +) +def test_when_cz_passport_validated_then_result_is_correct(number, expected): + recognizer = CzPassportRecognizer() + assert recognizer.validate_result(number) == expected diff --git a/presidio-analyzer/tests/test_czech_language_support.py b/presidio-analyzer/tests/test_czech_language_support.py new file mode 100644 index 0000000000..857cdb2c97 --- /dev/null +++ b/presidio-analyzer/tests/test_czech_language_support.py @@ -0,0 +1,312 @@ +""" +End-to-end tests for Czech (cs) language support. + +Reproduces the two historical failure modes and verifies both are fixed: + +(a) Standard entities (PERSON, LOCATION) were not detected because no NER + model was configured for Czech, and DATE_TIME was not detected because + no recognizer covered Czech date notation. NER output is simulated + here through precomputed ``NlpArtifacts`` (labels already mapped to + Presidio entities, exactly as the NLP engine configured in + ``docs/recipes/czech-language-support/spacy_en_cs.yaml`` produces + them), so the test runs without downloading a model. DATE_TIME comes + from the real ``CzDateRecognizer``. + +(b) Czech identifiers collided with PHONE_NUMBER: the phone recognizer + used to win on ``850412/0003`` (rodné číslo) and on part of + ``19-123456789/0800`` (bank account). The Czech recognizers use + context words and validation to outscore PHONE_NUMBER so that the + anonymizer's default conflict resolution (identical spans: higher + score wins; nested spans: the containing span wins) picks the + Czech entity. + +All personal data in the test text is fictitious. +""" +import copy +from typing import Dict, List, Tuple + +import pytest +import spacy + +from presidio_analyzer import AnalyzerEngine, RecognizerRegistry, RecognizerResult +from presidio_analyzer.nlp_engine import NlpArtifacts +from presidio_analyzer.predefined_recognizers import ( + CzBankAccountRecognizer, + CzBirthNumberRecognizer, + CzDateRecognizer, + CzDriverLicenseRecognizer, + CzIdCardRecognizer, + CzPassportRecognizer, + DateRecognizer, + EmailRecognizer, + IbanRecognizer, + PhoneRecognizer, + SpacyRecognizer, +) +from tests.mocks import NlpEngineMock + +TEXT = ( + "Jmenuji se Jan Novák a narodil jsem se dne 12. dubna 1985. " + "Moje rodné číslo je 850412/0003. " + "Bydlím na adrese Václavské náměstí 123/45, 110 00 Praha 1. " + "Kontaktovat mě můžete na telefonním čísle +420 601 234 567 " + "nebo prostřednictvím e-mailové adresy jan.novak@example.cz. " + "Moje číslo občanského průkazu je 123456789 a číslo cestovního pasu je " + "AB123456. Řidičský průkaz má číslo CZ123456789. " + "Bankovní účet mám vedený pod číslem 19-123456789/0800. " + "IBAN účtu je CZ6508000000192000145399 a BIC/SWIFT kód banky je GIBACZPX." +) + +EXPECTED_ANONYMIZED = ( + "Jmenuji se a narodil jsem se dne . " + "Moje rodné číslo je . " + "Bydlím na adrese . " + "Kontaktovat mě můžete na telefonním čísle " + "nebo prostřednictvím e-mailové adresy . " + "Moje číslo občanského průkazu je a číslo cestovního pasu je " + ". Řidičský průkaz má číslo . " + "Bankovní účet mám vedený pod číslem . " + "IBAN účtu je a BIC/SWIFT kód banky je GIBACZPX." +) + +# What the Czech NER model returns for this text, with labels already +# mapped to Presidio entities by the NLP engine (see the ner_model +# configuration in docs/recipes/czech-language-support/spacy_en_cs.yaml). +NER_SPANS = [ + ("Jan Novák", "PERSON"), + ("Václavské náměstí 123/45, 110 00 Praha 1", "LOCATION"), +] + +# Field-by-field expectations from the analyzer (exact span and entity). +EXPECTED_DETECTIONS = [ + ("Jan Novák", "PERSON"), + ("12. dubna 1985", "DATE_TIME"), + ("850412/0003", "CZ_BIRTH_NUMBER"), + ("Václavské náměstí 123/45, 110 00 Praha 1", "LOCATION"), + ("+420 601 234 567", "PHONE_NUMBER"), + ("jan.novak@example.cz", "EMAIL_ADDRESS"), + ("123456789", "CZ_ID_CARD"), + ("AB123456", "CZ_PASSPORT"), + ("CZ123456789", "CZ_DRIVER_LICENSE"), + ("19-123456789/0800", "CZ_BANK_ACCOUNT"), + ("CZ6508000000192000145399", "IBAN_CODE"), +] + + +def _czech_nlp_artifacts(text: str) -> NlpArtifacts: + """Build NlpArtifacts with real Czech tokenization and simulated NER.""" + nlp = spacy.blank("cs") + doc = nlp(text) + spans = [] + for span_text, label in NER_SPANS: + start = text.index(span_text) + spans.append( + doc.char_span( + start, start + len(span_text), label=label, alignment_mode="expand" + ) + ) + doc.ents = spans + return NlpArtifacts( + entities=doc.ents, + tokens=doc, + tokens_indices=[token.idx for token in doc], + lemmas=[token.text for token in doc], + nlp_engine=NlpEngineMock(), + language="cs", + ) + + +def _czech_recognizers() -> List: + """The recognizer set a Czech pipeline registers for language cs.""" + return [ + CzBirthNumberRecognizer(), + CzBankAccountRecognizer(), + CzIdCardRecognizer(), + CzPassportRecognizer(), + CzDriverLicenseRecognizer(), + CzDateRecognizer(), + # Generic recognizers are instantiated per registry language by + # the registry provider; mirrored here explicitly. + PhoneRecognizer(supported_language="cs"), + EmailRecognizer(supported_language="cs"), + IbanRecognizer(supported_language="cs"), + DateRecognizer(supported_language="cs"), + SpacyRecognizer(supported_language="cs"), + ] + + +def _resolve_conflicts(results: List[RecognizerResult]) -> List[RecognizerResult]: + """ + Mirror presidio-anonymizer's default conflict resolution. + + Replicates ``ConflictResolutionStrategy.MERGE_SIMILAR_OR_CONTAINED`` + (``AnonymizerEngine._remove_conflicts_and_get_text_manipulation_data``): + intersecting results of the same entity type are first merged into a + single span with the maximum score; then a result is dropped when + another result covers the same indices with an equal-or-higher + score, or fully contains it. Partial intersections between + different entity types are left untouched, exactly like the + anonymizer default — ``_anonymize`` asserts none remain, so a + regression that introduces one fails loudly instead of being + silently normalized away. + """ + results = copy.deepcopy(results) + + # Pass 1: merge intersecting results of the same entity type. + merged: List[RecognizerResult] = [] + other_elements = results.copy() + for result in results: + other_elements.remove(result) + merge_target = next( + ( + other + for other in other_elements + if other.entity_type == result.entity_type + and result.intersects(other) > 0 + ), + None, + ) + if merge_target is None: + other_elements.append(result) + merged.append(result) + else: + merge_target.start = min(result.start, merge_target.start) + merge_target.end = max(result.end, merge_target.end) + merge_target.score = max(result.score, merge_target.score) + + # Pass 2: drop results whose indices equal another's with an + # equal-or-higher score, and results fully contained in another. + unique: List[RecognizerResult] = [] + other_elements = merged.copy() + for result in merged: + other_elements.remove(result) + if not any(result.has_conflict(other) for other in other_elements): + other_elements.append(result) + unique.append(result) + + return sorted(unique, key=lambda r: r.start) + + +def _anonymize(text: str, resolved: List[RecognizerResult]) -> str: + """Replace resolved entities with numbered placeholders.""" + for previous, current in zip(resolved, resolved[1:]): + assert previous.end <= current.start, ( + f"Partial intersection survived conflict resolution " + f"([{previous}] vs [{current}]); the anonymizer's default " + f"strategy would leave both in place and the replacement " + f"output would be ill-defined" + ) + counters: Dict[str, int] = {} + pieces: List[str] = [] + cursor = 0 + for result in resolved: + counters[result.entity_type] = counters.get(result.entity_type, 0) + 1 + pieces.append(text[cursor : result.start]) + pieces.append(f"<{result.entity_type}_{counters[result.entity_type]}>") + cursor = result.end + pieces.append(text[cursor:]) + return "".join(pieces) + + +def _span_of(snippet: str) -> Tuple[int, int]: + start = TEXT.index(snippet) + return start, start + len(snippet) + + +@pytest.fixture(scope="module") +def czech_analyzer() -> AnalyzerEngine: + registry = RecognizerRegistry( + recognizers=_czech_recognizers(), supported_languages=["cs"] + ) + return AnalyzerEngine( + registry=registry, + nlp_engine=NlpEngineMock(nlp_artifacts=_czech_nlp_artifacts(TEXT)), + supported_languages=["cs"], + ) + + +@pytest.fixture(scope="module") +def analyzer_results(czech_analyzer) -> List[RecognizerResult]: + return czech_analyzer.analyze(TEXT, language="cs") + + +def test_when_no_czech_support_then_czech_entities_are_missed(): + """Before-state: without Czech support only generic entities appear. + + With only the generic recognizers (the pre-change setup), none of + the Czech entities, no PERSON/LOCATION and no Czech DATE_TIME can be + detected — and ``850412/0003`` / parts of ``19-123456789/0800`` can + only ever surface as PHONE_NUMBER false positives. + """ + registry = RecognizerRegistry( + recognizers=[ + PhoneRecognizer(supported_language="cs"), + EmailRecognizer(supported_language="cs"), + IbanRecognizer(supported_language="cs"), + DateRecognizer(supported_language="cs"), + ], + supported_languages=["cs"], + ) + analyzer = AnalyzerEngine( + registry=registry, + nlp_engine=NlpEngineMock( + nlp_artifacts=NlpArtifacts([], [], [], [], None, "cs") + ), + supported_languages=["cs"], + ) + results = analyzer.analyze(TEXT, language="cs") + + found_entities = {result.entity_type for result in results} + assert found_entities <= {"PHONE_NUMBER", "EMAIL_ADDRESS", "IBAN_CODE", "DATE_TIME"} + assert "CZ_BIRTH_NUMBER" not in found_entities + assert "PERSON" not in found_entities + + # The rodné číslo is mis-detected as a phone number (the collision + # this feature fixes). + birth_number_span = _span_of("850412/0003") + assert any( + result.entity_type == "PHONE_NUMBER" + and (result.start, result.end) == birth_number_span + for result in results + ) + + +@pytest.mark.parametrize("snippet, entity_type", EXPECTED_DETECTIONS) +def test_when_czech_text_analyzed_then_each_field_is_detected( + analyzer_results, snippet, entity_type +): + start, end = _span_of(snippet) + matching = [ + result + for result in analyzer_results + if result.entity_type == entity_type + and result.start == start + and result.end == end + ] + assert matching, ( + f"{entity_type} was not detected on {snippet!r} at ({start}, {end}). " + f"Got: {[(r.entity_type, TEXT[r.start:r.end]) for r in analyzer_results]}" + ) + + +def test_when_czech_text_analyzed_then_czech_entities_beat_phone_number( + analyzer_results, +): + """The overlap-resolution regression: CZ_* must outscore PHONE_NUMBER.""" + resolved = _resolve_conflicts(analyzer_results) + phone_numbers = [ + TEXT[result.start : result.end] + for result in resolved + if result.entity_type == "PHONE_NUMBER" + ] + assert phone_numbers == ["+420 601 234 567"] + + resolved_types = {result.entity_type for result in resolved} + assert "CZ_BIRTH_NUMBER" in resolved_types + assert "CZ_BANK_ACCOUNT" in resolved_types + + +def test_when_czech_text_anonymized_then_output_matches_expected(analyzer_results): + """The full before -> after scenario from the feature request.""" + resolved = _resolve_conflicts(analyzer_results) + assert _anonymize(TEXT, resolved) == EXPECTED_ANONYMIZED