fix(eval,detect): correct bench gold, gate geolocation, close URL bypass

[lBroth]

Supersedes #44, #45 and #46. #45's feature ships here with its redaction bypass closed; #44 and #46 are closed with the evidence on their own threads.

OOD-7 macro 0.7784 → 0.8290. Two independent halves, kept separate because only one is a product improvement.

	OOD-7	nullpii-bench
published (v0.3.0)	0.7784	0.4228
after the gold fix — metric correction, moves every tool	0.8043	0.4228
after the product changes	0.8290	0.4519

---

Part 1 — the bench was deleting gold, and the deletion was laundering precision

_map_ai4privacy_label and the isotonic loader resolved unknown upstream labels through dict.get(), which returns None for "deliberately excluded" and for "never seen" alike. A label the map had never heard of was dropped with no signal: 56.4% of gold on ai4privacy-300k-heldout (18,661 of 33,115 spans), ~11% on the isotonic -heldout rows.

Because macro_f1 skips classes with zero gold support, a prediction on a class whose gold had been deleted cost nothing at all. The bug was not only hiding recall, it was inflating precision — nullpii's 1,331 phone, 2,364 account_number, 1,625 geolocation, 839 passport, 676 driver-licence and 767 secret false positives on that one row were entirely unpenalised.

All 25 recovered labels are aliases of keys already in the map. _AI4PRIVACY_LABELS was authored against the 400k release; 300k spells the same taxonomy differently (GIVENNAME1/LASTNAME1/TEL/BOD/PASS vs GIVENNAME/SURNAME/TELEPHONENUM/DOB/PASSWORD). 44 of the map's 67 keys were dead, which is why it looked comprehensive.

Mapping and exclusion are now distinct, and anything in neither raises:

raise ValueError(
    f"[ai4privacy] unknown gold label {label!r}. Add it to "
    f"_AI4PRIVACY_LABELS (it is PII we detect) or to "
    f"_AI4PRIVACY_IGNORED (it is not) — never leave it to fall "
    f"through, that silently deletes gold and inflates precision.",
)

Selection is by schema membership, not by score. LITECOINADDRESS is mapped even though it lowers our macro by 0.0013; AGE, USERAGENT and ACCOUNTNAME stay excluded even though mapping them would lower it further (−0.0054, −0.0177, −0.0046) — the exclusion list was checked symmetrically rather than being only the mappings that flatter us.

This changes the ranking, not just the numbers. Re-running all 9 tools on the 5 affected rows reorders 8 of 9 on ai4privacy-300k-heldout: piiranha +0.2347, gliner-pii-large-v1 +0.1801, gliner-onnx-pii-fp32 +0.1423, nullpii −0.0519, presidio −0.1746. Tools whose output vocabulary covers the full ai4privacy taxonomy were the ones the bug punished hardest.

Also here: packages/eval/scripts/ood_macro.py becomes the single definition of the OOD-7 set (the root README said 7 datasets and packages/eval/datasets/README.md said 5 — a 0.0128 discrepancy that would silently break any regression gate), and the eval wheel gets a per-file licence allowlist instead of force-including the whole datasets/ directory.

---

Part 2 — private_geolocation was labelling place names

The schema defines the label as a coordinate: all three recognizers that own it match lat/lon literals and core:geo-latlon-decimal range-validates. The GLiNER head, prompted zero-shot and never trained on it, applies it to place names, regions and postcodes. On ai4privacy-300k-heldout only 7.6% of 1,625 emissions were coordinate-shaped; the rest were Holter Straße, Altrip, Deutschland, Sachsen, 52396.

Every gold span of the class is a lat/lon pair, and so is every correct prediction, so a shape gate cannot remove a true positive that matches the schema's own definition. Measured: −2,317 false positives, ±0 true positives, recall unchanged to four decimals, private_geolocation precision 0.129 → 0.902.

It runs before dedupe on purpose. A place name tagged geolocation is both a false positive and an unmatched private_address gold; filtering raw decoder output lets the competing candidate survive the overlap instead of losing it to a label it should never have won. That ordering alone recovers 83 true positives — private_ip +60, private_address +11, private_vehicle_id +7, private_mac +5.

This is not only a bench artifact: today a user who sends Deutschland or a postcode gets a GPS-coordinate placeholder back.

---

Part 3 — the public-URL allowlist, with its bypass closed

#45's goal is sound: github.com/acme/repo in a system prompt is not PII and redacting it destroys signal the upstream model needs. As written it opened a general redaction bypass. core:url matches [^\s<>"]+, so PII glued to a URL is swallowed into the URL span; removeContainedSpans then drops the inner span regardless of score, and dropping the outer span emitted the secret as plaintext. Verified end-to-end with the real model:

input	main	#45 as written	this PR
x https://github.com/a,AKIAIOSFODNN7EXAMPLE y	redacted	AWS key in the clear	redacted
x https://github.com/a,ghp_ABCDEF… y	redacted	GitHub PAT in the clear	redacted
x https://anthropic.com/x?key=sk-ant-api03-… y	redacted	Anthropic key in the clear	redacted
repo https://github.com/acme/infra-gitops here	redacted	preserved ✓	preserved ✓

A 20-character allowlisted prefix disabled redaction for an arbitrary secret.

dropCleanPublicUrlSpans drops an allowlisted URL span only when it carries no other PII span, running before dedupe while the nested spans still exist. Fail-safe by construction: a reference URL that embeds anything identifying reverts to whole-URL redaction. Note the naive fix — reordering unconditionally — is worse than either alternative: with realistic model output it shreds the URL into placeholders (repo https://{{PII_…}}.com/{{PII_…}}/infra-gitops), defeating the feature's own purpose.

Two smaller fixes: a multi-scheme guard, because core:url does not stop at , and https://github.com/foo,https://acme.io/internal was allowlisted on the first host's authority; and .hostname instead of .host, which silently disabled the allowlist on any non-default port.

Host matching itself needed no changes — 30+ attempted bypasses (evil-github.com, github.com.attacker.net, github.com@evil.com, Cyrillic homographs, punycode, github.com。evil.io, acme-corp.github.io) all correctly redact.

---

Verification

• 300 tests pass (was 271 on main): 10 for the geolocation gate, 10 for the nested-PII guard, plus feat(url): allowlist public-reference hosts from private_url output #45's own suite with one fixture corrected — it parked every span at offset 0, which under a containment-aware rule reads as "this URL has PII nested inside it".
• All 11 affected loaders load without raising, so the ignore-sets cover the real upstream vocabularies (full-release scan: 300k = 28 labels, 400k = 17).
• ai4privacy-400k provably unaffected: 0 drops across all 406,896 rows, empty vocabulary intersection with the 15 new keys.
• presidio-synthetic moves by exactly +0.0000 — it carries no private_geolocation gold and none of its 89 URLs are allowlisted. That is the negative control.
• ood_macro.py reproduces the published 0.7784 exactly, and exits non-zero if any cell is missing or CRASHED.

Not in this PR

• published-bench/ is untouched. The 11 rows the gold fix did not affect still carry v0.3.0 numbers; publishing the full table needs them re-run for run-consistency (~6h).
• Pre-existing, unrelated, still open: partially-overlapping spans corrupt the output and break the round-trip. On main, x https://github.com/users/john.doe@acme.com y yields private_url[2,44] and private_email[27,46] — IoU 0.39, below the dedupe threshold, so both survive and overwrite each other during vault substitution. The email span also ends past the end of the string.
• Three artifact inconsistencies in published-bench/: confusion.json (May 14) and matrix.json (May 20) are from different runs and use different dataset keys, and the key nullpii-internal-bench no longer exists in the code (the registry key is nullpii-bench). Related: --datasets <unknown-key> prints "no datasets selected" and exits 0 instead of failing loud the way --tools <unknown> does.
• ai4privacy-300k is 100% English and ai4privacy-300k-heldout is 100% German — the upstream file is grouped by language, so that pair confounds memorisation with a locale switch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01463aDpX9uxUxNrrNzLkCbg