feat: Add tokenizer-based text chunking for NER recognizers

[yuriihavrylko]

Change Description

Add TokenizerBasedTextChunker that uses HuggingFace tokenizers for accurate token-based text splitting, and make text chunking configurable from yaml for both GLiNERRecognizer and HuggingFaceNerRecognizer

What's new

• TokenizerBasedTextChunker: splits text by token count using the model's actual tokenizer, respecting its token limit
• YAML text_chunker config: text_chunker parameter on NER recognizers now accepts a dict (from YAML) in addition to Python objects, resolved via TextChunkerProvider
• GLiNER parity: added missing chunk_size/chunk_overlap params to GLiNERRecognizer (HuggingFaceRecognizer already had them)
• Lazy imports for transformers - only required when actually using the tokenizer chunker

YAML example

  - name: GLiNERRecognizer                                                                                                        
    type: predefined                                                                                                                
    text_chunker:
      chunker_type: tokenizer                                                                                                       
      tokenizer: urchade/gliner_multi_pii-v1                                                                                        
      max_tokens: 512                                                                                                               
      overlap_tokens: 32 

Why it matters

• Character-based approximation either wastes capacity or exceeds limits. Token-based chunking uses the exact budget available
• Configurable from YAML means users can tune chunking strategy without writing Python code, making it accessible for
  deployment-only workflows
• GLiNERRecognizer had no chunking configurability - chunk size and overlap were hardcoded (250 chars / 50 overlap) with no way to adjust them from config or constructor params. Now both chunk_size/chunk_overlap params and the full text_chunker config are supported

Issue reference

No linked issue

Checklist

• I have reviewed the contribution guidelines
• I have signed the CLA (if required)
• My code includes unit tests
• All unit tests and lint checks pass locally
• My PR contains documentation updates / additions if required

[Copilot]

Pull request overview

Adds configurable tokenizer-aware text chunking for NER recognizers, improving long-text handling for HuggingFace and GLiNER based recognition.

Changes:

• Introduces TokenizerBasedTextChunker and lazy exports/imports to keep transformers optional.
• Allows GLiNERRecognizer and HuggingFaceNerRecognizer to accept YAML-style text_chunker dict configs.
• Adds tests and documentation for chunker configuration and GLiNER chunking usage.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
presidio-analyzer/presidio_analyzer/chunkers/tokenizer_based_text_chunker.py	Adds tokenizer-based chunking implementation.
presidio-analyzer/presidio_analyzer/chunkers/text_chunker_provider.py	Adds provider support for tokenizer chunker type.
presidio-analyzer/presidio_analyzer/chunkers/__init__.py	Exposes tokenizer chunker through lazy import.
presidio-analyzer/presidio_analyzer/predefined_recognizers/ner/gliner_recognizer.py	Adds configurable chunker support and chunk size parameters.
presidio-analyzer/presidio_analyzer/predefined_recognizers/ner/huggingface_ner_recognizer.py	Adds dict-based chunker config support.
presidio-analyzer/tests/test_tokenizer_based_text_chunker.py	Adds unit tests for tokenizer chunking behavior.
presidio-analyzer/tests/test_gliner_recognizer.py	Adds GLiNER chunker configuration tests.
presidio-analyzer/tests/test_huggingface_ner_recognizer.py	Adds HuggingFace recognizer chunker configuration tests.
docs/samples/python/gliner.md	Documents GLiNER text chunking configuration.
docs/analyzer/recognizer_registry_provider.md	Documents YAML text_chunker configuration.

[omri374]

Thanks! Great addition!

Please make sure that the YAML flow is properly tested

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

[yuriihavrylko]

Looking at the solution, I noticed that requiring users to specify a tokenizer each time they want to use the tokenizer-based chunker isn't ideal - in most cases it will be the same tokenizer as the model itself. Specifying it separately is redundant and error-prone (risk of mismatch between chunker tokenizer and model tokenizer). For GLiNER models it's not even possible since their repos can't be loaded by AutoTokenizer (so users will be need to use backbone tokenizer).

So I made the tokenizer optional and added a deferred resolution pattern - the chunker is created without a tokenizer from YAML config, and the recognizer automatically resolves it at model-load time using the model's own tokenizer. This way users just configure chunk sizes in YAML and the right tokenizer is picked up automatically. Less config, fewer mistakes.

YAML example

  - name: GLiNERRecognizer                                                                                                        
    type: predefined 
    model_name: urchade/gliner_multi_pii-v1                                                                                                               
    text_chunker:
      chunker_type: tokenizer                                                                                                       
      # tokenizer: urchade/gliner_multi_pii-v1         This is now optional, and be derived from model                                                                           
      max_tokens: 512                                                                                                               
      overlap_tokens: 32 

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.