Skip to content

feat: Extend the evaluators to support external predictions stored in files #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
nikos-livathinos wants to merge 18 commits into
base: main
Choose a base branch
from
Draft

feat: Extend the evaluators to support external predictions stored in files #185

nikos-livathinos wants to merge 18 commits into from
+546 −118

Conversation

@nikos-livathinos
Copy link
Contributor

@nikos-livathinos nikos-livathinos commented Dec 4, 2025

Support external predictions according to the description given in #112

  • Extend all evaluators to support the optional parameter external_predictions_path.
  • If such a path is provided, it is used to load DoclingDocument objects from files instead of the parquet dataset.
  • GT is always taken from the parquet.
  • The path can contain files with predicted DoclingDocuments in various formats (json, doctags, yaml).
  • Update unit tests.
  • Extend CLI for docling-eval evaluate:
--external-predictions-path        PATH            Path to load existing DoclingDocument predictions. The filename must follow the pattern [doc_id].[json|dt|yaml|yml]

@nikos-livathinos
chore: Move the teds.py inside the subdir evaluators/table
6331723 
Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Introduce the external_predictions_path in BaseEvaluator and du…
85890fb 
…mmy entries in all evaluators.

Extend the CLI to support the --external-predictions-path

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Extend test_dataset_builder.py to save document predictions in …
5f9a279 
…various formats

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Extend MarkDownTextEvaluator to support external_predictions_pa…
e6e8409 
…th. Add unit test

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Extend LayoutEvaluator to support external_predictions_path. Ad…
5624e61 
…d unit test.

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos nikos-livathinos marked this pull request as draft December 4, 2025 16:20
@github-actions
Copy link
Contributor

github-actions bot commented Dec 4, 2025 •
edited
Loading

✅ DCO Check Passed

Thanks @nikos-livathinos, all your commits are properly signed off. 🎉

@nikos-livathinos nikos-livathinos self-assigned this Dec 4, 2025
@mergify
Copy link

mergify bot commented Dec 4, 2025

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🔴 Require two reviewer for test updates

This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@nikos-livathinos
Merge branch 'main' into nli/external_predictions
426b6d1
@nikos-livathinos
fix: Add missing pytest dependencies in tests
171ad74 
Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
fix: Fix loading the external predictions in LayoutEvaluator
0f0cfb5 
Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Introduce external predictions in DocStructureEvaluator. Add un…
8069571 
…it test.

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Extend the TableEvaluator to support external predictions. Add …
8ba6b45 
…unit test

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Extend the KeyValueEvaluator to support external predictions. A…
949d6cc 
…dd unit test.

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Extend the PixelLayoutEvaluator to support external predictions…
13badc5 
…. Add unit test

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Extend the BboxTextEvaluator to support external predictions. A…
8c2a065 
…dd unit test

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Disable the OCREvaluator when using the external predictions
08391b3 
Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
fix: Fixing guard for external predictions in TimingsEvaluator, Readi…
595ba6c 
…ngOrderEvaluator. Fix main

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
fix: Export the doctag files with the correct file extension
406b122 
Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
feat: Refactor the ExternalDoclingDocumentLoader to properly load a D…
ebe70b0 
…oclingDocument from doctags and

the GT image.
- Introduce the staticmethod load_doctags() which covers all cases on page image loading.
- Refactor the FilePredictionProvider to use the load_doctags() from ExternalDoclingDocumentLoader.
- Refactor all evaluators to use the new ExternalDoclingDocumentLoader.

Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
@nikos-livathinos
chore: Rename code file as external_docling_document_loader.py
33511c9 
Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cau-git cau-git Awaiting requested review from cau-git

At least 1 approving review is required to merge this pull request.

Assignees

@nikos-livathinos nikos-livathinos

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nikos-livathinos