-
Notifications
You must be signed in to change notification settings - Fork 80
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Description
Add DPG (Descriptive Prompt Generation) benchmarks for image understanding evaluation.
Details
- Source: GitHub CSV from
TencentQQGYLab/ELLArepo - Subcategories: entity, attribute, relation, global, other
- Collate:
prompt_with_auxiliaries_collate
Implementation
- Add
setup_dpg_datasetinsrc/pruna/data/datasets/prompt.py - Support
categoryparam for filtering subcategories - Register in
base_datasets - Add
BenchmarkInfoentry with metrics:["qa_accuracy"], subsets list - Auxiliaries should include
questionslist andcategory_broadfor evaluation - Add test
Acceptance Criteria
-
PrunaDataModule.from_string("DPG")works (all subcategories) -
PrunaDataModule.from_string("DPG", category="entity")works - Auxiliaries include
questionsandcategory_broadfields - Test passes
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request