-
Notifications
You must be signed in to change notification settings - Fork 80
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Description
Add OneIG benchmarks including alignment benchmarks for anime, human, and object generation, plus text rendering evaluation.
Details
OneIG Alignment
- Dataset:
OneIG-Bench/OneIG-Bench+ GitHub JSON for questions - Subcategories: anime_alignment, human_alignment, object_alignment
- Collate:
prompt_with_auxiliaries_collate
OneIG Text Rendering
- Source: GitHub CSV from
OneIG-Bench/OneIG-Benchmarkrepo - Collate:
prompt_with_auxiliaries_collate
Implementation
- Add
setup_oneig_alignment_datasetinsrc/pruna/data/datasets/prompt.py - Add
setup_oneig_text_rendering_datasetinsrc/pruna/data/datasets/prompt.py - Support
categoryparam for filtering (Anime_Stylization, Portrait, General_Object) - Register in
base_datasets - Add
BenchmarkInfoentries with appropriate metrics - Auxiliaries should include
questions,dependencies,category,text_contentfields - Add tests
Acceptance Criteria
OneIG Alignment
-
PrunaDataModule.from_string("OneIGAlignment")works -
PrunaDataModule.from_string("OneIGAlignment", category="anime")works - Auxiliaries include
questions,dependencies,categoryfields - Test passes
OneIG Text Rendering
-
PrunaDataModule.from_string("OneIGTextRendering")works - Auxiliaries include
text_contentfield - Test passes
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request