Skip to content

[BENCHMARK] Add OneIG benchmarks (Alignment + Text Rendering) #515

@davidberenstein1957

Description

@davidberenstein1957

Description

Add OneIG benchmarks including alignment benchmarks for anime, human, and object generation, plus text rendering evaluation.

Details

OneIG Alignment

  • Dataset: OneIG-Bench/OneIG-Bench + GitHub JSON for questions
  • Subcategories: anime_alignment, human_alignment, object_alignment
  • Collate: prompt_with_auxiliaries_collate

OneIG Text Rendering

  • Source: GitHub CSV from OneIG-Bench/OneIG-Benchmark repo
  • Collate: prompt_with_auxiliaries_collate

Implementation

  • Add setup_oneig_alignment_dataset in src/pruna/data/datasets/prompt.py
  • Add setup_oneig_text_rendering_dataset in src/pruna/data/datasets/prompt.py
  • Support category param for filtering (Anime_Stylization, Portrait, General_Object)
  • Register in base_datasets
  • Add BenchmarkInfo entries with appropriate metrics
  • Auxiliaries should include questions, dependencies, category, text_content fields
  • Add tests

Acceptance Criteria

OneIG Alignment

  • PrunaDataModule.from_string("OneIGAlignment") works
  • PrunaDataModule.from_string("OneIGAlignment", category="anime") works
  • Auxiliaries include questions, dependencies, category fields
  • Test passes

OneIG Text Rendering

  • PrunaDataModule.from_string("OneIGTextRendering") works
  • Auxiliaries include text_content field
  • Test passes

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions