-
Notifications
You must be signed in to change notification settings - Fork 80
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Summary
Add GEditBench benchmark for image editing evaluation with 11 task type subsets.
Dataset
- Source:
stepfun-ai/GEdit-Bench(HuggingFace) - Task types: background_change, color_alter, material_alter, motion_change, ps_human, style_change, subject_add, subject_remove, subject_replace, text_change, tone_transfer
- Collate:
prompt_with_auxiliaries_collate
Implementation
- Add
setup_gedit_datasetinsrc/pruna/data/datasets/prompt.py - Support
subsetparam for filtering task types - Filter to English instructions only
- Register in
base_datasets - Add
BenchmarkInfoentry with metrics:["accuracy"], subsets list - Auxiliaries should include
image(input_image),subset - Add test
Acceptance
PrunaDataModule.from_string("GEditBench")works (all subsets)PrunaDataModule.from_string("GEditBench", subset="background_change")works- Auxiliaries include
image,subsetfields - Test passes
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request