Add "agentic" model type to submission metadata & leaderboard (similar to GIFT-Eval)

[Jameswlepage]

Add an agentic value to submission-level model-type metadata so agentic forecasting systems can self-declare and be filtered on the leaderboard. Metadata only. No change to tasks, metrics, EvaluationWindow, or aggregation.

Motivation
Agentic systems (tool-using, multi-step pipelines that produce forecasts) are now a distinct submission class, not plain pretrained or fine-tuned models. GIFT-Eval added an Agentic model type for this reason, and it gets used: the Moirai 2.0 paper filters out agentic solutions to get a clean foundation-model comparison. Without a tag, agentic submissions get excluded informally or silently skew a board readers assume is model v model.

Approach

• Extend the submission-level model_type field with an agentic value, surfaced as a filterable leaderboard column.
• Submission-level, not task-level, so it describes how the forecast was produced, orthogonal to the task taxonomy.
• Same scoring and statistical treatment as everything else. The tag only enables include or exclude, not a separate ranking.

Happy to send a PR for this.

[shchur]

Hi @Jameswlepage, thanks for the suggestion. Agentic submissions are welcome, and I agree that we should make a separate category for them.

Currently this metadata is stored ad-hoc in the leaderboard code at https://huggingface.co/spaces/autogluon/fev-bench/blob/main/src/utils.py#L27-L83. Therefore it's probably sufficient to just specify the submission type in the PR - we can then update the metadata on HF accordingly.

One important detail: we should have clear instructions for authors of agentic submissions on how to disclose leakage on the benchmark tasks. I think a reasonable requirement will be "you should set trained_on_this_dataset=True if any of the base models used in the agentic system have seen this dataset during training". For example, if authors use TimesFM-2.5 within their system, they need to mark all of these datasets as leaky

fev/models/timesfm-2.5/model.py

Lines 11 to 21 in 7c9b78d

	trained_on_datasets = [
	"favorita_transactions_1D",
	"favorita_transactions_1W",
	"fred_md_2025",
	"proenfo_gfc12",
	"proenfo_gfc14",
	"proenfo_gfc17",
	"kdd_cup_2022_10T",
	"m5_1D",
	"m5_1W",
	]

This puts more transparency requirements on the submissions - the authors need to disclose which models the system uses under the hood. Do you think it's a reasonable requirement?