[FEATURE] Add MELO Benchmark datasets as a ranking task for job title normalization

[federetyk]

Problem

Is your proposal tackling an existing problem or limitation?

• No, it's an addition

Proposal

Add the MELO Benchmark datasets [*, **] as ranking tasks in WorkRB. The implementation would be similar to that of the new JobTitleSimilarityRanking task proposed in #24.

Architectural consideration: dataset indexing within each task
In the current WorkRB architecture, each task contains one or more datasets, indexed by language. This design limits each task to having at most one dataset per language. This constraint arises from the code in data loading, evaluation, and result aggregation. However, MELO datasets are identified by (country, query_language, corpus_languages) tuples, so multiple datasets share the same language. We propose generalizing the indexing from Language to arbitrary string identifiers. This would allow WorkRB to fully support MELO and accommodate future tasks with arbitrary indexing for datasets.

@Mattdl Thanks again for inviting us to contribute! The codebase is clean and well-designed. This proposal does add some complexity, but if you think this makes sense, I would be happy to open a separate issue to discuss the refactor. Once aligned, I can submit a PR for the refactor first, and then implement the MELO task on top of those changes.

• Type:

  • New Ontology (data source for multiple tasks)
  • New Task(s)
  • New Model(s)
  • New Metric(s)
  • Other

• Area(s) of code: paths, modules, or APIs you expect to touch
  src/workrb/tasks/__init__.py
src/workrb/tasks/abstract/ranking_base.py
src/workrb/tasks/ranking/__init__.py
src/workrb/tasks/ranking/melo.py
tests/test_task_loading.py

Additional Context

Dataset source:

• HuggingFace: https://huggingface.co/datasets/Avature/MELO-Benchmark
• GitHub: https://github.com/avature/melo-benchmark

Publication:
Retyk et al. (2024) introduced the MELO Benchmark in "MELO: Multilingual Entity Linking of Occupations" (RecSys in HR 2024).
https://ceur-ws.org/Vol-3788/RecSysHR2024-paper_2.pdf

Dataset statistics:

Full statistics for all 48 datasets are available in the HuggingFace dataset card.

Task characteristics:

• Task type: Ranking
• Label type: Multi-label (each query maps to one ESCO occupation, but occupations have multiple surface forms)
• Query input type: Job titles
• Target input type: Job titles (ESCO occupation surface forms)
• Evaluation metrics: MRR, Hit@1, Hit@5, and Hit@10, as used by Retyk et al. (2024).

Potential future addition:
We have unpublished equivalent datasets for skill entity linking to the ESCO Skills taxonomy (~8 datasets) [***]. These follow the same structure as MELO. We can discuss adding these as a separate task if you are interested!

Implementation

• I plan to implement this in a PR
• I am proposing the idea and would like someone else to pick it up

[Mattdl]

Thanks a lot for the neat PR and for taking the time to contribute, really excited to have you involved. 🙌
I’m a bit time-constrained on my side until February and don’t want to block you, so here’s a plan that keeps things moving:

Please go ahead and open a separate issue for the refactor. I left my thoughts below. Feel free to include the outline you already wrote here. Then we can proceed in one of two ways:

1. You build MELO by branching from the refactor branch right away, and we merge both PRs in February. We’ll first merge the refactor PR, then rebase MELO on top of the new main, and then open the MELO as a new PR.

2. You go ahead with the refactor PR, and wait until I can integrate the refactor in February before going for MELO.

Also, the extra datasets you mentioned (beyond MELO) sound super interesting. Once the refactor is in, adding more datasets of the same shape should be pretty low effort.

My thoughts for the refactor

• MELO is a benchmark suite by itself (which is awesome!). I think we might want to make each of the subtasks accessible individually as well. This could be achieved by defining the class instantiation __init__ requiring to define the country for example.
• To still enable users to just define MELO by a single task, we should probably make a abstract TaskList. For MELO this could already predefine the countries in the dataset, and in this way return a list of Task objects.
• Only downside I'm thinking of is that users might want to have also metrics just on separate parts of the benchmark. But either they run it separately then (just have a list of tasks that are in MELO), or we could go more complex and enable the summary_metrics also to be calculated separately as well per TaskList present. For now, I'd keep it simple and go for the first one.

Regarding the metrics:

• We have both the unit of exection (which is currently just Language), and also how they are aggregated for the final metrics. We will have to split this up into not just Language, but query_language and target_language properties per task. Together they should be the identifier for the unit of execution for a Task.
• Right now the metrics first aggregate per language, then task-level, then group-level, then over all groups for the final metric. Instead we would now want to make it more flexible to the user, with a default aggregation mode, but the choice to aggregate over source_language/target_language/cross_language. All current single-language tasks would just have a duplicate 'en-en' for example.
  • An example would be: es-es, es-de, nl-de. Aggregate over source (es-es,es-de would be merged), aggregate over target (es-de,nl-de would be merged), aggregate cross-language (would result in all 3: es-es, es-de, nl-de).

[federetyk]

@Mattdl Thanks for the detailed feedback, and for proposing this approach!

To make sure we are aligned, I wanted to compare the two options:

• Alternative A: Generalize dataset indexing within tasks. This is what I initially outlined in my proposal above. To make this alternative more concrete, I have prepared a working implementation in PR refactor: generalize dataset indexing from language-based to dataset_id-based #34. The code is backward compatible, existing tasks work as they did in the original code, and all tests pass. This refactor alternative is explained in more detail in Issue [FEATURE] Generalize Dataset Indexing Within Tasks #33, in case you'd like to discuss it further.

• Alternative B: Use query/corpus language pairs for dataset indexing. This what I understood based on your suggestion. Datasets would be indexed by (query_language, corpus_language) pairs with compound identifiers like "en-en" or "es-nl". This would later enable results aggregation by query language, corpus language, or specific cross-lingual scenarios. As an optional addition, we could implement the TaskList class as you suggested to aggregate suites of tasks.

Regarding trade-offs, both approaches handle cross-lingual datasets. However, Alternative A also supports multiple monolingual datasets per language (like regional variants) and multilingual datasets (such as corpora in many different languages), while Alternative B is specific to the cross-lingual case. Additionally, introducing the concepts of "query language" and "corpus language" in dataset indexing fits ranking tasks well but would not generalize to classification tasks, which I understand would require different indexing schemes per task type. Alternative A applies consistently to both task types.

For MELO specifically, the benchmark includes monolingual, cross-lingual, and multilingual datasets. In particular, one of the two USA datasets is multilingual, with queries in English and corpus elements in multiple ESCO languages. Here is how MELO would look like, under each alternative:

• With Alternative A, we would have a single ranking task for the entire suite of datasets. The __init__ would have the standard languages parameter for filtering by language, and perhaps we could add an optional countries parameter to filter by country. In this option, "subtasks" are just subgroups of datasets within this big MELO ranking task.

• With Alternative B, we would have one ranking task per country (as you suggested), which would cover most cases: each would have one monolingual dataset and one cross-lingual dataset. For example, for Austria we would have one task with two datasets, aus_q_de_c_de and aus_q_de_c_en, and for Germany we would have another task also with two datasets, deu_q_de_c_de and deu_q_de_c_en. However, the multilingual USA dataset would need to be excluded (which I think is a reasonable trade-off). Under this option, as you mentioned, a TaskList object could group the per-country tasks for aggregated reporting.

That said, I'm open to whatever direction you think is best for the project. The PR is mainly to illustrate the idea concretely, so feel free to discard it if you prefer to go with Alternative B. Let me know what you think!