diff --git a/src/_data/navigation.yml b/src/_data/navigation.yml index 7c346551..62ad12de 100644 --- a/src/_data/navigation.yml +++ b/src/_data/navigation.yml @@ -10,8 +10,6 @@ main: url: institution_rankings.html - title: Systems vs. Security url: overview.html - - title: ArtiFinder Discovery - url: artifinder.html - title: Geographic Statistics url: statistics/ - title: Profiles @@ -82,6 +80,25 @@ main: url: security/repo_stats.html - title: Methodology url: methodology.html + children: + - title: Overview + url: methodology.html + - title: Data Collection + url: methodology.html#data-collection + - title: Pipeline + url: methodology.html#pipeline + - title: Author Metrics + url: methodology.html#author-metrics + - title: Institution Metrics + url: methodology.html#institution-metrics + - title: Repository Statistics + url: methodology.html#repository-statistics + - title: Search Keywords + url: methodology.html#search-keywords + - title: How to Cite + url: methodology.html#how-to-cite + - title: ArtiFinder Discovery + url: artifinder.html - title: Contribute url: about.html children: diff --git a/src/about.md b/src/about.md index c2b12506..353250b7 100644 --- a/src/about.md +++ b/src/about.md @@ -20,7 +20,7 @@ AE results and committee lists live in the upstream community sites. Contribute - [sysartifacts.github.io](https://github.com/sysartifacts/sysartifacts.github.io) — systems conferences (OSDI, SOSP, ATC, EuroSys, FAST, …) - [secartifacts.github.io](https://github.com/secartifacts/secartifacts.github.io) — security conferences (USENIX Security, CCS, NDSS, S&P, …) -- [ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data) — automatically discovered artifact links (open to manual corrections via pull requests). These links are unverified, carry no badges, and never affect ReproDB scores; see the [ArtiFinder discovery page](/artifinder.html) and [methodology](/methodology.html#artifinder-discovered-artifacts). +- [ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data) — automatically discovered artifact links (open to manual corrections via pull requests). These links are unverified, carry no badges, and never affect ReproDB scores; see the [ArtiFinder discovery page](/methodology/artifinder.html) and [methodology](/methodology/data-collection.html#artifinder-discovered-artifacts). ### Suggest New Analyses diff --git a/src/index.md b/src/index.md index 86b784f1..63b3b0fc 100644 --- a/src/index.md +++ b/src/index.md @@ -8,7 +8,7 @@ skip_chartjs: true

- Research artifacts and artifact evaluation drive reproducibility and scientific impact. ReproDB aggregates and surfaces artifact evaluation outcomes across major security and systems conferences, recognizing the contributions of both artifact authors and artifact evaluation service. + Research artifacts and artifact evaluation drive reproducibility and scientific impact. ReproDB aggregates and surfaces artifact evaluation outcomes across major security and systems conferences, and also includes automatically discovered artifact links from ArtiFinder. The site recognizes the contributions of both artifact authors and artifact evaluation service.

diff --git a/src/methodology.md b/src/methodology.md deleted file mode 100644 index 5098dc0d..00000000 --- a/src/methodology.md +++ /dev/null @@ -1,319 +0,0 @@ ---- -title: "Methodology" -permalink: /methodology.html -skip_chartjs: true ---- - -This page explains how we collect, process, and analyze artifact evaluation data, including detailed calculation formulas for all metrics displayed on this site. For the analysis and additional details, see the [paper citation at the bottom of this page](#how-to-cite). - -{% if site.data.summary %} -{% assign af = site.data.artifinder_summary %} - -## Overview - - - - - - - - - - - -
AE-evaluatedArtiFinder-discovered
Data Schema Version{{ site.data.summary.schema_version }}—
Artifacts / discovered links{{ site.data.summary.total_artifacts }}{{ af.total_discovered | default: "—" }}
Conferences{{ site.data.summary.total_conferences }} ({{ site.data.summary.conferences_list | join: ", " }}){% if af %}{{ af.conferences | size }} ({{ af.conferences | join: ", " }}){% else %}—{% endif %}
Years Covered{{ site.data.summary.year_range }}{{ af.year_range | default: "—" }}
Authors{{ site.data.author_summary.total_authors }}{{ af.author_count | default: "—" }}
AE Committee Members{{ site.data.committee_stats.total_members }} ({{ site.data.committee_stats.unique_members }} unique)—
Badges / scoresYesNo (unverified)
Last Updated{{ site.data.summary.last_updated }}{{ af.data_updated | default: "—" }}
- -{% endif %} - -## Conferences Covered - -Data is collected from conferences tracked by [sysartifacts](https://sysartifacts.github.io) and [secartifacts](https://secartifacts.github.io): - -- **Systems**: {{ site.data.summary.systems_conferences | join: ", " }} -- **Security**: {{ site.data.summary.security_conferences | join: ", " }} - -## Data Collection - -We scrape artifact evaluation results from sysartifacts/secartifacts websites, extract paper titles, authors, badges (Available, Functional, Reproducible, Reusable) and repository URLs. For USENIX conferences (ATC, FAST) we also scrape badge data from technical session pages. AE committee data is gathered from sysartifacts/secartifacts plus direct scraping (USENIX, CHES, PETS websites). - -In addition to AE results, we ingest **automatically-discovered** artifact links from [ArtiFinder](https://github.com/DistriNet/ArtiFinder) (via the [ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data) dataset) and match them to papers by normalized title and author list. These links are **unverified**, carry no badges, and are **excluded from every score**; see [ArtiFinder-Discovered Artifacts](#artifinder-discovered-artifacts) below. - -Repository statistics (GitHub stars/forks, Zenodo/Figshare downloads) are collected via their public APIs. Artifact URLs are checked monthly for availability (HTTP HEAD requests). - -Author names are matched to [DBLP](https://dblp.org) for disambiguation and total-publication counts. Author affiliations are enriched using a priority cascade: -1. **Author Index** (canonical enriched source, when available) -2. **DBLP** person pages (`` tags) -3. **CSRankings** faculty records -4. **OpenAlex** API -5. **Semantic Scholar** API (fallback) -6. **AE committee data** (last resort) - -Affiliation names are normalized to canonical forms (e.g., "MIT" → "Massachusetts Institute of Technology"). - -All scripts are in the [reprodb-pipeline](https://github.com/ReproDB/reprodb-pipeline) repository. Full CLI reference, API documentation, and data model definitions are in the [pipeline documentation](https://reprodb.github.io/reprodb-pipeline/). - -## Pipeline - -The pipeline is orchestrated by a [stage dependency graph](https://github.com/ReproDB/reprodb-pipeline/blob/main/src/stages.py) and runs monthly (1st of each month) via [GitHub Actions](https://github.com/ReproDB/reprodb-pipeline/actions): - -1. **Extract DBLP data** — download and parse the DBLP XML database (~3 GB compressed) for author–paper mappings and affiliations -2. **Scrape artifact results** from sysartifacts/secartifacts websites and USENIX technical session pages -3. **Collect repository statistics** (GitHub stars/forks, Zenodo/Figshare downloads) and check artifact URL availability -4. **Compute author statistics** — match papers to DBLP authors, filter by AE-active years, compute per-author metrics -5. **Generate area-specific author data** — separate systems, security, and combined stats -6. **Aggregate committee statistics** — compile AE membership and chair roles from scraped data -7. **Compute combined rankings** — apply weighted scoring formula, enforce minimum threshold, assign ranks with dense tie-breaking -8. **Aggregate institution rankings** — sum across affiliated authors, classify institution roles -9. **Build author profiles and search index** — generate per-author detail records -10. **Record ranking history** — snapshot current rankings for trend tracking -11. **Export data** (JSON/YAML) to this website - -Independent stages run in parallel where the dependency graph allows. All output data structures are formally defined in the [Data Schemas](https://reprodb.github.io/data-schemas/) documentation. - -The complete pipeline takes ~30 minutes and processes {{ site.data.summary.total_artifacts }}+ artifact papers. - -## Author Metrics - -Individual author statistics are computed by matching artifact papers to DBLP records. Each metric is calculated as follows: - -### Artifacts -The total number of evaluated artifacts (papers with at least one badge) authored by this person across all tracked conferences. - -### Total Papers -The total number of papers this author published at tracked conferences, **counting only years when that conference was conducting artifact evaluation**. For example: -- If ACSAC started AE in 2017, only papers from 2017–present are counted -- If an author published at ACSAC in 2010–2024, only 2017–2024 papers contribute to the denominator -- This prevents artificial deflation of Artifact Rate by excluding pre-AE papers - -The paper count is determined by matching author names to DBLP records and filtering by conference and year. - -**Clamping rule:** If DBLP undercounts papers (i.e., artifact_count > total_papers due to incomplete DBLP records), total_papers is clamped to equal artifact_count. This guarantees that Artifact Rate ≤ 100%. - -### Artifact Rate (AR%) -The percentage of an author's papers (at AE-active conferences) that have artifact badges: **AR% = (Artifacts / Total Papers) × 100**. - -**Key point:** The denominator includes only papers from years when the venue had artifact evaluation. This ensures the rate reflects artifact adoption within the relevant time window, avoiding both over-inflation (counting only artifact papers) and under-inflation (counting all historical publications). - -**Cross-area handling:** For authors active in both systems and security, contributions are **summed**. If an author has 10 systems papers and 5 security papers (all in AE-active years), the denominator is 15. This additive approach is correct because systems and security conferences are disjoint publication venues. - -### Reproducibility Rate (RR%) -Among papers with artifacts, the percentage achieving the highest-tier badge (Reproduced or Reusable): **RR% = (Reproduced badges / Total artifacts) × 100**. -This measures the depth of reproducibility beyond mere artifact availability. - -### Artifact:Evaluation Ratio (A:E) - -The A:E ratio characterizes the balance between artifact production and evaluation service: **A:E = Artifact Score / AE Score**, where: -- **Artifact Score** = sum of badge points (Available+1, Functional+1, Reproduced+1 per artifact) -- **AE Score** = committee service points (member=3, chair=5) - -### AE Memberships -The number of times this author served on an artifact evaluation committee across all tracked conferences. - -### Chair Count -The number of times this author served as an AE chair or co-chair. - -### Combined Score - -A composite metric balancing artifact production, artifact quality, and AE service: **Combined Score = Σᵢ(Aᵢ + Fᵢ + Rᵢ) + Σⱼ(3 + Bⱼ × 2)**, summing over *n* artifacts and *m* AE terms, where: -- **First sum** (per artifact *i*): - - Aᵢ = 1 if artifact *i* is Available, 0 otherwise - - Fᵢ = 1 if artifact *i* is Functional, 0 otherwise - - Rᵢ = 1 if artifact *i* is Reproduced/Reusable, 0 otherwise - - **Maximum per artifact: 3 points** (all three badges) - -- **Second sum** (committee service, per AE term *j*): - - Each committee membership contributes **3 points** - - Bⱼ = 1 if term *j* is a chair role, 0 otherwise — chairs receive a **+2 bonus** for a total of **5 points** per chair term - -**Minimum Score Threshold:** Only individuals and institutions with combined score ≥ 3 are included in rankings. Institutions with placeholder affiliations ("Unknown", etc.) are excluded. - -**Ranking Method:** Dense ranking with ties — authors with the same combined score receive the same rank; the next rank is incremented by the number of tied entries. - -**Why These Weights?** -- **Additive badge scoring (1 point each)** reflects that each badge level requires distinct effort (availability, functionality, reproducibility) -- **AE membership = 3 points** estimates the substantial time investment (~50 hours per evaluation cycle) -- **Chair role = 5 points** recognizes leadership and coordination responsibilities -- This formula balances artifact producers and evaluators, countering the traditional invisibility of evaluation labor in academic metrics - ---- - -## Institution Metrics - -Institution-level statistics aggregate contributions from all authors affiliated with that institution. Affiliations are determined from DBLP person pages and CSRankings faculty data. - -### How Institution Data is Aggregated - -All metrics are **summed across affiliated authors**: - -- **Artifacts**: Total artifacts from all affiliated authors -- **Total Papers**: Total papers from all affiliated authors (AE-active years only) -- **AE Memberships**: Total committee memberships from all affiliated authors -- **Combined Score**: Sum of all affiliated authors' combined scores -- **Researchers**: Count of unique authors affiliated with the institution - -**Artifact Rate and Reproducibility Rate** are then computed from these aggregated totals: - -- **Institution AR%** = (Total artifacts / Total papers) × 100 -- **Institution RR%** = (Total reproduced badges / Total artifacts) × 100 - -### Institution Role Classification - -Each institution is classified based on its Artifact:Evaluation ratio (artifact_score / ae_score): - -- **Artifact-focused** (Producer): A:E ratio > 2.0, or artifact-only (no AE service) -- **Evaluation-focused** (Consumer): A:E ratio < 0.5, or AE-only (no artifacts) -- **Balanced**: 0.5 ≤ A:E ratio ≤ 2.0 - -### Cross-Area Aggregation - -For institution rankings broken down by area (systems vs. security): - -- **Systems rankings**: Include only artifacts, papers, and AE service from systems conferences -- **Security rankings**: Include only artifacts, papers, and AE service from security conferences -- **Overall rankings**: Sum of systems + security contributions - ---- - -## Badge Definitions - -We rely on each conference's official badge definitions. We treat the same badge name as comparable across venues (e.g., Available in one venue is assumed to mean the same or similar level of availability in another). We make the same assumption for Functional. For the highest tier, Reproduced (security) and Reusable (systems) are treated as equivalent. - ---- - -## Repository Statistics - -For artifacts with GitHub/GitLab repositories or Zenodo/Figshare archives, we collect engagement metrics as supplementary signals of community uptake: - -### GitHub/GitLab Metrics -- **Stars**: Number of users who starred the repository -- **Forks**: Number of times the repository was forked - -### Zenodo/Figshare Metrics -- **Downloads**: Total download count from the archive platform -- **Views**: Number of views/visits to the artifact page - -**Important notes:** -- Repository statistics are **displayed separately** and do not contribute to the combined score -- These metrics reflect external reuse signals but are subject to biases: - - Age effects (older artifacts accumulate more stars) - - Repository type differences (libraries vs. experiment code) - - Discovery algorithm effects (GitHub trending, recommendation systems) -- We report these as observational data, not as quality judgments - ---- - -## Artifact Citations (Experimental) - -We attempted to track academic citations to artifact DOIs using [OpenAlex](https://openalex.org), querying citation counts for artifact DOIs (Zenodo and Figshare). - -### Why Citation Data Is Not Included in Rankings - -OpenAlex reported 14 artifacts with a total of 43 citing DOIs. We verified each citing DOI by checking [Crossref](https://www.crossref.org/) publisher-submitted reference lists for the actual artifact DOI, and detected self-citations by comparing author lists between the artifact and the citing paper. - -**Result: zero genuine third-party artifact citations.** All 43 were: -- **36 false positives** — the citing paper's bibliography contains the *paper* DOI, not the artifact DOI. OpenAlex conflates these when the artifact and paper share a title. -- **6 self-citations** — the paper cites its own artifact (same authors). -- **1 unknown** — an arXiv preprint whose references could not be resolved. - -Because current bibliographic indexes do not reliably distinguish artifact citations from paper citations, **citation counts are excluded from the combined score and ranking tables**. The citation collection pipeline remains available as an optional, experimental module for future use as citation infrastructure matures. - -See the [verification scripts and detailed results](https://github.com/ReproDB/reprodb-pipeline) for the full analysis. - ---- - -## ArtiFinder-Discovered Artifacts - -[ArtiFinder](https://github.com/DistriNet/ArtiFinder) scrapes conference papers directly to discover links to their artifacts, independently of any artifact-evaluation process. ReproDB ingests the published [ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data) set and integrates it as follows: - -- **Matching.** Each ArtiFinder link is matched to a ReproDB paper by **normalised title + author list** (ArtiFinder uses DBLP author names, as we do), requiring the same conference and year. When author lists are known on both sides, at least one author must overlap to guard against title collisions. -- **No badges, no scores.** ArtiFinder links are **not manually verified** and carry **no badges**. They **never** contribute to the Artifact Rate, Reproducibility Rate, combined score, or any author/institution ranking. The list also contains many papers that never went through artifact evaluation; these do not count towards any denominator either. -- **Configurable start year.** Only editions from **2017 onwards** (the start of the AE era) are ingested by default; this is configurable in the pipeline (`artifinder_min_year`). -- **Repository-statistics exception.** When ArtiFinder discovers a **GitHub** repository for a paper that *did* go through AE, that repository *may* be counted in the [repository statistics](#repository-statistics), since those stats are descriptive and separate from scoring. -- **Display.** Wherever an ArtiFinder link appears in search results or profiles it is marked with an **Artifinder** sign indicating it was found automatically and has not been manually verified. A dedicated [ArtiFinder discovery page](/artifinder.html) reports discovery counts and rates over time. - -Throughout the site, statistics focus on **AE-evaluated** artifacts; ArtiFinder figures are reported separately (here and on the ArtiFinder page) and are always distinguished from AE results. - -## Data Sources - -- **[sysartifacts.github.io](https://sysartifacts.github.io)** — Systems conference artifact evaluation results ({{ site.data.summary.systems_conferences | join: ", " }}) -- **[secartifacts.github.io](https://secartifacts.github.io)** — Security conference artifact evaluation results ({{ site.data.summary.security_conferences | join: ", " }}) -- **[ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data)** — Automatically discovered artifact links (unverified, no badges, excluded from all scores) -- **[usenix.org](https://www.usenix.org)** — Badge information and AE committee data for USENIX conferences -- **[DBLP](https://dblp.org)** — Author name matching, disambiguation, and total publication counts -- **[OpenAlex](https://openalex.org)** — Author affiliation enrichment -- **[Semantic Scholar](https://www.semanticscholar.org)** — Author affiliation fallback lookup -- **[CSRankings](https://csrankings.org)** — Faculty affiliation data -- **[GitHub](https://docs.github.com/en/rest)**, **[GitLab](https://docs.gitlab.com/ee/api/)** — Repository statistics (stars, forks) -- **[Zenodo](https://developers.zenodo.org)**, **[Figshare](https://docs.figshare.com)** — Archive statistics (downloads, views) -- **[Data Schemas](https://reprodb.github.io/data-schemas/)** — JSON Schema definitions for all data structures used by this site (versioned; see [all schema versions](https://reprodb.github.io/data-schemas/)) - ---- - -## Acknowledgements - -This project celebrates the work of **artifact authors** who go the extra mile to make research reproducible, and **artifact evaluation committees** (AE chairs and members) who invest time reviewing and certifying artifacts. Their contributions strengthen our scientific record. We thank the communities maintaining [sysartifacts](https://sysartifacts.github.io) and [secartifacts](https://secartifacts.github.io) for publishing detailed evaluation results. Inspired by [Systems Circus](https://nebelwelt.net/pubstats/) and [csrankings.org](https://csrankings.org). - ---- - -## API Access - -The full artifact dataset is available as a public JSON endpoint for programmatic access: - -``` -GET {{ site.url }}{{ site.baseurl }}/assets/data/search_data.json -``` - -Returns an array of all {{ site.data.summary.total_artifacts }} artifacts with title, authors, affiliations, conference, year, badges, and repository/artifact URLs. No authentication required. - -Example using `curl`: - -```bash -# Get all artifacts -curl -s {{ site.url }}{{ site.baseurl }}/assets/data/search_data.json | python3 -c " -import sys, json -data = json.load(sys.stdin) -# Filter: fuzzing papers from 2024 -results = [a for a in data if 'fuzz' in a['title'].lower() and a['year'] == 2024] -print(json.dumps(results, indent=2)) -" -``` - ---- - -## Search Keywords - -The [search bar]({{ '/' | relative_url }}) supports special `#` keywords that filter results by specific criteria. Keywords can be combined with each other and with free text. - -| Keyword | Description | -|---|---| -| `#unavailable` | Artifacts whose URLs may no longer be accessible (checked periodically via automated HTTP probes) | -| `#awarded` | Award-winning artifacts only | -| `#github` | Artifacts hosted on GitHub | -| `#zenodo` | Artifacts hosted on Zenodo | -| `#nourl` | Artifacts with no artifact URL recorded | - -**Examples:** - -- `#unavailable` — all artifacts with potentially dead links -- `#awarded OSDI` — award-winning OSDI artifacts -- `#github #unavailable 2022` — GitHub-hosted artifacts from 2022 with dead links -- `#zenodo fuzzing` — Zenodo-hosted fuzzing artifacts - -Keywords also work alongside the year, venue, and area dropdown filters. - -**Note on `#unavailable`:** Availability is determined by automated URL checks that run as part of the monthly pipeline. A URL marked as unavailable may be temporarily down, rate-limited, or require authentication. The hover tooltip on each flagged artifact shows when the check was last performed. - ---- - -## How to Cite - -If you use ReproDB in your research, please cite: - -```bibtex -@inproceedings{vahldiek2026reprodb, - author = {Vahldiek-Oberwagner, Anjo and Bognar, Marton and Signorello, Salvatore}, - title = {ReproDB: An Open Platform for Discovering Research Artifacts and Analyzing their Evaluation in Security and Systems}, - booktitle = {Proceedings of the ACM Conference on Reproducibility and Replicability (ACM REP)}, - year = {2026} -} -``` diff --git a/src/artifinder.md b/src/methodology/artifinder.md similarity index 98% rename from src/artifinder.md rename to src/methodology/artifinder.md index 86b89980..a85bc344 100644 --- a/src/artifinder.md +++ b/src/methodology/artifinder.md @@ -1,6 +1,6 @@ --- title: "ArtiFinder — Automatically Discovered Artifacts" -permalink: /artifinder.html +permalink: /methodology/artifinder.html --- [ArtiFinder](https://github.com/DistriNet/ArtiFinder) scrapes conference papers directly and identifies links to their artifacts, independently of any artifact-evaluation (AE) process. This page summarises the ArtiFinder links that ReproDB ingests from the [ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data) repository and how they have grown over time. @@ -94,4 +94,4 @@ Discovered artifact links by conference and year. Darker cells indicate more dis ] } - + \ No newline at end of file diff --git a/src/methodology/author-metrics.md b/src/methodology/author-metrics.md new file mode 100644 index 00000000..26aca6d4 --- /dev/null +++ b/src/methodology/author-metrics.md @@ -0,0 +1,68 @@ +--- +title: "Methodology: Author Metrics" +permalink: /methodology/author-metrics.html +skip_chartjs: true +--- + +## Author Metrics + +Individual author statistics are computed by matching artifact papers to DBLP records. Each metric is calculated as follows: + +### Artifacts +The total number of evaluated artifacts (papers with at least one badge) authored by this person across all tracked conferences. + +### Total Papers +The total number of papers this author published at tracked conferences, **counting only years when that conference was conducting artifact evaluation**. For example: +- If ACSAC started AE in 2017, only papers from 2017-present are counted +- If an author published at ACSAC in 2010-2024, only 2017-2024 papers contribute to the denominator +- This prevents artificial deflation of Artifact Rate by excluding pre-AE papers + +The paper count is determined by matching author names to DBLP records and filtering by conference and year. + +**Clamping rule:** If DBLP undercounts papers (i.e., artifact_count > total_papers due to incomplete DBLP records), total_papers is clamped to equal artifact_count. This guarantees that Artifact Rate <= 100%. + +### Artifact Rate (AR%) +The percentage of an author's papers (at AE-active conferences) that have artifact badges: **AR% = (Artifacts / Total Papers) x 100**. + +**Key point:** The denominator includes only papers from years when the venue had artifact evaluation. This ensures the rate reflects artifact adoption within the relevant time window, avoiding both over-inflation (counting only artifact papers) and under-inflation (counting all historical publications). + +**Cross-area handling:** For authors active in both systems and security, contributions are **summed**. If an author has 10 systems papers and 5 security papers (all in AE-active years), the denominator is 15. This additive approach is correct because systems and security conferences are disjoint publication venues. + +### Reproducibility Rate (RR%) +Among papers with artifacts, the percentage achieving the highest-tier badge (Reproduced or Reusable): **RR% = (Reproduced badges / Total artifacts) x 100**. +This measures the depth of reproducibility beyond mere artifact availability. + +### Artifact:Evaluation Ratio (A:E) + +The A:E ratio characterizes the balance between artifact production and evaluation service: **A:E = Artifact Score / AE Score**, where: +- **Artifact Score** = sum of badge points (Available+1, Functional+1, Reproduced+1 per artifact) +- **AE Score** = committee service points (member=3, chair=5) + +### AE Memberships +The number of times this author served on an artifact evaluation committee across all tracked conferences. + +### Chair Count +The number of times this author served as an AE chair or co-chair. + +### Combined Score + +A composite metric balancing artifact production, artifact quality, and AE service: **Combined Score = Sum_i(A_i + F_i + R_i) + Sum_j(3 + B_j x 2)**, summing over *n* artifacts and *m* AE terms, where: +- **First sum** (per artifact *i*): + - A_i = 1 if artifact *i* is Available, 0 otherwise + - F_i = 1 if artifact *i* is Functional, 0 otherwise + - R_i = 1 if artifact *i* is Reproduced/Reusable, 0 otherwise + - **Maximum per artifact: 3 points** (all three badges) + +- **Second sum** (committee service, per AE term *j*): + - Each committee membership contributes **3 points** + - B_j = 1 if term *j* is a chair role, 0 otherwise - chairs receive a **+2 bonus** for a total of **5 points** per chair term + +**Minimum Score Threshold:** Only individuals and institutions with combined score >= 3 are included in rankings. Institutions with placeholder affiliations ("Unknown", etc.) are excluded. + +**Ranking Method:** Dense ranking with ties - authors with the same combined score receive the same rank; the next rank is incremented by the number of tied entries. + +**Why These Weights?** +- **Additive badge scoring (1 point each)** reflects that each badge level requires distinct effort (availability, functionality, reproducibility) +- **AE membership = 3 points** estimates the substantial time investment (~50 hours per evaluation cycle) +- **Chair role = 5 points** recognizes leadership and coordination responsibilities +- This formula balances artifact producers and evaluators, countering the traditional invisibility of evaluation labor in academic metrics diff --git a/src/methodology/data-collection.md b/src/methodology/data-collection.md new file mode 100644 index 00000000..96d4fc21 --- /dev/null +++ b/src/methodology/data-collection.md @@ -0,0 +1,53 @@ +--- +title: "Methodology: Data Collection" +permalink: /methodology/data-collection.html +skip_chartjs: true +--- + +## Data Collection + +We scrape artifact evaluation results from sysartifacts/secartifacts websites, extract paper titles, authors, badges (Available, Functional, Reproducible, Reusable) and repository URLs. For USENIX conferences (ATC, FAST) we also scrape badge data from technical session pages. AE committee data is gathered from sysartifacts/secartifacts plus direct scraping (USENIX, CHES, PETS websites). + +In addition to AE results, we ingest **automatically-discovered** artifact links from [ArtiFinder](https://github.com/DistriNet/ArtiFinder) (via the [ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data) dataset) and match them to papers by normalized title and author list. These links are **unverified**, carry no badges, and are **excluded from every score**; see [ArtiFinder-Discovered Artifacts](#artifinder-discovered-artifacts) below. + +Repository statistics (GitHub stars/forks, Zenodo/Figshare downloads) are collected via their public APIs. Artifact URLs are checked monthly for availability (HTTP HEAD requests). + +Author names are matched to [DBLP](https://dblp.org) for disambiguation and total-publication counts. Author affiliations are enriched using a priority cascade: +1. **Author Index** (canonical enriched source, when available) +2. **DBLP** person pages (`` tags) +3. **CSRankings** faculty records +4. **OpenAlex** API +5. **Semantic Scholar** API (fallback) +6. **AE committee data** (last resort) + +Affiliation names are normalized to canonical forms (e.g., "MIT" -> "Massachusetts Institute of Technology"). + +All scripts are in the [reprodb-pipeline](https://github.com/ReproDB/reprodb-pipeline) repository. Full CLI reference, API documentation, and data model definitions are in the [pipeline documentation](https://reprodb.github.io/reprodb-pipeline/). + +--- + +## ArtiFinder-Discovered Artifacts + +[ArtiFinder](https://github.com/DistriNet/ArtiFinder) scrapes conference papers directly to discover links to their artifacts, independently of any artifact-evaluation process. ReproDB ingests the published [ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data) set and integrates it as follows: + +- **Matching.** Each ArtiFinder link is matched to a ReproDB paper by **normalised title + author list** (ArtiFinder uses DBLP author names, as we do), requiring the same conference and year. When author lists are known on both sides, at least one author must overlap to guard against title collisions. +- **No badges, no scores.** ArtiFinder links are **not manually verified** and carry **no badges**. They **never** contribute to the Artifact Rate, Reproducibility Rate, combined score, or any author/institution ranking. The list also contains many papers that never went through artifact evaluation; these do not count towards any denominator either. +- **Configurable start year.** Only editions from **2017 onwards** (the start of the AE era) are ingested by default; this is configurable in the pipeline (`artifinder_min_year`). +- **Repository-statistics exception.** When ArtiFinder discovers a **GitHub** repository for a paper that *did* go through AE, that repository *may* be counted in the [repository statistics](/methodology/repository-statistics.html), since those stats are descriptive and separate from scoring. +- **Display.** Wherever an ArtiFinder link appears in search results or profiles it is marked with an **Artifinder** sign indicating it was found automatically and has not been manually verified. A dedicated [ArtiFinder discovery page](/methodology/artifinder.html) reports discovery counts and rates over time. + +Throughout the site, statistics focus on **AE-evaluated** artifacts; ArtiFinder figures are reported separately (here and on the ArtiFinder page) and are always distinguished from AE results. + +## Data Sources + +- **[sysartifacts.github.io](https://sysartifacts.github.io)** — Systems conference artifact evaluation results ({{ site.data.summary.systems_conferences | join: ", " }}) +- **[secartifacts.github.io](https://secartifacts.github.io)** — Security conference artifact evaluation results ({{ site.data.summary.security_conferences | join: ", " }}) +- **[ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data)** — Automatically discovered artifact links (unverified, no badges, excluded from all scores) +- **[usenix.org](https://www.usenix.org)** — Badge information and AE committee data for USENIX conferences +- **[DBLP](https://dblp.org)** — Author name matching, disambiguation, and total publication counts +- **[OpenAlex](https://openalex.org)** — Author affiliation enrichment +- **[Semantic Scholar](https://www.semanticscholar.org)** — Author affiliation fallback lookup +- **[CSRankings](https://csrankings.org)** — Faculty affiliation data +- **[GitHub](https://docs.github.com/en/rest)**, **[GitLab](https://docs.gitlab.com/ee/api/)** — Repository statistics (stars, forks) +- **[Zenodo](https://developers.zenodo.org)**, **[Figshare](https://docs.figshare.com)** — Archive statistics (downloads, views) +- **[Data Schemas](https://reprodb.github.io/data-schemas/)** — JSON Schema definitions for all data structures used by this site (versioned; see [all schema versions](https://reprodb.github.io/data-schemas/)) diff --git a/src/methodology/index.md b/src/methodology/index.md new file mode 100644 index 00000000..b1fcad31 --- /dev/null +++ b/src/methodology/index.md @@ -0,0 +1,66 @@ +--- +title: "Methodology" +permalink: /methodology.html +skip_chartjs: true +--- + +The following pages explain how we collect, process, and analyze artifact evaluation data, including detailed calculation formulas for all metrics displayed on this site. ReproDB also includes automatically discovered artifact links from [ArtiFinder](/methodology/artifinder.html), documented on the dedicated ArtiFinder page below. + +- [Data Collection](/methodology/data-collection.html) +- [Pipeline](/methodology/pipeline.html) +- [Author Metrics](/methodology/author-metrics.html) +- [Institution Metrics](/methodology/institution-metrics.html) +- [Repository Statistics](/methodology/repository-statistics.html) +- [Search Keywords](/methodology/search-keywords.html) +- [ArtiFinder Discovery](/methodology/artifinder.html) + +{% if site.data.summary %} +{% assign af = site.data.artifinder_summary %} + +## Overview + + + + + + + + + + + +
AE-evaluatedArtiFinder-discovered
Data Schema Version{{ site.data.summary.schema_version }}—
Artifacts / discovered links{{ site.data.summary.total_artifacts }}{{ af.total_discovered | default: "—" }}
Conferences{{ site.data.summary.total_conferences }} ({{ site.data.summary.conferences_list | join: ", " }}){% if af %}{{ af.conferences | size }} ({{ af.conferences | join: ", " }}){% else %}—{% endif %}
Years Covered{{ site.data.summary.year_range }}{{ af.year_range | default: "—" }}
Authors{{ site.data.author_summary.total_authors }}{{ af.author_count | default: "—" }}
AE Committee Members{{ site.data.committee_stats.total_members }} ({{ site.data.committee_stats.unique_members }} unique)—
Badges / scoresYesNo (unverified)
Last Updated{{ site.data.summary.last_updated }}{{ af.data_updated | default: "—" }}
+ +{% endif %} + +## Conferences Covered + +Data is collected from conferences tracked by [sysartifacts](https://sysartifacts.github.io) and [secartifacts](https://secartifacts.github.io): + +- **Systems**: {{ site.data.summary.systems_conferences | join: ", " }} +- **Security**: {{ site.data.summary.security_conferences | join: ", " }} + +--- + +## API Access + +The full artifact dataset is available as a public JSON endpoint for programmatic access: + +``` +GET {{ site.url }}{{ site.baseurl }}/assets/data/search_data.json +``` + +Returns an array of all {{ site.data.summary.total_artifacts }} artifacts with title, authors, affiliations, conference, year, badges, and repository/artifact URLs. No authentication required. + +Example using `curl`: + +```bash +# Get all artifacts +curl -s {{ site.url }}{{ site.baseurl }}/assets/data/search_data.json | python3 -c " +import sys, json +data = json.load(sys.stdin) +# Filter: fuzzing papers from 2024 +results = [a for a in data if 'fuzz' in a['title'].lower() and a['year'] == 2024] +print(json.dumps(results, indent=2)) +" +``` \ No newline at end of file diff --git a/src/methodology/institution-metrics.md b/src/methodology/institution-metrics.md new file mode 100644 index 00000000..2544220a --- /dev/null +++ b/src/methodology/institution-metrics.md @@ -0,0 +1,40 @@ +--- +title: "Methodology: Institution Metrics" +permalink: /methodology/institution-metrics.html +skip_chartjs: true +--- + +## Institution Metrics + +Institution-level statistics aggregate contributions from all authors affiliated with that institution. Affiliations are determined from DBLP person pages and CSRankings faculty data. + +### How Institution Data is Aggregated + +All metrics are **summed across affiliated authors**: + +- **Artifacts**: Total artifacts from all affiliated authors +- **Total Papers**: Total papers from all affiliated authors (AE-active years only) +- **AE Memberships**: Total committee memberships from all affiliated authors +- **Combined Score**: Sum of all affiliated authors' combined scores +- **Researchers**: Count of unique authors affiliated with the institution + +**Artifact Rate and Reproducibility Rate** are then computed from these aggregated totals: + +- **Institution AR%** = (Total artifacts / Total papers) x 100 +- **Institution RR%** = (Total reproduced badges / Total artifacts) x 100 + +### Institution Role Classification + +Each institution is classified based on its Artifact:Evaluation ratio (artifact_score / ae_score): + +- **Artifact-focused** (Producer): A:E ratio > 2.0, or artifact-only (no AE service) +- **Evaluation-focused** (Consumer): A:E ratio < 0.5, or AE-only (no artifacts) +- **Balanced**: 0.5 <= A:E ratio <= 2.0 + +### Cross-Area Aggregation + +For institution rankings broken down by area (systems vs. security): + +- **Systems rankings**: Include only artifacts, papers, and AE service from systems conferences +- **Security rankings**: Include only artifacts, papers, and AE service from security conferences +- **Overall rankings**: Sum of systems + security contributions diff --git a/src/methodology/pipeline.md b/src/methodology/pipeline.md new file mode 100644 index 00000000..bcf75143 --- /dev/null +++ b/src/methodology/pipeline.md @@ -0,0 +1,25 @@ +--- +title: "Methodology: Pipeline" +permalink: /methodology/pipeline.html +skip_chartjs: true +--- + +## Pipeline + +The pipeline is orchestrated by a [stage dependency graph](https://github.com/ReproDB/reprodb-pipeline/blob/main/src/stages.py) and runs monthly (1st of each month) via [GitHub Actions](https://github.com/ReproDB/reprodb-pipeline/actions): + +1. **Extract DBLP data** — download and parse the DBLP XML database (~3 GB compressed) for author-paper mappings and affiliations +2. **Scrape artifact results** from sysartifacts/secartifacts websites and USENIX technical session pages +3. **Collect repository statistics** (GitHub stars/forks, Zenodo/Figshare downloads) and check artifact URL availability +4. **Compute author statistics** — match papers to DBLP authors, filter by AE-active years, compute per-author metrics +5. **Generate area-specific author data** — separate systems, security, and combined stats +6. **Aggregate committee statistics** — compile AE membership and chair roles from scraped data +7. **Compute combined rankings** — apply weighted scoring formula, enforce minimum threshold, assign ranks with dense tie-breaking +8. **Aggregate institution rankings** — sum across affiliated authors, classify institution roles +9. **Build author profiles and search index** — generate per-author detail records +10. **Record ranking history** — snapshot current rankings for trend tracking +11. **Export data** (JSON/YAML) to this website + +Independent stages run in parallel where the dependency graph allows. All output data structures are formally defined in the [Data Schemas](https://reprodb.github.io/data-schemas/) documentation. + +The complete pipeline takes ~30 minutes and processes {{ site.data.summary.total_artifacts }}+ artifact papers. diff --git a/src/methodology/repository-statistics.md b/src/methodology/repository-statistics.md new file mode 100644 index 00000000..a1b381e2 --- /dev/null +++ b/src/methodology/repository-statistics.md @@ -0,0 +1,44 @@ +--- +title: "Methodology: Repository Statistics" +permalink: /methodology/repository-statistics.html +skip_chartjs: true +--- + +## Repository Statistics + +For artifacts with GitHub/GitLab repositories or Zenodo/Figshare archives, we collect engagement metrics as supplementary signals of community uptake: + +### GitHub/GitLab Metrics +- **Stars**: Number of users who starred the repository +- **Forks**: Number of times the repository was forked + +### Zenodo/Figshare Metrics +- **Downloads**: Total download count from the archive platform +- **Views**: Number of views/visits to the artifact page + +**Important notes:** +- Repository statistics are **displayed separately** and do not contribute to the combined score +- These metrics reflect external reuse signals but are subject to biases: + - Age effects (older artifacts accumulate more stars) + - Repository type differences (libraries vs. experiment code) + - Discovery algorithm effects (GitHub trending, recommendation systems) +- We report these as observational data, not as quality judgments + +--- + +## Artifact Citations (Experimental) + +We attempted to track academic citations to artifact DOIs using [OpenAlex](https://openalex.org), querying citation counts for artifact DOIs (Zenodo and Figshare). + +### Why Citation Data Is Not Included in Rankings + +OpenAlex reported 14 artifacts with a total of 43 citing DOIs. We verified each citing DOI by checking [Crossref](https://www.crossref.org/) publisher-submitted reference lists for the actual artifact DOI, and detected self-citations by comparing author lists between the artifact and the citing paper. + +**Result: zero genuine third-party artifact citations.** All 43 were: +- **36 false positives** - the citing paper's bibliography contains the *paper* DOI, not the artifact DOI. OpenAlex conflates these when the artifact and paper share a title. +- **6 self-citations** - the paper cites its own artifact (same authors). +- **1 unknown** - an arXiv preprint whose references could not be resolved. + +Because current bibliographic indexes do not reliably distinguish artifact citations from paper citations, **citation counts are excluded from the combined score and ranking tables**. The citation collection pipeline remains available as an optional, experimental module for future use as citation infrastructure matures. + +See the [verification scripts and detailed results](https://github.com/ReproDB/reprodb-pipeline) for the full analysis. diff --git a/src/methodology/search-keywords.md b/src/methodology/search-keywords.md new file mode 100644 index 00000000..8cfafbf3 --- /dev/null +++ b/src/methodology/search-keywords.md @@ -0,0 +1,28 @@ +--- +title: "Methodology: Search Keywords" +permalink: /methodology/search-keywords.html +skip_chartjs: true +--- + +## Search Keywords + +The [search bar]({{ '/' | relative_url }}) supports special `#` keywords that filter results by specific criteria. Keywords can be combined with each other and with free text. + +| Keyword | Description | +|---|---| +| `#unavailable` | Artifacts whose URLs may no longer be accessible (checked periodically via automated HTTP probes) | +| `#awarded` | Award-winning artifacts only | +| `#github` | Artifacts hosted on GitHub | +| `#zenodo` | Artifacts hosted on Zenodo | +| `#nourl` | Artifacts with no artifact URL recorded | + +**Examples:** + +- `#unavailable` - all artifacts with potentially dead links +- `#awarded OSDI` - award-winning OSDI artifacts +- `#github #unavailable 2022` - GitHub-hosted artifacts from 2022 with dead links +- `#zenodo fuzzing` - Zenodo-hosted fuzzing artifacts + +Keywords also work alongside the year, venue, and area dropdown filters. + +**Note on `#unavailable`:** Availability is determined by automated URL checks that run as part of the monthly pipeline. A URL marked as unavailable may be temporarily down, rate-limited, or require authentication. The hover tooltip on each flagged artifact shows when the check was last performed.