diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 1da4481a..0174f564 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -85,7 +85,10 @@ jobs: for data in "$DATA_DIR"/*_conf_authors.json; do [ -f "$data" ] && validate "$data" "$SCHEMA_DIR/conf_authors.schema.json" done - for data in "$DATA_DIR"/*_institution_rankings.json; do + # Only validate files that match the full InstitutionRanking schema. + # Per-conference *_institution_rankings.json variants intentionally + # omit fields such as top_authors and should not be validated here. + for data in "$DATA_DIR"/systems_institution_rankings.json "$DATA_DIR"/security_institution_rankings.json; do [ -f "$data" ] && validate "$data" "$SCHEMA_DIR/institution_rankings.schema.json" done diff --git a/src/_data/navigation.yml b/src/_data/navigation.yml index 03b254a6..7c346551 100644 --- a/src/_data/navigation.yml +++ b/src/_data/navigation.yml @@ -10,6 +10,8 @@ main: url: institution_rankings.html - title: Systems vs. Security url: overview.html + - title: ArtiFinder Discovery + url: artifinder.html - title: Geographic Statistics url: statistics/ - title: Profiles diff --git a/src/_includes/head/custom.html b/src/_includes/head/custom.html index 80ad863c..ee25ae39 100644 --- a/src/_includes/head/custom.html +++ b/src/_includes/head/custom.html @@ -50,6 +50,7 @@ + diff --git a/src/about.md b/src/about.md index 980e7764..c2b12506 100644 --- a/src/about.md +++ b/src/about.md @@ -20,6 +20,7 @@ AE results and committee lists live in the upstream community sites. Contribute - [sysartifacts.github.io](https://github.com/sysartifacts/sysartifacts.github.io) — systems conferences (OSDI, SOSP, ATC, EuroSys, FAST, …) - [secartifacts.github.io](https://github.com/secartifacts/secartifacts.github.io) — security conferences (USENIX Security, CCS, NDSS, S&P, …) +- [ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data) — automatically discovered artifact links (open to manual corrections via pull requests). These links are unverified, carry no badges, and never affect ReproDB scores; see the [ArtiFinder discovery page](/artifinder.html) and [methodology](/methodology.html#artifinder-discovered-artifacts). ### Suggest New Analyses diff --git a/src/artifinder.md b/src/artifinder.md new file mode 100644 index 00000000..86b89980 --- /dev/null +++ b/src/artifinder.md @@ -0,0 +1,97 @@ +--- +title: "ArtiFinder — Automatically Discovered Artifacts" +permalink: /artifinder.html +--- + +[ArtiFinder](https://github.com/DistriNet/ArtiFinder) scrapes conference papers directly and identifies links to their artifacts, independently of any artifact-evaluation (AE) process. This page summarises the ArtiFinder links that ReproDB ingests from the [ArtiFinder-Data](https://github.com/DistriNet/ArtiFinder-Data) repository and how they have grown over time. + +ArtiFinder and its underlying longitudinal study are described by Vansteenhuyse et al., *[Not All Those Who Share Are Lost: Analyzing 25 Years of Cybersecurity Artifact Sharing Practices Through Automated Discovery](https://github.com/DistriNet/ArtiFinder)* (USENIX Security 2026). + +{: .artifinder-note} +**These links are _not_ manually verified.** ArtiFinder-discovered artifacts carry **no badges** and are **excluded from every ReproDB score** (artifact rate, reproducibility rate, combined score, institution and author rankings). They are shown for discovery purposes only, marked with an **Artifinder** sign wherever they appear in search and profiles. The one exception is repository statistics: when ArtiFinder finds a GitHub repository for a paper that _did_ go through AE, that repository may be counted in the repository stats. + +{% if site.data.artifinder_summary %} + +## High-Level Summary + +
+
+
{{ site.data.artifinder_summary.total_discovered }}
+
Discovered Artifacts
+
+
+
{{ site.data.artifinder_summary.total_matched_ae }}
+
Matched to AE Papers
+
+
+
{{ site.data.artifinder_summary.github_count }}
+
GitHub Links
+
+
+
{{ site.data.artifinder_summary.discovery_pct }}%
+
Discovery Rate
+
+
+
{{ site.data.artifinder_summary.year_range }}
+
Years Included
+
+
+ +

Across {{ site.data.artifinder_summary.total_papers }} scanned papers in {{ site.data.artifinder_summary.conferences | join: ", " }}, ArtiFinder discovered {{ site.data.artifinder_summary.total_discovered }} artifact links. ArtiFinder data last updated {{ site.data.artifinder_summary.data_updated }} (only editions from {{ site.data.artifinder_summary.year_range | split: "\u2013" | first }} onward are included).

+ +--- + +## Discovered Artifacts Over Time + +Number of ArtiFinder-discovered artifact links per year, split by whether the paper also went through artifact evaluation (AE) in ReproDB. Papers that never went through AE make up the bulk of ArtiFinder's coverage. + +
+
+
+ +## Discovery Rate Over Time + +Share of scanned papers for which ArtiFinder found an artifact link, per year. This is a coverage measure of ArtiFinder itself, not an artifact-evaluation rate. + +
+
+
+ +## By Conference + +Discovered artifact links per conference, and how many were matched to an AE paper tracked by ReproDB. + +
+
+
+ +## Conference Timeline Heatmap + +Discovered artifact links by conference and year. Darker cells indicate more discovered artifacts that year. + +
+
+
+ +{% else %} + +*ArtiFinder statistics are being generated. Please check back soon.* + +{% endif %} + + + + diff --git a/src/assets/css/reprodb-artifinder.css b/src/assets/css/reprodb-artifinder.css new file mode 100644 index 00000000..8dc3dd00 --- /dev/null +++ b/src/assets/css/reprodb-artifinder.css @@ -0,0 +1,105 @@ +/** + * reprodb-artifinder.css — styling for the ArtiFinder provenance marker + * (shown in search results and profile tables) and the ArtiFinder statistics + * page. Mirrors the ".avail-warn" tooltip pattern so the two markers feel + * consistent. + */ + +/* ── Provenance marker (search + profile) ─────────────────────────── */ +.artifinder-tag { + position: relative; + cursor: help; + display: inline-flex; + align-items: center; + gap: 3px; + font-size: 0.8em; + color: #4a5aa8; + background: #eef1fb; + padding: 1px 6px; + border-radius: 3px; + border: 1px solid #c9d2f0; + vertical-align: middle; +} +.artifinder-tag .artifinder-logo { + width: 0.95em; + height: 0.95em; + flex: 0 0 auto; +} +.artifinder-tag .avail-tip { + display: none; + position: absolute; + bottom: 130%; + left: 50%; + transform: translateX(-50%); + background: #333; + color: #fff; + font-size: 0.9em; + font-weight: normal; + line-height: 1.3; + padding: 5px 9px; + border-radius: 4px; + width: max-content; + max-width: 240px; + white-space: normal; + text-align: left; + z-index: 100; + pointer-events: none; +} +.artifinder-tag:hover .avail-tip { + display: block; +} + +/* An ArtiFinder-only artifact link (no AE artifact_urls alongside it). */ +.rdb-result-links .artifinder-link { + color: #4a5aa8; +} + +/* ArtiFinder-discovered paper rows in profile tables (author + institution). */ +.af-title, +a.af-title { + color: #4a5aa8; + font-style: italic; +} + +/* ── Statistics page ──────────────────────────────────────────────── */ +.artifinder-note { + font-size: 0.92em; + color: var(--ov-muted, #666); + border-left: 3px solid #c9d2f0; + padding: 0.4em 0.9em; + margin: 1em 0; + background: rgba(200, 210, 240, 0.12); + border-radius: 0 4px 4px 0; +} + +/* ── Dark mode ────────────────────────────────────────────────────── */ +html[data-theme="dark"] .artifinder-tag { + color: #aeb8ea; + background: #262a3d; + border-color: #3d445f; +} +html[data-theme="dark"] .rdb-result-links .artifinder-link { + color: #aeb8ea; +} +html[data-theme="dark"] .af-title, +html[data-theme="dark"] a.af-title { + color: #aeb8ea; +} +html[data-theme="dark"] .artifinder-note { + border-left-color: #3d445f; + background: rgba(60, 68, 95, 0.25); +} +@media (prefers-color-scheme: dark) { + html:not([data-theme="light"]) .artifinder-tag { + color: #aeb8ea; + background: #262a3d; + border-color: #3d445f; + } + html:not([data-theme="light"]) .rdb-result-links .artifinder-link { + color: #aeb8ea; + } + html:not([data-theme="light"]) .artifinder-note { + border-left-color: #3d445f; + background: rgba(60, 68, 95, 0.25); + } +} diff --git a/src/assets/images/artifinder-logo.svg b/src/assets/images/artifinder-logo.svg new file mode 100644 index 00000000..3e088820 --- /dev/null +++ b/src/assets/images/artifinder-logo.svg @@ -0,0 +1,11 @@ + + ArtiFinder + + + + + + + + + diff --git a/src/assets/js/reprodb-artifinder.js b/src/assets/js/reprodb-artifinder.js new file mode 100644 index 00000000..e3d59ff4 --- /dev/null +++ b/src/assets/js/reprodb-artifinder.js @@ -0,0 +1,146 @@ +/** + * reprodb-artifinder.js — Chart logic for the ArtiFinder discovery page. + * + * Depends on: ECharts 5 (loaded globally via head/custom.html). + * Reads inline JSON from