Skip to content

Commit 8e21778

Browse files
BreakthroughBreakthrough
committed
[benchmarks] Update sweep results
1 parent 4986739 commit 8e21778

18 files changed

Lines changed: 3541 additions & 3 deletions

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ See [the documentation](https://www.scenedetect.com/docs/latest/api.html) for mo
102102

103103
**Benchmark**:
104104

105-
We evaluate the performance of different detectors in terms of accuracy and processing speed. See the [benchmark report](benchmark/README.md) for details.
105+
We evaluate the performance of different detectors in terms of accuracy and processing speed. See [www.scenedetect.com/benchmarks](https://www.scenedetect.com/benchmarks/) for results, or the [benchmark report](benchmark/README.md) for details on the datasets and methodology.
106106

107107
## Reference
108108

‎benchmark/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,33 @@ Elapsed is mean wall-clock seconds per video.
140140
| HistogramDetector | 69.67 | 81.99 | 75.33 |
141141
| ThresholdDetector | 5.69 | 99.24 | 10.77 |
142142

143+
## Parameter sweep results
144+
145+
The tables above use each detector's v0.7 defaults. A grid sweep over the key parameters
146+
scored by hard-cut F1 at 1-frame tolerance, averaged across BBC / AutoShot / ClipShots gives the
147+
best single parameter set for this corpus mix:
148+
149+
| Detector | Best mean F1 | Best params | v0.7 default |
150+
|:-----------------:|:------------:|:-----------------------------------------------------------|:---------------------------------------|
151+
| ContentDetector | 73.4 | threshold=31, min_scene_len=0.6s | threshold=27 |
152+
| AdaptiveDetector | 76.3 | adaptive_threshold=3.5, window_width=3, min_scene_len=0.6s | adaptive_threshold=3.0, window_width=2 |
153+
| HashDetector | 69.8 | threshold=0.35, size=8 | threshold=0.395, size=16 |
154+
| HistogramDetector | 66.3 | threshold=0.20, bins=128 | threshold=0.05, bins=256 |
155+
| ThresholdDetector | -- | detects fades, not hard cuts (validation only) | threshold=12 |
156+
157+
Per-dataset optima differ (e.g. HistogramDetector peaks at threshold=0.11 on BBC but keeps climbing
158+
to 0.35 on ClipShots); the "best params" column is the single cell with the highest cross-dataset
159+
mean. Full per-dataset breakdowns - including F1 at 0-frame tolerance and precision/recall - are in
160+
[`SWEEP_REPORT.md`](SWEEP_REPORT.md), regenerated with `python -m benchmark.report_sweep`.
161+
The full grids (all detectors and datasets) are driven by `scripts/benchmark_sweep.sh`.
162+
163+
Notes:
164+
165+
- Every sweep cell requires a full decode pass (shared across up to `--workers` cells); `size` /
166+
`bins` change the per-frame metric itself, so hash / hist cells cannot share metrics either.
167+
- HistogramDetector's v0.7 default (0.05 / 256) scores only 54.3 mean F1 - well below the swept
168+
optimum, making it the most miscalibrated default of the five.
169+
143170
## Citations
144171

145172
### BBC

‎benchmark/SWEEP_REPORT.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Detector parameter sweep report
2+
3+
Generated by `benchmark/report_sweep.py` from `benchmark/sweep.py` grid results (hash/hist were swept with `min_scene_len` fixed at its default). F1/precision/recall are percentages on hard cuts; tol = frame tolerance.
4+
5+
## detect-content
6+
7+
**Best cell per dataset (by F1@1):**
8+
9+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
10+
| --------- | ----- | ------ | ----- | ----- | ------------------------------- |
11+
| BBC | 88.34 | 90.00 | 86.75 | 87.35 | min_scene_len=0.8, threshold=25 |
12+
| AutoShot | 73.44 | 79.54 | 68.21 | 70.54 | min_scene_len=0.4, threshold=29 |
13+
| ClipShots | 66.74 | 58.93 | 76.95 | 66.44 | min_scene_len=0.8, threshold=35 |
14+
15+
**Best params averaged across all datasets (mean F1@1):**
16+
17+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
18+
| --------- | ----- | -------- | --------- | ------------------------------- |
19+
| 73.39 | 85.95 | 71.51 | 62.73 | min_scene_len=0.6, threshold=31 |
20+
| 73.37 | 84.27 | 70.95 | 64.89 | min_scene_len=0.6, threshold=33 |
21+
| 73.32 | 83.99 | 72.43 | 63.54 | min_scene_len=0.4, threshold=33 |
22+
| 73.15 | 87.28 | 72.05 | 60.13 | min_scene_len=0.6, threshold=29 |
23+
| 73.09 | 85.18 | 72.97 | 61.10 | min_scene_len=0.4, threshold=31 |
24+
25+
## detect-adaptive
26+
27+
**Best cell per dataset (by F1@1):**
28+
29+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
30+
| --------- | ----- | ------ | ----- | ----- | --------------------------------------------------------- |
31+
| BBC | 94.57 | 94.89 | 94.26 | 93.50 | adaptive_threshold=2, min_scene_len=0.6, window_width=2 |
32+
| AutoShot | 77.19 | 80.48 | 74.16 | 75.45 | adaptive_threshold=3.5, min_scene_len=0.4, window_width=3 |
33+
| ClipShots | 65.53 | 60.19 | 71.89 | 65.47 | adaptive_threshold=5.5, min_scene_len=0.6, window_width=3 |
34+
35+
**Best params averaged across all datasets (mean F1@1):**
36+
37+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
38+
| --------- | ----- | -------- | --------- | --------------------------------------------------------- |
39+
| 76.34 | 90.41 | 76.27 | 62.32 | adaptive_threshold=3.5, min_scene_len=0.6, window_width=3 |
40+
| 76.21 | 87.87 | 76.17 | 64.58 | adaptive_threshold=4, min_scene_len=0.6, window_width=3 |
41+
| 76.18 | 90.43 | 77.19 | 60.93 | adaptive_threshold=3.5, min_scene_len=0.4, window_width=3 |
42+
| 76.16 | 87.91 | 77.07 | 63.51 | adaptive_threshold=4, min_scene_len=0.4, window_width=3 |
43+
| 75.45 | 85.28 | 75.71 | 65.37 | adaptive_threshold=4.5, min_scene_len=0.6, window_width=3 |
44+
45+
## detect-hash
46+
47+
**Best cell per dataset (by F1@1):**
48+
49+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
50+
| --------- | ----- | ------ | ----- | ----- | ------------------------ |
51+
| BBC | 86.91 | 81.59 | 92.96 | 85.81 | size=16, threshold=0.425 |
52+
| AutoShot | 70.17 | 76.06 | 65.12 | 66.89 | size=8, threshold=0.325 |
53+
| ClipShots | 56.38 | 44.46 | 77.03 | 55.66 | size=8, threshold=0.4 |
54+
55+
**Best params averaged across all datasets (mean F1@1):**
56+
57+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
58+
| --------- | ----- | -------- | --------- | ----------------------- |
59+
| 69.83 | 86.38 | 68.98 | 54.12 | size=8, threshold=0.35 |
60+
| 69.63 | 86.65 | 65.86 | 56.38 | size=8, threshold=0.4 |
61+
| 69.63 | 86.65 | 65.86 | 56.37 | size=8, threshold=0.375 |
62+
| 68.18 | 84.28 | 70.17 | 50.10 | size=8, threshold=0.325 |
63+
| 67.00 | 83.71 | 61.66 | 55.65 | size=8, threshold=0.425 |
64+
65+
## detect-hist
66+
67+
> Note: thresholds >= 0.21 come from a grid-extension run (`detect-hist-ext-<dataset>.json`) after the initial grid's best cell landed on its 0.20 upper edge.
68+
69+
**Best cell per dataset (by F1@1):**
70+
71+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
72+
| --------- | ----- | ------ | ----- | ----- | ------------------------ |
73+
| BBC | 86.58 | 87.32 | 85.86 | 85.42 | bins=128, threshold=0.11 |
74+
| AutoShot | 68.99 | 75.29 | 63.67 | 65.70 | bins=128, threshold=0.2 |
75+
| ClipShots | 53.25 | 46.26 | 62.74 | 52.90 | bins=128, threshold=0.35 |
76+
77+
**Best params averaged across all datasets (mean F1@1):**
78+
79+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
80+
| --------- | ----- | -------- | --------- | ------------------------ |
81+
| 66.27 | 82.10 | 68.99 | 47.72 | bins=128, threshold=0.2 |
82+
| 66.23 | 81.47 | 68.87 | 48.36 | bins=128, threshold=0.21 |
83+
| 66.20 | 82.74 | 68.81 | 47.06 | bins=128, threshold=0.19 |
84+
| 66.19 | 80.69 | 68.92 | 48.97 | bins=128, threshold=0.22 |
85+
| 66.17 | 79.53 | 68.88 | 50.10 | bins=128, threshold=0.24 |
86+
87+
## detect-threshold
88+
89+
> Note: `detect-threshold` detects **fades** (fade to/from black), not hard cuts. These datasets' ground truth is hard cuts, so the hard-cut F1 below is expectedly near zero. It is included to validate the sweep pipeline end-to-end, not as a meaningful hard-cut accuracy result.
90+
91+
**Best cell per dataset (by F1@1):**
92+
93+
| Dataset | F1@1 | Prec@1 | Rec@1 | F1@0 | Params |
94+
| --------- | ---- | ------ | ----- | ---- | ------------------------------- |
95+
| BBC | 0.79 | 2.89 | 0.45 | 0.32 | min_scene_len=0.2, threshold=19 |
96+
| AutoShot | 3.98 | 51.09 | 2.07 | 3.14 | min_scene_len=0.4, threshold=20 |
97+
| ClipShots | 1.75 | 6.21 | 1.02 | 0.18 | min_scene_len=0, threshold=10 |
98+
99+
**Best params averaged across all datasets (mean F1@1):**
100+
101+
| Mean F1@1 | BBC | AutoShot | ClipShots | Params |
102+
| --------- | ---- | -------- | --------- | ------------------------------- |
103+
| 2.07 | 0.77 | 3.90 | 1.55 | min_scene_len=0, threshold=19 |
104+
| 2.06 | 0.73 | 3.98 | 1.48 | min_scene_len=0, threshold=20 |
105+
| 2.02 | 0.79 | 3.90 | 1.37 | min_scene_len=0.2, threshold=19 |
106+
| 2.00 | 0.71 | 3.98 | 1.30 | min_scene_len=0.2, threshold=20 |
107+
| 1.96 | 0.69 | 3.90 | 1.28 | min_scene_len=0.4, threshold=19 |

‎benchmark/_common.py‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
#
2+
# PySceneDetect: Python-Based Video Scene Detector
3+
# -------------------------------------------------------------------
4+
# [ Site: https://scenedetect.com ]
5+
# [ Docs: https://scenedetect.com/docs/ ]
6+
# [ Github: https://github.com/Breakthrough/PySceneDetect/ ]
7+
#
8+
# Copyright (C) 2026 Brandon Castellano <http://www.bcastell.com>.
9+
# PySceneDetect is licensed under the BSD 3-Clause License; see the
10+
# included LICENSE file, or visit one of the above pages for details.
11+
#
12+
"""Shared formatting and detector-registry helpers for ``python -m benchmark`` and
13+
``python -m benchmark.sweep``.
14+
15+
Kept intentionally small: the two entry points have different prediction loops (one
16+
default-kwargs pass vs a fan-out parameter sweep) but render results into the same
17+
tables.
18+
"""
19+
20+
from __future__ import annotations
21+
22+
import json
23+
import math
24+
from typing import Any
25+
26+
from benchmark.evaluator import BenchmarkResult
27+
from scenedetect import (
28+
AdaptiveDetector,
29+
ContentDetector,
30+
HashDetector,
31+
HistogramDetector,
32+
ThresholdDetector,
33+
)
34+
35+
DEFAULT_BACKEND = "opencv"
36+
37+
DETECTORS: dict[str, type] = {
38+
"detect-adaptive": AdaptiveDetector,
39+
"detect-content": ContentDetector,
40+
"detect-hash": HashDetector,
41+
"detect-hist": HistogramDetector,
42+
"detect-threshold": ThresholdDetector,
43+
}
44+
45+
46+
def parse_tolerances(spec: str) -> tuple[int, ...]:
47+
"""Parse ``"0,1,5"`` into ``(0, 1, 5)``. Blank entries (e.g. trailing comma) are dropped."""
48+
return tuple(int(x.strip()) for x in spec.split(",") if x.strip())
49+
50+
51+
def fmt_pct(value: float, count: int) -> str:
52+
"""Percentage, or ``n/a`` when the underlying class has zero events."""
53+
return "n/a" if count == 0 else f"{value * 100:.2f}"
54+
55+
56+
def fmt_offset(value: float) -> str:
57+
return "n/a" if math.isnan(value) else f"{value:.3f}"
58+
59+
60+
def render_table(header: list[str], rows: list[list[str]]) -> str:
61+
"""Build a pipe-delimited GitHub-flavored Markdown table as a single string."""
62+
widths = [max(len(header[i]), *(len(r[i]) for r in rows)) for i in range(len(header))]
63+
sep = "| " + " | ".join("-" * w for w in widths) + " |"
64+
header_line = "| " + " | ".join(h.ljust(w) for h, w in zip(header, widths, strict=True)) + " |"
65+
body = [
66+
"| " + " | ".join(c.ljust(w) for c, w in zip(r, widths, strict=True)) + " |" for r in rows
67+
]
68+
return "\n".join([header_line, sep, *body])
69+
70+
71+
HARD_HEADER = ["Tolerance", "Precision", "Recall", "F1", "Offset", "Elapsed"]
72+
FADE_HEADER = ["Tolerance", "Precision", "Recall", "F1"]
73+
74+
75+
def hard_row(result: BenchmarkResult) -> list[str]:
76+
hard = result.hard_cuts
77+
hard_predictions = hard.matched + hard.false_positives
78+
hard_events = hard.matched + hard.missed
79+
return [
80+
str(result.tolerance),
81+
fmt_pct(hard.precision, hard_predictions),
82+
fmt_pct(hard.recall, hard_events),
83+
fmt_pct(hard.f1, hard_events),
84+
fmt_offset(result.mean_abs_offset_hard_cuts),
85+
f"{result.elapsed_mean:.2f}",
86+
]
87+
88+
89+
def fade_row(result: BenchmarkResult) -> list[str]:
90+
fades = result.fades
91+
fade_predictions = fades.matched + fades.false_positives
92+
fade_events = fades.matched + fades.missed
93+
return [
94+
str(result.tolerance),
95+
fmt_pct(fades.precision, fade_predictions),
96+
fmt_pct(fades.recall, fade_events),
97+
fmt_pct(fades.f1, fade_events),
98+
]
99+
100+
101+
def write_json(out_path: str, payload: dict[str, Any]) -> None:
102+
with open(out_path, "w") as f:
103+
json.dump(payload, f, indent=2, default=str)
104+
print(f"\nWrote results to {out_path}")

0 commit comments

Comments
 (0)