feat(examples): add custom benchmark examples for BEAM#821
Merged
egordm merged 7 commits intorelease/v4.0.0from Mar 27, 2026
Merged
feat(examples): add custom benchmark examples for BEAM#821egordm merged 7 commits intorelease/v4.0.0from
egordm merged 7 commits intorelease/v4.0.0from
Conversation
…mments - example_baseline.py: minimal forecaster (constant median) implementing BacktestForecasterMixin - example_benchmark.py: custom target provider extending SimpleTargetProvider - run_liander2024_benchmark.py: run example baseline + GBLinear on Liander 2024 - run_benchmark.py: run example baseline + GBLinear using custom benchmark pipeline - README.md: beginner-friendly guide with setup and usage instructions Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
Allows users to inject their own forecast predictions and run only evaluation + analysis, skipping the backtesting step entirely. Includes format_predictions() helper and a stub forecaster factory. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
…erConfig - Correct import paths for BacktestForecasterMixin and RestrictedHorizonVersionedTimeSeries - Add missing required fields to BacktestForecasterConfig in stub - Verified: pipeline correctly skips backtesting and runs eval + analysis Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
… data format - Rename evaluate_forecasts.py → evaluate_existing_forecasts.py - Remove format_predictions() helper and save loop (step 1) - Just point LocalBenchmarkStorage at existing parquets - Use DummyForecaster instead of custom _QuantileStub - Add README section: directory layout, parquet format table, example rows - Add LeadTime to evaluation config in example_benchmark.py Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
Add compare_liander2024_results.py and compare_custom_results.py that generate side-by-side comparison plots across benchmark runs using BenchmarkComparisonPipeline. Update README with comparison section. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
egordm
force-pushed
the
chore/example-custom-baseline-benchmark
branch
from
faed530 to
2c68f9a
Compare
egordm
force-pushed
the
chore/example-custom-baseline-benchmark
branch
from
2c68f9a to
b742df8
Compare
Baselines are no longer re-exported from the package __init__.py. Import directly from openstef_beam.benchmarking.baselines.openstef4. Also fix improper Path import from openstef_core.base_model. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
Add missing __init__.py files so examples can be run with python -m. Use re-exported StrictExecutionCallback from openstef_beam.benchmarking. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
egordm
force-pushed
the
chore/example-custom-baseline-benchmark
branch
from
b742df8 to
c4e0283
Compare
|
bartpleiter
approved these changes
Mar 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end examples showing how to create and run custom BEAM benchmarks, plus a Windows compatibility fix.
Custom Benchmark Examples
example_baseline.pyBacktestForecasterMixin— predicts a constant medianexample_benchmark.pySimpleTargetProviderwith metrics and pipeline assemblyrun_liander2024_benchmark.pyrun_benchmark.pyREADME.mdAll files include inline comments explaining concepts for newcomers (quantiles,
RestrictedHorizonVersionedTimeSeries, config parameters, metrics, etc.).Windows Compatibility Fix
AvailableAt.__str__()now producesD-1T0600instead ofD-1T06:00. Colons are illegal in Windows file/directory names, which broke benchmark output paths likebenchmark_results/analysis/mv_feeder/OS Edam/D-1T06:00/.from_string()accepts bothD-1T06:00(legacy) andD-1T0600(new) for backward compatibility