bench: varying-slopes reference benchmark vs R fixest (#65)#66
Merged
Conversation
c7ea865 to
ca84e65
Compare
Standalone, on-demand perf comparison for the varying-slopes epic (#64), timing slope estimation on shared synthetic DGPs before within's slope solver exists. R fixest is the only live arm; within is a stub behind --with-within, wired in once #58/#59 land. - benchmarks/slopes_fixest.py: Case dataclass, fixest/akm slope DGP builders (adapted from the defunct prototype, reusing live benchmarks._problems panel helpers), DATASETS catalog, run_fixest + run_within stub, argparse CLI + table printer. - benchmarks/fixest_fit.R: R worker (recovered ~verbatim from the lost local harness) parameterised over (csv, formula, n_repeat); warmup + median-of-n feols, fixef() slope recovery, RESULT_* markers. - pixi.toml: dedicated fixest feature+env (r-base/r-fixest/r-data.table) so R stays out of the default dev/test/CI env; bench-slopes task. Run: pixi run -e fixest bench-slopes
ca84e65 to
0318709
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A standalone, on-demand benchmark that times varying-slopes estimation on shared synthetic DGPs and prints R fixest's fit time + iteration counts. Captures reference numbers before within's slope solver exists (#58/#59), so the implementation slices can be measured against an external baseline.
Not a test: no correctness gating, run by hand.
Design (slim by intent)
benchmarks/slopes_fixest.py+ one R workerbenchmarks/fixest_fit.R.main()runs the fixedCASEScatalog at a module-levelSIZESconstant; sweeping scale is a one-line edit.run_fixest(case)(the only live arm) + arun_within(case)stub — the documented Varying slopes: benchmark script vs R fixest (within pluggable) #65 seam to fill once the slope solver lands.Casedataclass carries the columns plusfe_terms(which factor holds which slopes) — the one structured thing both the R formula and the future within wiring derive from. Builders self-name from their params.benchmarks._problemspanel helpers; the DGP slope-injection math is adapted from the (defunct) prototype, and the R worker is recovered from the lost local harness.Case.formula()→worker[z1, z2] + year + firm); the R worker just fits the formula it's handed and printsRESULT_*markers (warmup + median-of-n viaproc.time(), plus an untimedfixef()slope-recovery sanity count).DGP catalog
Covers both the topology axis (where within's value lives) and the slope-count axis:
fixest_{simple,difficult}_v{1,3}— worker×year×firm panel, 1 vs 3 worker slopes, on i.i.d. (simple, R's home turf) vs band-structured (difficult, within's sweet spot) firm membership.akm_experience/akm_both— AKM mobility panel (Zipf firms, low mobility, clusters, largest component) with structured experience/tenure slopes.pixi wiring
R is kept out of the default dev/test/CI env via a dedicated
fixestfeature+environment (r-base,r-fixest,r-data.table). Run:The default env is unchanged.
Reference numbers (n=100k)
fixest_simple_v1fixest_simple_v3fixest_difficult_v1fixest_difficult_v3akm_experienceakm_bothThe simple→difficult jump on an identical panel (5 → 186 iters at V1) isolates topology as fixest's cost driver — exactly the regime within's preconditioned solve targets.
Verification
RESULT_FIXEFconfirms both slope components recovered on the dual-slope case.ruff check+ruff formatclean.Out of scope (per #65)
within's slope solver (#58/#59) — only the stub seam here; Julia arm; frozen fixtures / correctness assertions.
Closes #65.