Skip to content

bench: varying-slopes reference benchmark vs R fixest (#65)#66

Merged
schroedk merged 1 commit into
feature/slopesfrom
slopes/65-fixest-benchmark
Jun 28, 2026
Merged

bench: varying-slopes reference benchmark vs R fixest (#65)#66
schroedk merged 1 commit into
feature/slopesfrom
slopes/65-fixest-benchmark

Conversation

@schroedk

@schroedk schroedk commented Jun 28, 2026

Copy link
Copy Markdown
Collaborator

What

A standalone, on-demand benchmark that times varying-slopes estimation on shared synthetic DGPs and prints R fixest's fit time + iteration counts. Captures reference numbers before within's slope solver exists (#58/#59), so the implementation slices can be measured against an external baseline.

Not a test: no correctness gating, run by hand.

Design (slim by intent)

  • One module benchmarks/slopes_fixest.py + one R worker benchmarks/fixest_fit.R.
  • No CLI. main() runs the fixed CASES catalog at a module-level SIZES constant; sweeping scale is a one-line edit.
  • No backend Protocol / classes. One run_fixest(case) (the only live arm) + a run_within(case) stub — the documented Varying slopes: benchmark script vs R fixest (within pluggable) #65 seam to fill once the slope solver lands.
  • One Case dataclass carries the columns plus fe_terms (which factor holds which slopes) — the one structured thing both the R formula and the future within wiring derive from. Builders self-name from their params.
  • Reuses the live benchmarks._problems panel helpers; the DGP slope-injection math is adapted from the (defunct) prototype, and the R worker is recovered from the lost local harness.
  • Python owns formula construction (Case.formula()worker[z1, z2] + year + firm); the R worker just fits the formula it's handed and prints RESULT_* markers (warmup + median-of-n via proc.time(), plus an untimed fixef() slope-recovery sanity count).

DGP catalog

Covers both the topology axis (where within's value lives) and the slope-count axis:

  • fixest_{simple,difficult}_v{1,3} — worker×year×firm panel, 1 vs 3 worker slopes, on i.i.d. (simple, R's home turf) vs band-structured (difficult, within's sweet spot) firm membership.
  • akm_experience / akm_both — AKM mobility panel (Zipf firms, low mobility, clusters, largest component) with structured experience/tenure slopes.

pixi wiring

R is kept out of the default dev/test/CI env via a dedicated fixest feature+environment (r-base, r-fixest, r-data.table). Run:

pixi run -e fixest bench-slopes     # first run provisions R + fixest

The default env is unchanged.

Reference numbers (n=100k)

case time_s iters
fixest_simple_v1 0.027 5/5
fixest_simple_v3 0.038 6/6
fixest_difficult_v1 0.444 186/153
fixest_difficult_v3 1.111 292/255
akm_experience 1.047 295/295
akm_both 3.626 895/975

The simple→difficult jump on an identical panel (5 → 186 iters at V1) isolates topology as fixest's cost driver — exactly the regime within's preconditioned solve targets.

Verification

  • R-skipped path: prints a clear "R fixest: skipped" message, no crash.
  • Full path: feols fits across the catalog (numbers above); RESULT_FIXEF confirms both slope components recovered on the dual-slope case.
  • ruff check + ruff format clean.

Out of scope (per #65)

within's slope solver (#58/#59) — only the stub seam here; Julia arm; frozen fixtures / correctness assertions.

Closes #65.

@schroedk schroedk force-pushed the slopes/65-fixest-benchmark branch from c7ea865 to ca84e65 Compare June 28, 2026 14:30
Standalone, on-demand perf comparison for the varying-slopes epic (#64),
timing slope estimation on shared synthetic DGPs before within's slope
solver exists. R fixest is the only live arm; within is a stub behind
--with-within, wired in once #58/#59 land.

- benchmarks/slopes_fixest.py: Case dataclass, fixest/akm slope DGP
  builders (adapted from the defunct prototype, reusing live
  benchmarks._problems panel helpers), DATASETS catalog, run_fixest +
  run_within stub, argparse CLI + table printer.
- benchmarks/fixest_fit.R: R worker (recovered ~verbatim from the lost
  local harness) parameterised over (csv, formula, n_repeat); warmup +
  median-of-n feols, fixef() slope recovery, RESULT_* markers.
- pixi.toml: dedicated fixest feature+env (r-base/r-fixest/r-data.table)
  so R stays out of the default dev/test/CI env; bench-slopes task.

Run: pixi run -e fixest bench-slopes
@schroedk schroedk force-pushed the slopes/65-fixest-benchmark branch from ca84e65 to 0318709 Compare June 28, 2026 14:45
@schroedk schroedk merged commit f4cd724 into feature/slopes Jun 28, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant