[Question] How to measure gain/savings?

[nisrulz]

• I was wondering if there is a way to see the savings/gains . Something like this

[DeusData]

Good question — there's no written guidance on measuring the token/time savings yet. Will put something together; thanks for raising it.

[koriyoshi2041]

Thanks for confirming there is no written guidance yet. I did a quick read through the existing measurement surfaces and it looks like there may be a small docs-first path here before adding any new command.

What I found:

• README already states two different savings frames: the preprint-level 10x fewer tokens / 2.1x fewer tool calls result, and the smaller five-query example of about 3.4K tokens vs about 412K.
• docs/BENCHMARK.md has the quality/attempt methodology, but it does not yet explain how a user should reproduce a "gain" measurement on their own repo.
• CBM_DIAGNOSTICS=1 already exposes query count/error/time stats through /tmp/cbm-diagnostics-<pid>.json, and scripts/soak-test.sh records latency/diagnostic CSVs, but those are closer to stability/perf metrics than token/tool savings.

Would a first useful contribution be a short docs/MEASURING_SAVINGS.md that separates:

1. quality: pass/partial/fail benchmark answer quality;
2. latency/stability: existing diagnostics and soak-test CSVs;
3. token/tool-call savings: how to compare a fixed question set against a file-by-file baseline, with caveats that CBM itself does not know the agent's final token count unless the client records it?

If that scope matches what you had in mind, I can draft the docs-only first pass and keep it tied to the existing benchmark/diagnostics surfaces rather than inventing a new CLI command prematurely.