Skip to content

feat(cache-proposals): cost-weighted threshold optimization#227

Open
jamby77 wants to merge 10 commits into
feature/cache-proposals-confidence-scorefrom
feature/cache-proposals-cost-weighted-threshold
Open

feat(cache-proposals): cost-weighted threshold optimization#227
jamby77 wants to merge 10 commits into
feature/cache-proposals-confidence-scorefrom
feature/cache-proposals-cost-weighted-threshold

Conversation

@jamby77
Copy link
Copy Markdown
Collaborator

@jamby77 jamby77 commented May 30, 2026

Stacked on #224 (confidence-score).

Summary

  • Records cost_saved_micros on similarity-window entries in the semantic cache, with retroactive population from later store() calls.
  • Feeds cost_saved_micros from similarity samples into the cache-proposals threshold recommendation engine.
  • Tier-gated cost-weighted threshold decisions (enterprise only).
  • Cost-aware reasoning text when cost-weighting fires; cost response fields scoped to the active direction.
  • Cleanup: zscore lookup, symmetric loosen cost fields, denominator note.

Closes the "Cost-Weighted Threshold Optimization" card on the BetterDB project board.

Test plan

  • semantic-cache cost-instrumentation tests pass
  • cache-proposals threshold recommendation tests cover cost-weighting path
  • enterprise tier gate verified (non-enterprise unaffected)
  • integration: end-to-end cost flow from store() → similarity sample → recommendation

Note

Medium Risk
Changes threshold tuning behavior for enterprise customers and adds best-effort Valkey bookkeeping on the hot check/store path; failures are swallowed but could skew cost metrics early in rollout.

Overview
Semantic cache now attaches cost_saved_micros to rolling similarity-window events (hits from entry cost_micros, misses as null), tracks pending misses in a __miss_pending zset keyed by prompt hash, and backfills miss rows when a later store() supplies token cost—so threshold analytics can see dollar impact on near-misses, not only on hits.

Cache proposals reads that field from the window, and on enterprise license only switches uncertain-hit / near-miss tighten and loosen decisions to cost-weighted rates (with optional USD fields and cost-aware reasoning); Pro and below keep count-based logic.

New unit tests cover instrumentation, pending-miss lifecycle, and tier-gated recommendations.

Reviewed by Cursor Bugbot for commit 394da28. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread packages/cache-benchmark-ts/src/adapters/betterdb.ts
Comment thread packages/cache-benchmark-ts/src/adapters/betterdb.ts
Comment thread packages/semantic-cache/src/SemanticCache.ts
Comment thread packages/cache-benchmark-ts/src/adapters/betterdb.ts
@jamby77 jamby77 force-pushed the feature/cache-proposals-confidence-score branch from 05305d7 to bbcd2c9 Compare May 30, 2026 09:42
jamby77 added 9 commits May 30, 2026 12:42
- Inject LicenseService into CacheReadonlyService constructor
- Compute cost-weighted uncertain-hit and near-miss rates from per-sample cost_saved_micros
- Use cost-weighted rates as trigger for paths 1 (uncertain_hits) and 3 (near_misses) on Enterprise tier
- Fall back to count-based rates for Pro+/Community or when costed samples < minSamples
- Populate cost_weighted_uncertain_hit_rate, cost_weighted_near_miss_rate, total_hit_cost_usd, uncertain_hit_cost_usd fields when cost-weighting was active
- Add StubLicenseService, seedSamplesWithCost helper, and 3 new tests (23 total)
@jamby77 jamby77 force-pushed the feature/cache-proposals-cost-weighted-threshold branch from d8438ad to 572b257 Compare May 30, 2026 09:43
Comment thread packages/semantic-cache/src/SemanticCache.ts
- Prune entries older than the 5-minute bound on every recordMissPending,
  so miss-only traffic cannot grow the bookkeeping zset unbounded
- Delete the miss-pending key in flush() alongside stats and window keys
- Add coverage for the record-time prune
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 394da28. Configure here.

Comment thread proprietary/cache-proposals/cache-readonly.service.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants