Skip to content

feat(semantic-cache): cancel judgeFn on timeout via AbortSignal#220

Open
amitkojha05 wants to merge 1 commit into
BetterDB-inc:masterfrom
amitkojha05:feat/judge-abort-signal
Open

feat(semantic-cache): cancel judgeFn on timeout via AbortSignal#220
amitkojha05 wants to merge 1 commit into
BetterDB-inc:masterfrom
amitkojha05:feat/judge-abort-signal

Conversation

@amitkojha05
Copy link
Copy Markdown
Contributor

@amitkojha05 amitkojha05 commented May 21, 2026

Summary

JudgeOptions.timeoutMs races the judge promise against a timeout, but when it fires the in-flight LLM request keeps running in the background, burning API quota. The v0.5.0 JSDoc even acknowledged this: "use an AbortController inside judgeFn and abort it when the signal you manage fires" — but the library never passes the user a signal.

This PR closes that gap. The cache now creates one AbortController per judge invocation, passes controller.signal into judgeFn, and aborts it the instant timeoutMs fires. Forward input.signal to your LLM client and the in-flight request is cancelled on timeout instead of leaking.

Fully backward compatible — signal is additive. A judgeFn that ignores it behaves exactly as before. AbortController is global on Node ≥ 20 (package engine floor) — no new dependency.

Changes

  • types.tsjudgeFn input gains signal: AbortSignal; timeoutMs JSDoc rewritten to reflect the new abort behavior.
  • SemanticCache.tsraceWithTimeout gains onTimeout?: () => void callback (fires only on timeout branch); judge block creates controller, passes signal, aborts on timeout.
  • judge.test.ts — 4 new tests: signal supplied, signal fires on timeout, signal does NOT fire on normal resolution, backward-compat.
  • CHANGELOG.md, README.md, examples/judge/index.ts — updated.

Usage

async function judgeFn({ prompt, response, signal }) {
  const res = await openai.chat.completions.create(
    { model: 'gpt-4o-mini', messages: [/* ... */] },
    { signal }, // cancelled on timeout — no leaked request, no wasted quota
  );
  return res.choices[0].message.content?.trim() === 'YES';
}

@amitkojha05 amitkojha05 force-pushed the feat/judge-abort-signal branch from 15fbb9b to 4280b7a Compare May 21, 2026 21:58
@amitkojha05
Copy link
Copy Markdown
Contributor Author

Hi @KIvanow and @jamby77 ,Please review this PR ,would love to hear your feedback .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant