State handling for static-shape inference engine - #160
Open
carinapeng wants to merge 4 commits into
Open
Conversation
carinapeng
force-pushed
the
carina/static-engine-runner-design
branch
2 times, most recently
from
August 10, 2026 22:47
6265330 to
50693ed
Compare
- Right-size KV cache per context bucket - Extend shared SyncInputHandler / InputContext (drop Static* fork) - Unit tests + gemma4/qwen static verified
carinapeng
force-pushed
the
carina/static-engine-runner-design
branch
from
August 10, 2026 22:57
50693ed to
35b1494
Compare
carinapeng
requested review from
alejandro-isaza,
blevine1,
kevchengcodes,
stikves and
tjia1818
August 11, 2026 19:04
carinapeng
force-pushed
the
carina/static-engine-runner-design
branch
4 times, most recently
from
August 12, 2026 16:55
4bbfe62 to
6af4562
Compare
carinapeng
force-pushed
the
carina/static-engine-runner-design
branch
from
August 12, 2026 16:58
6af4562 to
f8ed131
Compare
blevine1
reviewed
Aug 12, 2026
blevine1
reviewed
Aug 12, 2026
blevine1
left a comment
blevine1
left a comment
Contributor
There was a problem hiding this comment.
The changes looks good to me in general given the pre-existing design, but I'd consider zooming out and potentially changing some details of the "input handler" structure/interface at a later point. It is a pretty thick abstraction that'll limit what performance can be achieved.
Left some comments mostly about structure and readability.
carinapeng
force-pushed
the
carina/static-engine-runner-design
branch
from
August 12, 2026 23:43
9f2706b to
09f695d
Compare
Contributor
Author
|
@blevine1 Thanks Ben! Agree with your comments, the SyncInputHandler abstraction (existentials + prepare -> [String: NDArray]) is inherited, so I'd like to keep the interface change out of this PR. I'll file a follow up to revisit the handler structure / interface. Appreciate the review! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Static-shape LLM inference engine (StaticShapeEngine) built on the
bind(into:)state binding from #156. States are discovered by name and the KV cache is right-sized per context bucket; ctx is parsed from the function name. Input preparation is pluggable per model family via StaticInputProvider + StaticModelProfile; StaticInputContext composes the shared InputContext and the provider mirrors SyncInputHandler.Testing: LanguageModelsTests (317) pass; correct output on static assets.