Jvmncs/dumber delta flash example

[jvmncs]

adds a dumb, heavily vibecoded, minimal example of doing delta compression between slime trainer Modal function and a Flash rollout server. most of the logic is in the modal app.

slime side changes:

• need a post-delta hook that runs volume.commit()
• need to supply external sglang url without port, just base url
• when running external sglang if /workers (or other sglang-router specific routes) can't be found, need to fallback accordingly

sglang side changes:

• None

autoinference deployment side:

• run a hook that reloads the volume. in the current impl, this catches requests to update_weights_from_disk and then calls vol.reload() before forwarding to the engine.

there's a lot I don't like here but it's just proof that it works the way we expect. some particular call outs that won't generalize past max_containers=1:

• is it justified to replace the port of http_server with an arbitrary proxy like this? does flash proxy make any assumptions about its port being the engine?
• fallback to external URL when /workers is unreachable. flash endpoints generally don't deploy with sglang_router, and when they do (e.g. for PD-disagg) we never use DP and instead horizontally scale each PD replica group behind the flash proxy. any assumptions made from this (e.g. that the endpoint represents a single engine rather than a container pool) are going to be flawed. it's a result of this jank, which seems to be required:

"--rollout-external",
"--rollout-num-gpus",
"1",
"--rollout-num-gpus-per-engine",
"1",
"--sglang-router-url",
rollout_url,
"--rollout-external-engine-addrs",
rollout_url,

rollout-external-engine-addrs is going to be dynamic over time, we can't set it at config-time.

• the modal app is verbose and jank, in particular local_entrypoint can steal the flash url from the deployed app? or at least I see warnings to that effect
• there's a lot of repeated code in delta_sidecar that needs to be pruned or replaced or refactored. did not want to spend too much time worrying about that

[nanjiangwill]

i think this is default None so there is no need to do getattr

[nanjiangwill]

ditto

[nanjiangwill]

i dont think this is needed if you are using the latest slime docker slimerl/slime:nightly-dev-20260527a

[nanjiangwill]

ideally dont have slime arg to start with sglang cuz it will parse to sglang ServerArg by removing sglang. which will make ServerArg for sglang to have router-url. if passing router-url is intended then there is no need to add this extra argument slime will automatically parse