Krea VACE 14b Pipeline #297
Conversation
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
BuffMcBigHuge
force-pushed
the
marco/feat/krea-vace-14b
branch
from
4bc5216 to
6d40951
Compare
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
| .cursorindexingignore | ||
|
|
||
| .cursor/ | ||
| .specstory/ |
There was a problem hiding this comment.
@yondonfu I can remove this if you like. Spec story is a plugin for cursor chat history export that I use.
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
|
FWIW, I've played with this PR and I'm seeing this weird loop. @ryanontheinside mentioned it's probably that we're sending the reference image to each chunk. demo_krea.mp4 |
| # - Only start skipping once the cache is in steady-state: | ||
| # current_start_frame >= kv_cache_num_frames | ||
| # - Recompute every N "blocks" (a block produces num_frame_per_block frames). | ||
| every_n = int(os.getenv("SCOPE_KV_CACHE_RECOMPUTE_EVERY", "1") or "1") |
There was a problem hiding this comment.
I'm not sure if we use env variables in other blocks, but I think we may want to try to use input params instead. This would help someone wanting to reuse the block or the pipeline to interact with it.
There was a problem hiding this comment.
this is interesting - if you do move to input params would be good so see some documentation explaining what different values can be used
There was a problem hiding this comment.
More testing needed - I'll remove it for now.
It could also be a problem with the vae after looking at another example. I had claude replicate the test_vace.py script from longlive and tested krea offline. The demarcation at chunk boundaries is familiar and makes me think vae issue. "She jumps": output_r2v.mp4I also tested longlive; there is no regression there. |
| WARMUP_PROMPT = [{"text": "a majestic sunset", "weight": 1.0}] | ||
|
|
||
|
|
||
| class KreaRealtimeVideoPipeline(Pipeline, LoRAEnabledPipeline): |
There was a problem hiding this comment.
Can we use VACEEnabledPipeline here instead of the lazy loading? We switched to the mixin recently
There was a problem hiding this comment.
I spent some time attempting this, but kept running into OOM on 5090. Any insight?
src/scope/server/frame_processor.py
Outdated
| # This fixes the chicken-and-egg problem where VACE isn't enabled until vace_input_frames arrives | ||
| vace_enabled = self.parameters.get("vace_enabled", False) | ||
|
|
||
| # Robustness fallback: If vace_enabled is missing (e.g. old frontend build), |
There was a problem hiding this comment.
we could expect the front end to match for the build here instead of adding fallbacks, wdyt?
| # - Only start skipping once the cache is in steady-state: | ||
| # current_start_frame >= kv_cache_num_frames | ||
| # - Recompute every N "blocks" (a block produces num_frame_per_block frames). | ||
| every_n = int(os.getenv("SCOPE_KV_CACHE_RECOMPUTE_EVERY", "1") or "1") |
There was a problem hiding this comment.
this is interesting - if you do move to input params would be good so see some documentation explaining what different values can be used
src/scope/core/pipelines/krea_realtime_video/blocks/recompute_kv_cache.py
Outdated
Show resolved
Hide resolved
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
This is a known limitation of image input with VACE and autoregression afaik. |
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
Signed-off-by: BuffMcBigHuge <marco@bymar.co>
|
I created a new PR with #311 |
This PR implements full VACE (Video All-In-One Creation and Editing) support for the KREA Realtime Video pipeline. It addresses significant architectural challenges in adapting the VACE system—originally designed for the 1.3B "Longlive" model—to work with the much larger Wan 2.1 14B model used by KREA.