feat(sae): SAE steering — delta clamps and full reconstruction#271
Draft
RhizoNymph wants to merge 327 commits into
Draft
feat(sae): SAE steering — delta clamps and full reconstruction#271RhizoNymph wants to merge 327 commits into
RhizoNymph wants to merge 327 commits into
Conversation
fix: stop writing phase-specific globals to shared steering buffers
…registration fix: register decode steering configs during lazy manager init
…phase-capacity fix: scheduler steering capacity counts only active phase
…k-hash fix: include decode steering hash in generated cache block keys
…ission check Co-authored-by: Claude
…g status Co-authored-by: Claude
…e tracking Co-authored-by: Claude
…gistration, and phase-specific norms
…sition fix: graceful steering transition + phase-tracked release
…idation fix: remove double validation and add phase-specific norms to steering status
…ion-capacity fix: transition-aware steering capacity and decode-start admission
Co-authored-by: Claude
Co-authored-by: Claude
fix: include decode hash in boundary block cache keys
fix: track req_id in deferred decode registrations to prevent leak
…obals fix: report pending phase-specific globals in steering status
docs: update steering docs for deferred cleanup and pre-init status
Co-authored-by: Claude
…alidation fix: validate matching dimensions between base and phase steering vectors
fix: swap steering hashes in InputBatch.swap_states()
…ropping Replace the silent fallback-to-global approach with a proper deferral mechanism. Failed registrations during lazy init and prefill-to-decode transitions are now appended to a unified `_pending_steering_registrations` list that is retried with priority before new admissions on every step. The retry loop also filters stale entries: - Entries for finished requests are dropped (prevents row leaks). - Entries whose request changed phase are dropped (e.g. prefill deferred but request already transitioned to decode). This subsumes the decode-only `_pending_decode_registrations` list and the row-leak fix from PR #46.
fix(sae-steering): full-recon row routing, hash separation, and scheduler capacity
# Conflicts: # .buildkite/ci_config.yaml # .buildkite/hardware_tests/amd.yaml # .buildkite/hardware_tests/cpu.yaml # .buildkite/intel_jobs/lora_intel.yaml # .buildkite/intel_jobs/misc_intel.yaml # .buildkite/intel_jobs/test-intel.yaml # .buildkite/performance-benchmarks/tests/serving-tests.json # .buildkite/release-pipeline.yaml # .buildkite/scripts/annotate-release.sh # .buildkite/scripts/hardware_ci/run-amd-test.sh # .buildkite/scripts/hardware_ci/run-cpu-test-arm.sh # .buildkite/scripts/hardware_ci/run-intel-test.sh # .buildkite/scripts/hardware_ci/run-xpu-test.sh # .buildkite/test-amd.yaml # .buildkite/test_areas/attention.yaml # .buildkite/test_areas/benchmarks.yaml # .buildkite/test_areas/disaggregated.yaml # .buildkite/test_areas/distributed.yaml # .buildkite/test_areas/e2e_integration.yaml # .buildkite/test_areas/engine.yaml # .buildkite/test_areas/entrypoints.yaml # .buildkite/test_areas/kernels.yaml # .buildkite/test_areas/lm_eval.yaml # .buildkite/test_areas/lora.yaml # .buildkite/test_areas/misc.yaml # .buildkite/test_areas/model_executor.yaml # .buildkite/test_areas/model_runner_v2.yaml # .buildkite/test_areas/models_basic.yaml # .buildkite/test_areas/models_language.yaml # .buildkite/test_areas/models_multimodal.yaml # .buildkite/test_areas/pytorch.yaml # .buildkite/test_areas/samplers.yaml # .buildkite/test_areas/spec_decode.yaml # .github/CODEOWNERS # .github/mergify.yml # .github/workflows/pre-commit.yml # .github/workflows/scripts/build.sh # .gitignore # .pre-commit-config.yaml # CMakeLists.txt # README.md # benchmarks/kernels/benchmark_router_gemm.py # benchmarks/multi_turn/benchmark_serving_multi_turn.py # cmake/cpu_extension.cmake # cmake/external_projects/deepgemm.cmake # cmake/external_projects/qutlass.cmake # cmake/external_projects/vllm_flash_attn.cmake # csrc/attention/dtype_fp8.cuh # csrc/attention/merge_attn_states.cu # csrc/cache.h # csrc/cpu/cpu_attn.cpp # csrc/cpu/cpu_attn_impl.hpp # csrc/cpu/cpu_attn_vec.hpp # csrc/cpu/cpu_fused_moe.cpp # csrc/cpu/cpu_types_riscv_defs.hpp # csrc/cpu/cpu_types_riscv_impl.hpp # csrc/cpu/generate_cpu_attn_dispatch.py # csrc/cpu/sgl-kernels/gemm.h # csrc/cpu/sgl-kernels/gemm_int4.cpp # csrc/cpu/torch_bindings.cpp # csrc/cpu/utils.hpp # csrc/cutlass_extensions/torch_utils.hpp # csrc/fused_qknorm_rope_kernel.cu # csrc/libtorch_stable/activation_kernels.cu # csrc/libtorch_stable/cache_kernels.cu # csrc/libtorch_stable/cutlass_extensions/epilogue/scaled_mm_epilogues_c2x.hpp # csrc/libtorch_stable/dispatch_utils.h # csrc/libtorch_stable/layernorm_kernels.cu # csrc/libtorch_stable/layernorm_quant_kernels.cu # csrc/libtorch_stable/moe/marlin_moe_wna16/marlin_template.h # csrc/libtorch_stable/moe/marlin_moe_wna16/ops.cu # csrc/libtorch_stable/moe/torch_bindings.cpp # csrc/libtorch_stable/ops.h # csrc/libtorch_stable/quantization/fp4/activation_nvfp4_quant_fusion_kernels.cu # csrc/libtorch_stable/quantization/fp4/mxfp4_blockwise_moe_kernel.cu # csrc/libtorch_stable/quantization/fp4/mxfp4_experts_quant.cu # csrc/libtorch_stable/quantization/fp4/nvfp4_blockwise_moe_kernel.cu # csrc/libtorch_stable/quantization/fp4/nvfp4_experts_quant.cu # csrc/libtorch_stable/quantization/fp4/nvfp4_quant_entry.cu # csrc/libtorch_stable/quantization/fp4/nvfp4_quant_kernels.cu # csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_entry.cu # csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_kernels.cu # csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu # csrc/libtorch_stable/quantization/fused_kernels/fused_silu_mul_block_quant.cu # csrc/libtorch_stable/quantization/fused_kernels/quant_conversions.cuh # csrc/libtorch_stable/quantization/marlin/marlin.cu # csrc/libtorch_stable/quantization/marlin/marlin_template.h # csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm90_fp8_dispatch.cuh # csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm100_fp8.cu # csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm100_fp8_dispatch.cuh # csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8.cu # csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_fp8.cu # csrc/libtorch_stable/quantization/w8a8/cutlass/c3x/scaled_mm_sm90_fp8_dispatch.cuh # csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_c2x.cu # csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_entry.cu # csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu # csrc/libtorch_stable/sampler.cu # csrc/libtorch_stable/torch_bindings.cpp # csrc/minimax_reduce_rms_kernel.h # csrc/moe/moe_ops.h # csrc/ops.h # csrc/pos_encoding_kernels.cu # csrc/quantization/w8a8/fp8/amd/quant_utils.cuh # csrc/quantization/w8a8/fp8/nvidia/quant_utils.cuh # csrc/topk.cu # csrc/torch_bindings.cpp # docker/Dockerfile # docker/Dockerfile.cpu # docker/Dockerfile.nightly_torch # docker/Dockerfile.rocm # docker/Dockerfile.rocm_base # docker/Dockerfile.xpu # docker/versions.json # docs/OVERVIEW.md # docs/assets/contributing/dockerfile-stages-dependency.png # docs/benchmarking/cli.md # docs/contributing/profiling.md # docs/design/attention_backends.md # docs/design/capture_consumers.md # docs/design/cuda_graphs_multimodal.md # docs/design/moe_kernel_features.md # docs/features/batch_invariance.md # docs/features/capture_consumers.md # docs/features/disagg_prefill.md # docs/features/quantization/online.md # docs/features/speculative_decoding/README.md # docs/features/steering.md # docs/getting_started/installation/cpu.arm.inc.md # docs/getting_started/installation/cpu.x86.inc.md # docs/getting_started/installation/gpu.cuda.inc.md # docs/getting_started/installation/gpu.xpu.inc.md # docs/models/pooling_models/README.md # docs/models/pooling_models/reward.md # docs/models/pooling_models/token_classify.md # docs/models/pooling_models/token_embed.md # docs/models/supported_models.md # docs/serving/openai_compatible_server.md # docs/training/layerwise.md # docs/usage/security.md # examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py # examples/disaggregated/disaggregated_serving/example_mm_serve.py # examples/features/speculative_decoding/extract_hidden_states_offline.py # examples/generate/multimodal/vision_language_multi_image_offline.py # examples/generate/multimodal/vision_language_offline.py # examples/online_serving/disaggregated_serving/moriio_toy_proxy_server.py # examples/speech_to_text/lid/openai_lid_client.py # examples/speech_to_text/openai/openai_transcription_client.py # examples/tool_chat_template_gemma4.jinja # pyproject.toml # requirements/build/cpu.txt # requirements/build/cuda.txt # requirements/common.txt # requirements/cuda.txt # requirements/kv_connectors.txt # requirements/rocm.txt # requirements/test/cuda.in # requirements/test/cuda.txt # requirements/test/nightly-torch.txt # requirements/test/rocm.in # requirements/test/rocm.txt # requirements/test/xpu.in # requirements/test/xpu.txt # requirements/tpu.txt # requirements/xpu.txt # setup.py # tests/basic_correctness/test_mem.py # tests/compile/conftest.py # tests/compile/fusions_e2e/conftest.py # tests/compile/fusions_e2e/test_tp2_ar_rms.py # tests/compile/passes/distributed/test_fusion_all_reduce.py # tests/compile/passes/ir/test_clone_cleanup.py # tests/compile/passes/test_rope_kvcache_fusion.py # tests/compile/test_dynamic_shapes_compilation.py # tests/conftest.py # tests/distributed/test_dcp_a2a.py # tests/distributed/test_eplb_execute.py # tests/distributed/test_kv_cache_events.py # tests/distributed/test_mnnvl_alltoall.py # tests/distributed/test_ray_v2_executor.py # tests/distributed/test_torchrun_example.py # tests/distributed/test_torchrun_example_moe.py # tests/engine/test_arg_utils.py # tests/entrypoints/openai/chat_completion/test_chat_completion_with_prompt_embeds.py # tests/entrypoints/openai/chat_completion/test_chat_error.py # tests/entrypoints/openai/chat_completion/test_serving_chat.py # tests/entrypoints/openai/chat_completion/test_thinking_token_budget.py # tests/entrypoints/openai/completion/test_shutdown.py # tests/entrypoints/openai/correctness/test_transcription_api_correctness.py # tests/entrypoints/openai/responses/test_function_call.py # tests/entrypoints/openai/responses/test_responses_utils.py # tests/entrypoints/openai/responses/test_serving_responses.py # tests/entrypoints/openai/test_capture_protocol.py # tests/entrypoints/openai/test_openai_schema.py # tests/entrypoints/openai/test_steering_modules.py # tests/entrypoints/openai/test_steering_protocol.py # tests/entrypoints/pooling/embed/test_io_processor.py # tests/entrypoints/serve/disagg/test_generate_stream.py # tests/entrypoints/serve/disagg/test_serving_tokens.py # tests/entrypoints/serve/steering/test_api_router.py # tests/entrypoints/serve/steering/test_api_router_distributed.py # tests/entrypoints/serve/steering/test_protocol.py # tests/entrypoints/serve/steering/test_worker_steering.py # tests/entrypoints/speech_to_text/transcription/test_transcription_inter_chunk_spacing.py # tests/evals/gsm8k/configs/models-mi3xx.txt # tests/evals/gsm8k/configs/models-qwen35-mi355.txt # tests/ir/test_op.py # tests/kernels/attention/test_attention_selector.py # tests/kernels/attention/test_cpu_attn.py # tests/kernels/attention/test_lightning_attn.py # tests/kernels/core/test_fused_q_kv_rmsnorm.py # tests/kernels/core/test_layernorm.py # tests/kernels/core/test_minimax_reduce_rms.py # tests/kernels/helion/test_register.py # tests/kernels/ir/test_layernorm.py # tests/kernels/mamba/test_causal_conv1d.py # tests/kernels/mamba/test_mamba_ssm.py # tests/kernels/mamba/test_ssu_dispatch.py # tests/kernels/moe/modular_kernel_tools/mk_objects.py # tests/kernels/moe/modular_kernel_tools/parallel_utils.py # tests/kernels/moe/test_batched_deepgemm.py # tests/kernels/moe/test_cpu_fused_moe.py # tests/kernels/moe/test_cutlass_moe.py # tests/kernels/moe/test_deepgemm.py # tests/kernels/moe/test_moe.py # tests/kernels/moe/test_moe_layer.py # tests/kernels/moe/test_moe_weight_loading_padded.py # tests/kernels/moe/test_mxfp4_moe.py # tests/kernels/moe/test_topk_softplus_sqrt.py # tests/kernels/moe/test_trtllm_nvfp4_moe.py # tests/kernels/moe/test_unquantized_backend_selection.py # tests/kernels/moe/test_zero_expert_moe.py # tests/kernels/moe/utils.py # tests/kernels/quantization/test_mxfp4_triton_ep.py # tests/kernels/quantization/test_per_token_group_quant.py # tests/kernels/test_awq_int4_to_int8.py # tests/kernels/test_compressor_kv_cache.py # tests/kernels/test_flex_attention.py # tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py # tests/kernels/test_fused_indexer_q_rope_quant.py # tests/kernels/test_fused_inv_rope_fp8_quant.py # tests/kernels/test_kda.py # tests/lora/test_layers.py # tests/lora/test_lora_manager.py # tests/lora/test_olmoe_tp.py # tests/lora/test_worker.py # tests/model_executor/layers/test_steering_any_active.py # tests/model_executor/layers/test_steering_op.py # tests/model_executor/model_loader/test_reload.py # tests/model_executor/test_routed_experts_capture.py # tests/models/language/pooling/test_colbert.py # tests/models/language/pooling/test_max_tokens_per_doc.py # tests/models/language/pooling/test_nomic_max_model_len.py # tests/models/multimodal/generation/test_nemotron_parse.py # tests/models/multimodal/generation/test_qwen2_5_vl.py # tests/models/multimodal/generation/test_vit_cudagraph.py # tests/models/multimodal/generation/vlm_utils/model_utils.py # tests/models/multimodal/processing/test_gemma4.py # tests/models/quantization/test_gpt_oss.py # tests/models/quantization/test_nvfp4.py # tests/models/registry.py # tests/models/test_deepseek_v4_mega_moe.py # tests/models/utils.py # tests/multimodal/test_video.py # tests/quantization/test_compressed_tensors.py # tests/quantization/test_cpu_wna16.py # tests/quantization/test_cutlass_w4a16.py # tests/quantization/test_gfx950_moe.py # tests/quantization/test_mi3xx_moe.py # tests/quantization/test_modelopt.py # tests/quantization/test_online.py # tests/quantization/test_quark.py # tests/quantization/test_turboquant.py # tests/reasoning/test_gemma4_reasoning_parser.py # tests/renderers/test_gemma4_chat_template.py # tests/test_config.py # tests/tokenizers_/test_mistral.py # tests/tool_parsers/test_deepseekv32_tool_parser.py # tests/tool_parsers/test_deepseekv4_tool_parser.py # tests/tool_parsers/test_gemma4_tool_parser.py # tests/tool_parsers/test_glm4_moe_tool_parser.py # tests/tool_parsers/test_minimax_m2_tool_parser.py # tests/tool_parsers/test_mistral_tool_parser.py # tests/tool_parsers/test_qwen3coder_tool_parser.py # tests/tool_parsers/test_qwen3xml_tool_parser.py # tests/tool_use/test_gemma4_responses_adjust_request.py # tests/transformers_utils/test_utils.py # tests/utils.py # tests/utils_/test_numa_utils.py # tests/v1/attention/test_mla_prefill_selector.py # tests/v1/attention/utils.py # tests/v1/capture/consumers/filesystem/test_consumer.py # tests/v1/capture/consumers/test_logging.py # tests/v1/capture/test_config.py # tests/v1/capture/test_consumer_base.py # tests/v1/capture/test_driver_bridge.py # tests/v1/capture/test_manager.py # tests/v1/capture/test_multi_consumer_runner.py # tests/v1/capture/test_plan.py # tests/v1/capture/test_registry.py # tests/v1/capture/test_runner_integration.py # tests/v1/capture/test_sampling_params.py # tests/v1/capture/test_types.py # tests/v1/core/test_async_scheduler.py # tests/v1/core/test_prefix_caching.py # tests/v1/core/test_scheduler.py # tests/v1/core/test_single_type_kv_cache_manager.py # tests/v1/core/test_steering_hash_determinism.py # tests/v1/core/test_steering_scheduler.py # tests/v1/cudagraph/test_cudagraph_dispatch.py # tests/v1/cudagraph/test_encoder_cudagraph.py # tests/v1/determinism/test_batch_invariance.py # tests/v1/determinism/test_rms_norm_batch_invariant.py # tests/v1/e2e/general/test_mamba_prefix_cache.py # tests/v1/e2e/spec_decode/test_async_spec_decode.py # tests/v1/e2e/spec_decode/test_spec_decode.py # tests/v1/engine/test_input_processor_steering.py # tests/v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh # tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py # tests/v1/kv_connector/unit/offloading_connector/test_worker_metadata.py # tests/v1/kv_connector/unit/offloading_connector/utils.py # tests/v1/kv_connector/unit/test_bidirectional_kv_transfer.py # tests/v1/kv_connector/unit/test_mooncake_connector.py # tests/v1/kv_connector/unit/test_mooncake_connector_hma.py # tests/v1/kv_connector/unit/test_moriio_connector.py # tests/v1/kv_connector/unit/test_multi_connector.py # tests/v1/kv_connector/unit/test_nixl_connector.py # tests/v1/kv_connector/unit/test_nixl_connector_hma.py # tests/v1/kv_connector/unit/test_offloading_connector.py # tests/v1/kv_connector/unit/test_remote_prefill_lifecycle.py # tests/v1/kv_connector/unit/test_rixl_gpu_mem_diag.py # tests/v1/kv_connector/unit/utils.py # tests/v1/kv_offload/cpu/test_gpu_worker.py # tests/v1/kv_offload/cpu/test_manager.py # tests/v1/kv_offload/cpu/test_shared_offload_region.py # tests/v1/logits_processors/test_correctness.py # tests/v1/sample/test_rejection_sampler.py # tests/v1/sample/test_topk_topp_sampler.py # tests/v1/simple_kv_offload/test_integration.py # tests/v1/simple_kv_offload/test_scheduler.py # tests/v1/spec_decode/test_acceptance_length.py # tests/v1/spec_decode/test_eagle.py # tests/v1/spec_decode/test_tree_attention.py # tests/v1/streaming_input/test_gpu_model_runner_streaming.py # tests/v1/test_request_steering.py # tests/v1/test_steering_inline_packed.py # tests/v1/test_steering_types.py # tests/v1/worker/test_gpu_input_batch.py # tests/v1/worker/test_gpu_model_runner.py # tests/v1/worker/test_steering_manager.py # tests/v1/worker/test_steering_manager_ownership.py # tests/v1/worker/test_steering_named_resolve_cache.py # tools/pre_commit/generate_attention_backend_docs.py # vllm/_aiter_ops.py # vllm/_custom_ops.py # vllm/_xpu_ops.py # vllm/benchmarks/datasets/__init__.py # vllm/benchmarks/datasets/datasets.py # vllm/compilation/backends.py # vllm/compilation/caching.py # vllm/compilation/codegen.py # vllm/compilation/passes/fusion/allreduce_rms_fusion.py # vllm/compilation/passes/fusion/collective_fusion.py # vllm/compilation/passes/fusion/matcher_utils.py # vllm/compilation/passes/fusion/rms_quant_fusion.py # vllm/compilation/passes/fusion/rocm_aiter_fusion.py # vllm/compilation/passes/fusion/rope_kvcache_fusion.py # vllm/compilation/passes/fusion/sequence_parallelism.py # vllm/compilation/passes/inductor_pass.py # vllm/compilation/passes/ir/clone_elimination.py # vllm/compilation/passes/pass_manager.py # vllm/compilation/passes/vllm_inductor_pass.py # vllm/compilation/wrapper.py # vllm/config/__init__.py # vllm/config/attention.py # vllm/config/cache.py # vllm/config/compilation.py # vllm/config/kernel.py # vllm/config/model.py # vllm/config/parallel.py # vllm/config/pooler.py # vllm/config/quantization.py # vllm/config/speculative.py # vllm/config/steering.py # vllm/config/steering_types.py # vllm/config/vllm.py # vllm/distributed/device_communicators/all2all.py # vllm/distributed/device_communicators/cuda_communicator.py # vllm/distributed/device_communicators/xpu_communicator.py # vllm/distributed/elastic_ep/elastic_execute.py # vllm/distributed/eplb/async_worker.py # vllm/distributed/eplb/eplb_communicator.py # vllm/distributed/eplb/eplb_state.py # vllm/distributed/eplb/eplb_utils.py # vllm/distributed/eplb/rebalance_execute.py # vllm/distributed/kv_events.py # vllm/distributed/kv_transfer/kv_connector/factory.py # vllm/distributed/kv_transfer/kv_connector/utils.py # vllm/distributed/kv_transfer/kv_connector/v1/lmcache_mp_connector.py # vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py # vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py # vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py # vllm/distributed/kv_transfer/kv_connector/v1/nixl/__init__.py # vllm/distributed/kv_transfer/kv_connector/v1/nixl/connector.py # vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py # vllm/distributed/kv_transfer/kv_connector/v1/nixl/scheduler.py # vllm/distributed/kv_transfer/kv_connector/v1/nixl/stats.py # vllm/distributed/kv_transfer/kv_connector/v1/nixl/utils.py # vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py # vllm/distributed/kv_transfer/kv_connector/v1/offloading/common.py # vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py # vllm/distributed/kv_transfer/kv_connector/v1/offloading/worker.py # vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py # vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py # vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py # vllm/distributed/nixl_utils.py # vllm/engine/arg_utils.py # vllm/engine/protocol.py # vllm/entrypoints/anthropic/protocol.py # vllm/entrypoints/anthropic/serving.py # vllm/entrypoints/chat_utils.py # vllm/entrypoints/cli/serve.py # vllm/entrypoints/generate/factories.py # vllm/entrypoints/llm.py # vllm/entrypoints/openai/api_server.py # vllm/entrypoints/openai/chat_completion/batch_serving.py # vllm/entrypoints/openai/chat_completion/protocol.py # vllm/entrypoints/openai/chat_completion/serving.py # vllm/entrypoints/openai/cli_args.py # vllm/entrypoints/openai/completion/protocol.py # vllm/entrypoints/openai/completion/serving.py # vllm/entrypoints/openai/engine/protocol.py # vllm/entrypoints/openai/engine/serving.py # vllm/entrypoints/openai/models/protocol.py # vllm/entrypoints/openai/parser/harmony_utils.py # vllm/entrypoints/openai/parser/responses_parser.py # vllm/entrypoints/openai/responses/context.py # vllm/entrypoints/openai/responses/protocol.py # vllm/entrypoints/openai/responses/serving.py # vllm/entrypoints/openai/responses/streaming_events.py # vllm/entrypoints/openai/steering/registry.py # vllm/entrypoints/pooling/base/serving.py # vllm/entrypoints/pooling/classify/api_router.py # vllm/entrypoints/pooling/classify/protocol.py # vllm/entrypoints/pooling/embed/api_router.py # vllm/entrypoints/pooling/embed/io_processor.py # vllm/entrypoints/pooling/embed/protocol.py # vllm/entrypoints/pooling/embed/serving.py # vllm/entrypoints/pooling/factories.py # vllm/entrypoints/pooling/pooling/api_router.py # vllm/entrypoints/pooling/pooling/serving.py # vllm/entrypoints/pooling/scoring/api_router.py # vllm/entrypoints/pooling/scoring/io_processor.py # vllm/entrypoints/pooling/scoring/protocol.py # vllm/entrypoints/pooling/utils.py # vllm/entrypoints/serve/__init__.py # vllm/entrypoints/serve/disagg/protocol.py # vllm/entrypoints/serve/disagg/serving.py # vllm/entrypoints/serve/render/serving.py # vllm/entrypoints/serve/sagemaker/api_router.py # vllm/entrypoints/serve/steering/_merge.py # vllm/entrypoints/serve/steering/api_router.py # vllm/entrypoints/serve/steering/modules_protocol.py # vllm/entrypoints/serve/steering/modules_router.py # vllm/entrypoints/serve/steering/protocol.py # vllm/entrypoints/speech_to_text/base/serving.py # vllm/entrypoints/speech_to_text/transcription/protocol.py # vllm/entrypoints/speech_to_text/transcription/serving.py # vllm/env_override.py # vllm/envs.py # vllm/ir/__init__.py # vllm/ir/op.py # vllm/ir/util.py # vllm/kernels/helion/register.py # vllm/kernels/helion/utils.py # vllm/kernels/vllm_c.py # vllm/kernels/xpu_ops.py # vllm/lora/layers/fused_moe.py # vllm/lora/punica_wrapper/punica_base.py # vllm/lora/worker_manager.py # vllm/model_executor/kernels/linear/__init__.py # vllm/model_executor/kernels/linear/base.py # vllm/model_executor/kernels/linear/mixed_precision/__init__.py # vllm/model_executor/kernels/linear/mixed_precision/triton_w4a16.py # vllm/model_executor/kernels/linear/mxfp8/emulation.py # vllm/model_executor/kernels/linear/mxfp8/flashinfer.py # vllm/model_executor/kernels/linear/nvfp4/base.py # vllm/model_executor/kernels/linear/nvfp4/cutlass.py # vllm/model_executor/kernels/linear/nvfp4/flashinfer.py # vllm/model_executor/kernels/linear/scaled_mm/BlockScaledMMLinearKernel.py # vllm/model_executor/kernels/linear/scaled_mm/__init__.py # vllm/model_executor/kernels/linear/scaled_mm/aiter.py # vllm/model_executor/kernels/linear/scaled_mm/cutlass.py # vllm/model_executor/kernels/linear/scaled_mm/flashinfer.py # vllm/model_executor/kernels/linear/scaled_mm/triton.py # vllm/model_executor/kernels/linear/scaled_mm/xpu.py # vllm/model_executor/layers/activation.py # vllm/model_executor/layers/activation_capture.py # vllm/model_executor/layers/attention/attention.py # vllm/model_executor/layers/attention/mla_attention.py # vllm/model_executor/layers/attention/mm_encoder_attention.py # vllm/model_executor/layers/batch_invariant.py # vllm/model_executor/layers/fla/ops/chunk.py # vllm/model_executor/layers/fla/ops/chunk_delta_h.py # vllm/model_executor/layers/fla/ops/chunk_o.py # vllm/model_executor/layers/fla/ops/kda.py # vllm/model_executor/layers/fused_moe/__init__.py # vllm/model_executor/layers/fused_moe/all2all_utils.py # vllm/model_executor/layers/fused_moe/config.py # vllm/model_executor/layers/fused_moe/cpu_fused_moe.py # vllm/model_executor/layers/fused_moe/experts/cutlass_moe.py # vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe.py # vllm/model_executor/layers/fused_moe/experts/flashinfer_cutedsl_batched_moe.py # vllm/model_executor/layers/fused_moe/experts/flashinfer_cutedsl_moe.py # vllm/model_executor/layers/fused_moe/experts/fused_batched_moe.py # vllm/model_executor/layers/fused_moe/experts/gpt_oss_triton_kernels_moe.py # vllm/model_executor/layers/fused_moe/experts/marlin_moe.py # vllm/model_executor/layers/fused_moe/experts/nvfp4_emulation_moe.py # vllm/model_executor/layers/fused_moe/experts/ocp_mx_emulation_moe.py # vllm/model_executor/layers/fused_moe/experts/triton_cutlass_moe.py # vllm/model_executor/layers/fused_moe/experts/trtllm_bf16_moe.py # vllm/model_executor/layers/fused_moe/experts/trtllm_fp8_moe.py # vllm/model_executor/layers/fused_moe/experts/trtllm_mxfp4_moe.py # vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py # vllm/model_executor/layers/fused_moe/experts/xpu_moe.py # vllm/model_executor/layers/fused_moe/fused_moe.py # vllm/model_executor/layers/fused_moe/fused_moe_method_base.py # vllm/model_executor/layers/fused_moe/fused_moe_modular_method.py # vllm/model_executor/layers/fused_moe/layer.py # vllm/model_executor/layers/fused_moe/modular_kernel.py # vllm/model_executor/layers/fused_moe/oracle/fp8.py # vllm/model_executor/layers/fused_moe/oracle/int8.py # vllm/model_executor/layers/fused_moe/oracle/int_wna16.py # vllm/model_executor/layers/fused_moe/oracle/mxfp4.py # vllm/model_executor/layers/fused_moe/oracle/mxfp8.py # vllm/model_executor/layers/fused_moe/oracle/nvfp4.py # vllm/model_executor/layers/fused_moe/oracle/unquantized.py # vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_one_sided.py # vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_two_sided.py # vllm/model_executor/layers/fused_moe/prepare_finalize/naive_dp_ep.py # vllm/model_executor/layers/fused_moe/prepare_finalize/no_dp_ep.py # vllm/model_executor/layers/fused_moe/routed_experts_capturer.py # vllm/model_executor/layers/fused_moe/router/base_router.py # vllm/model_executor/layers/fused_moe/router/custom_routing_router.py # vllm/model_executor/layers/fused_moe/router/fused_moe_router.py # vllm/model_executor/layers/fused_moe/router/fused_topk_bias_router.py # vllm/model_executor/layers/fused_moe/router/gate_linear.py # vllm/model_executor/layers/fused_moe/router/router_factory.py # vllm/model_executor/layers/fused_moe/router/zero_expert_router.py # vllm/model_executor/layers/fused_moe/runner/moe_runner.py # vllm/model_executor/layers/fused_moe/runner/moe_runner_interface.py # vllm/model_executor/layers/fused_moe/runner/shared_experts.py # vllm/model_executor/layers/fused_moe/unquantized_fused_moe_method.py # vllm/model_executor/layers/fused_moe/utils.py # vllm/model_executor/layers/layernorm.py # vllm/model_executor/layers/linear.py # vllm/model_executor/layers/mamba/gdn/kimi_gdn_linear_attn.py # vllm/model_executor/layers/mamba/gdn_linear_attn.py # vllm/model_executor/layers/mamba/linear/minimax_linear_attn.py # vllm/model_executor/layers/mamba/mamba_utils.py # vllm/model_executor/layers/mamba/ops/mamba_ssm.py # vllm/model_executor/layers/mamba/ops/ssu_dispatch.py # vllm/model_executor/layers/mhc.py # vllm/model_executor/layers/pooler/seqwise/methods.py # vllm/model_executor/layers/pooler/seqwise/poolers.py # vllm/model_executor/layers/pooler/tokwise/methods.py # vllm/model_executor/layers/pooler/tokwise/poolers.py # vllm/model_executor/layers/quantization/__init__.py # vllm/model_executor/layers/quantization/auto_gptq.py # vllm/model_executor/layers/quantization/awq_marlin.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w4a4_mxfp4.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w4a4_nvfp4.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w4a8_fp8.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w4a8_int8.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w8a8_fp8.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w8a8_int8.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w8a8_mxfp8.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_wna16.py # vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_wna16_marlin.py # vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4.py # vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py # vllm/model_executor/layers/quantization/cpu_wna16.py # vllm/model_executor/layers/quantization/experts_int8.py # vllm/model_executor/layers/quantization/fp8.py # vllm/model_executor/layers/quantization/gguf.py # vllm/model_executor/layers/quantization/gptq.py # vllm/model_executor/layers/quantization/humming.py # vllm/model_executor/layers/quantization/inc.py # vllm/model_executor/layers/quantization/inc/__init__.py # vllm/model_executor/layers/quantization/modelopt.py # vllm/model_executor/layers/quantization/mxfp4.py # vllm/model_executor/layers/quantization/online/base.py # vllm/model_executor/layers/quantization/online/fp8.py # vllm/model_executor/layers/quantization/online/int8.py # vllm/model_executor/layers/quantization/online/moe_base.py # vllm/model_executor/layers/quantization/online/mxfp8.py # vllm/model_executor/layers/quantization/quark/quark.py # vllm/model_executor/layers/quantization/quark/quark_moe.py # vllm/model_executor/layers/quantization/turboquant/__init__.py # vllm/model_executor/layers/quantization/turboquant/config.py # vllm/model_executor/layers/quantization/utils/flashinfer_fp4_moe.py # vllm/model_executor/layers/quantization/utils/flashinfer_utils.py # vllm/model_executor/layers/quantization/utils/fp8_utils.py # vllm/model_executor/layers/quantization/utils/marlin_utils_fp8.py # vllm/model_executor/layers/quantization/utils/mxfp8_utils.py # vllm/model_executor/layers/quantization/utils/quant_utils.py # vllm/model_executor/layers/rotary_embedding/deepseek_scaling_rope.py # vllm/model_executor/layers/sparse_attn_indexer.py # vllm/model_executor/layers/steering.py # vllm/model_executor/layers/steering_kernel.py # vllm/model_executor/layers/utils.py # vllm/model_executor/model_loader/default_loader.py # vllm/model_executor/model_loader/gguf_loader.py # vllm/model_executor/model_loader/reload/layerwise.py # vllm/model_executor/model_loader/reload/meta.py # vllm/model_executor/model_loader/reload/utils.py # vllm/model_executor/model_loader/weight_utils.py # vllm/model_executor/models/AXK1.py # vllm/model_executor/models/afmoe.py # vllm/model_executor/models/apertus.py # vllm/model_executor/models/arcee.py # vllm/model_executor/models/arctic.py # vllm/model_executor/models/aria.py # vllm/model_executor/models/baichuan.py # vllm/model_executor/models/bailing_moe_linear.py # vllm/model_executor/models/commandr.py # vllm/model_executor/models/config.py # vllm/model_executor/models/dbrx.py # vllm/model_executor/models/deepseek_mtp.py # vllm/model_executor/models/deepseek_v2.py # vllm/model_executor/models/dots1.py # vllm/model_executor/models/ernie45_moe.py # vllm/model_executor/models/exaone.py # vllm/model_executor/models/exaone4.py # vllm/model_executor/models/exaone4_5.py # vllm/model_executor/models/exaone4_5_mtp.py # vllm/model_executor/models/exaone_moe.py # vllm/model_executor/models/falcon.py # vllm/model_executor/models/flex_olmo.py # vllm/model_executor/models/gemma.py # vllm/model_executor/models/gemma2.py # vllm/model_executor/models/gemma3.py # vllm/model_executor/models/gemma3n.py # vllm/model_executor/models/gemma4.py # vllm/model_executor/models/gemma4_mm.py # vllm/model_executor/models/glm4.py # vllm/model_executor/models/glm4_moe.py # vllm/model_executor/models/glm4_moe_lite.py # vllm/model_executor/models/glm4_moe_lite_mtp.py # vllm/model_executor/models/glm4_moe_mtp.py # vllm/model_executor/models/gpt_neox.py # vllm/model_executor/models/granite.py # vllm/model_executor/models/granite4_vision.py # vllm/model_executor/models/granitemoe.py # vllm/model_executor/models/granitemoeshared.py # vllm/model_executor/models/grok1.py # vllm/model_executor/models/hunyuan_v1.py # vllm/model_executor/models/hy_v3.py # vllm/model_executor/models/hy_v3_mtp.py # vllm/model_executor/models/hyperclovax.py # vllm/model_executor/models/interfaces.py # vllm/model_executor/models/internlm2.py # vllm/model_executor/models/internlm2_ve.py # vllm/model_executor/models/interns1_pro.py # vllm/model_executor/models/iquest_loopcoder.py # vllm/model_executor/models/jais2.py # vllm/model_executor/models/kimi_linear.py # vllm/model_executor/models/laguna.py # vllm/model_executor/models/llama.py # vllm/model_executor/models/llama4.py # vllm/model_executor/models/mimo_v2.py # vllm/model_executor/models/mimo_v2_mtp.py # vllm/model_executor/models/minicpm.py # vllm/model_executor/models/minimax_m2.py # vllm/model_executor/models/minimax_text_01.py # vllm/model_executor/models/mistral.py # vllm/model_executor/models/mistral_eagle.py # vllm/model_executor/models/mixtral.py # vllm/model_executor/models/molmo.py # vllm/model_executor/models/molmo2.py # vllm/model_executor/models/nemotron.py # vllm/model_executor/models/nemotron_nas.py # vllm/model_executor/models/olmo.py # vllm/model_executor/models/olmo2.py # vllm/model_executor/models/olmo_hybrid.py # vllm/model_executor/models/olmoe.py # vllm/model_executor/models/openpangu.py # vllm/model_executor/models/opt.py # vllm/model_executor/models/orion.py # vllm/model_executor/models/ouro.py # vllm/model_executor/models/param2moe.py # vllm/model_executor/models/persimmon.py # vllm/model_executor/models/phi.py # vllm/model_executor/models/phimoe.py # vllm/model_executor/models/plamo2.py # vllm/model_executor/models/plamo3.py # vllm/model_executor/models/qwen2.py # vllm/model_executor/models/qwen2_5_vl.py # vllm/model_executor/models/qwen2_moe.py # vllm/model_executor/models/qwen3.py # vllm/model_executor/models/qwen3_5.py # vllm/model_executor/models/qwen3_5_mtp.py # vllm/model_executor/models/qwen3_asr.py # vllm/model_executor/models/qwen3_dflash.py # vllm/model_executor/models/qwen3_moe.py # vllm/model_executor/models/qwen3_next.py # vllm/model_executor/models/qwen3_next_mtp.py # vllm/model_executor/models/qwen3_omni_moe_thinker.py # vllm/model_executor/models/qwen3_vl.py # vllm/model_executor/models/qwen3_vl_moe.py # vllm/model_executor/models/registry.py # vllm/model_executor/models/rnj1.py # vllm/model_executor/models/sarvam.py # vllm/model_executor/models/seed_oss.py # vllm/model_executor/models/solar.py # vllm/model_executor/models/stablelm.py # vllm/model_executor/models/starcoder2.py # vllm/model_executor/models/step1.py # vllm/model_executor/models/step3_text.py # vllm/model_executor/models/step3p5.py # vllm/model_executor/models/transformers/moe.py # vllm/model_executor/models/transformers/multimodal.py # vllm/model_executor/models/transformers/utils.py # vllm/model_executor/warmup/deep_gemm_warmup.py # vllm/multimodal/audio.py # vllm/multimodal/media/audio.py # vllm/multimodal/media/connector.py # vllm/multimodal/media/image.py # vllm/multimodal/video.py # vllm/parser/abstract_parser.py # vllm/parser/minimax_m2_parser.py # vllm/platforms/cpu.py # vllm/platforms/cuda.py # vllm/platforms/interface.py # vllm/platforms/rocm.py # vllm/reasoning/__init__.py # vllm/reasoning/abs_reasoning_parsers.py # vllm/reasoning/cohere_command_reasoning_parser.py # vllm/reasoning/qwen3_reasoning_parser.py # vllm/renderers/base.py # vllm/renderers/hf.py # vllm/renderers/registry.py # vllm/sampling_params.py # vllm/tokenizers/mistral.py # vllm/tokenizers/registry.py # vllm/tool_parsers/__init__.py # vllm/tool_parsers/abstract_tool_parser.py # vllm/tool_parsers/cohere_command_tool_parser.py # vllm/tool_parsers/deepseekv32_tool_parser.py # vllm/tool_parsers/deepseekv4_tool_parser.py # vllm/tool_parsers/gemma4_utils.py # vllm/tool_parsers/glm47_moe_tool_parser.py # vllm/tool_parsers/glm4_moe_tool_parser.py # vllm/tool_parsers/hy_v3_tool_parser.py # vllm/tool_parsers/llama_tool_parser.py # vllm/tool_parsers/minimax_m2_tool_parser.py # vllm/tool_parsers/mistral_tool_parser.py # vllm/tool_parsers/poolside_v1_tool_parser.py # vllm/tool_parsers/qwen3coder_tool_parser.py # vllm/tool_parsers/qwen3xml_tool_parser.py # vllm/tool_parsers/streaming.py # vllm/tool_parsers/utils.py # vllm/transformers_utils/config.py # vllm/transformers_utils/configs/__init__.py # vllm/transformers_utils/configs/speculators/algos.py # vllm/transformers_utils/gguf_utils.py # vllm/transformers_utils/model_arch_config_convertor.py # vllm/transformers_utils/processors/__init__.py # vllm/transformers_utils/processors/pixtral.py # vllm/transformers_utils/processors/voxtral.py # vllm/utils/__init__.py # vllm/utils/cpu_resource_utils.py # vllm/utils/cpu_triton_utils.py # vllm/utils/deep_gemm.py # vllm/utils/flashinfer.py # vllm/utils/import_utils.py # vllm/utils/mem_utils.py # vllm/utils/numa_utils.py # vllm/utils/ompmultiprocessing.py # vllm/utils/profiling.py # vllm/utils/torch_utils.py # vllm/v1/attention/backend.py # vllm/v1/attention/backends/cpu_attn.py # vllm/v1/attention/backends/fa_utils.py # vllm/v1/attention/backends/flash_attn.py # vllm/v1/attention/backends/flash_attn_diffkv.py # vllm/v1/attention/backends/flashinfer.py # vllm/v1/attention/backends/flex_attention.py # vllm/v1/attention/backends/gdn_attn.py # vllm/v1/attention/backends/mla/flashmla_sparse.py # vllm/v1/attention/backends/mla/indexer.py # vllm/v1/attention/backends/mla/prefill/__init__.py # vllm/v1/attention/backends/mla/prefill/base.py # vllm/v1/attention/backends/mla/prefill/flash_attn.py # vllm/v1/attention/backends/mla/prefill/flashinfer.py # vllm/v1/attention/backends/mla/prefill/registry.py # vllm/v1/attention/backends/mla/prefill/selector.py # vllm/v1/attention/backends/mla/prefill/trtllm_ragged.py # vllm/v1/attention/backends/mla/rocm_aiter_mla.py # vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py # vllm/v1/attention/backends/mla/sparse_swa.py # vllm/v1/attention/backends/rocm_aiter_fa.py # vllm/v1/attention/backends/rocm_aiter_unified_attn.py # vllm/v1/attention/backends/rocm_attn.py # vllm/v1/attention/backends/tree_attn.py # vllm/v1/attention/backends/triton_attn.py # vllm/v1/attention/backends/turboquant_attn.py # vllm/v1/attention/backends/utils.py # vllm/v1/attention/ops/dcp_alltoall.py # vllm/v1/attention/ops/rocm_aiter_mla_sparse.py # vllm/v1/attention/ops/triton_attention_helpers.py # vllm/v1/attention/ops/triton_decode_attention.py # vllm/v1/attention/ops/triton_reshape_and_cache_flash.py # vllm/v1/attention/ops/triton_unified_attention.py # vllm/v1/attention/selector.py # vllm/v1/capture/__init__.py # vllm/v1/capture/config.py # vllm/v1/capture/consumer.py # vllm/v1/capture/consumers/filesystem/consumer.py # vllm/v1/capture/consumers/filesystem/types.py # vllm/v1/capture/consumers/filesystem/validation.py # vllm/v1/capture/consumers/filesystem/writer.py # vllm/v1/capture/consumers/logging.py # vllm/v1/capture/driver_bridge.py # vllm/v1/capture/manager.py # vllm/v1/capture/plan.py # vllm/v1/capture/registry.py # vllm/v1/capture/sink.py # vllm/v1/capture/types.py # vllm/v1/core/kv_cache_coordinator.py # vllm/v1/core/kv_cache_manager.py # vllm/v1/core/kv_cache_utils.py # vllm/v1/core/sched/output.py # vllm/v1/core/sched/scheduler.py # vllm/v1/core/single_type_kv_cache_manager.py # vllm/v1/engine/__init__.py # vllm/v1/engine/async_llm.py # vllm/v1/engine/core.py # vllm/v1/engine/core_client.py # vllm/v1/engine/input_processor.py # vllm/v1/engine/output_processor.py # vllm/v1/engine/utils.py # vllm/v1/executor/multiproc_executor.py # vllm/v1/executor/ray_executor.py # vllm/v1/executor/ray_executor_v2.py # vllm/v1/executor/ray_utils.py # vllm/v1/kv_cache_interface.py # vllm/v1/kv_offload/base.py # vllm/v1/kv_offload/cpu/common.py # vllm/v1/kv_offload/cpu/gpu_worker.py # vllm/v1/kv_offload/cpu/manager.py # vllm/v1/kv_offload/cpu/policies/arc.py # vllm/v1/kv_offload/cpu/policies/lru.py # vllm/v1/kv_offload/cpu/shared_offload_region.py # vllm/v1/kv_offload/cpu/spec.py # vllm/v1/kv_offload/reuse_manager.py # vllm/v1/outputs.py # vllm/v1/pool/metadata.py # vllm/v1/request.py # vllm/v1/sample/ops/topk_topp_sampler.py # vllm/v1/sample/rejection_sampler.py # vllm/v1/sample/sampler.py # vllm/v1/sample/thinking_budget_state.py # vllm/v1/simple_kv_offload/cuda_mem_ops.py # vllm/v1/simple_kv_offload/manager.py # vllm/v1/spec_decode/dflash.py # vllm/v1/spec_decode/extract_hidden_states.py # vllm/v1/spec_decode/llm_base_proposer.py # vllm/v1/worker/cpu_model_runner.py # vllm/v1/worker/cpu_worker.py # vllm/v1/worker/encoder_cudagraph.py # vllm/v1/worker/encoder_cudagraph_defs.py # vllm/v1/worker/gpu/attn_utils.py # vllm/v1/worker/gpu/block_table.py # vllm/v1/worker/gpu/cudagraph_utils.py # vllm/v1/worker/gpu/dp_utils.py # vllm/v1/worker/gpu/kv_connector.py # vllm/v1/worker/gpu/model_runner.py # vllm/v1/worker/gpu/model_states/default.py # vllm/v1/worker/gpu/sample/gumbel.py # vllm/v1/worker/gpu/sample/logprob.py # vllm/v1/worker/gpu/sample/prompt_logprob.py # vllm/v1/worker/gpu/sample/sampler.py # vllm/v1/worker/gpu/spec_decode/eagle/cudagraph.py # vllm/v1/worker/gpu/spec_decode/eagle/speculator.py # vllm/v1/worker/gpu/spec_decode/rejection_sampler.py # vllm/v1/worker/gpu/spec_decode/synthetic_rejection_sampler_utils.py # vllm/v1/worker/gpu/warmup.py # vllm/v1/worker/gpu_input_batch.py # vllm/v1/worker/gpu_model_runner.py # vllm/v1/worker/gpu_worker.py # vllm/v1/worker/steering_manager.py # vllm/v1/worker/steering_model_runner_mixin.py # vllm/v1/worker/ubatch_utils.py # vllm/v1/worker/worker_base.py # vllm/v1/worker/xpu_worker.py # vllm/vllm_flash_attn/flash_attn_interface.py
refactor(sae): migrate post_mlp hook to post_block with true block-output semantics
# Conflicts: # docs/features/sae_steering.md
# Conflicts: # docs/features/sae_steering.md # tests/v1/worker/test_sae_mixin_integration.py
…h entry; add FR attach worker delegate
feat(sae): close all SAE steering limitations — thresholds, multi-module, Rust, HTTP API
…r and api-utils helpers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings SAE-based steering into the integration line: per-(layer, hook) SAE feature surgery composing with the existing additive steering, capture, and patching stack.
What this adds
SAE delta tier (
sae_steering.py,SAEClampManager): encode the live residual with a registered SAE, replace a chosen set of feature activations with caller-supplied clamp values (absolute or additive), and add the resulting decoder-direction delta back into the residual stream. Clamp tables use row 0 as the no-op sentinel, rows 1/2 as global prefill/decode tiers, rows 3+ per-request; global and per-request clamps validate against overlap in both directions andset_global_clampsis atomic.SAE full-reconstruction tier (
sae_full_reconstruction.py,SAEFullReconstructionManager): replace the residual withdecode(activate(encode(h))) + b_dec, optionally clamping the activation before decode (Scaling Monosemanticity / Golden Gate semantics). Rows are gated per-site by an active-row table (a row owned by another module's site never reconstructs here) and keyed by FR-only phase hashes so additive state can't route tokens into a reconstruction row in the wrong phase.Runtime integration: both tiers ride the existing machinery —
SamplingParams.sae_clamp_specs/sae_full_reconstruction_specswith cached phase hashes folded into the steering config hashes (prefix-cache aware), three independent scheduler capacity pools (additive / SAE clamp / full-recon) under the config-pool backpressure model, admission/release/streaming-refresh/preemption-resumption lifecycle in the steering runner mixin, TP/PP via the same collective_rpc discipline, and SAE module registration (manifest + encoder/decoder weights) through the steering modules registry and/v1/steering/modulesrouter withkind: sae_delta | sae_full_reconstruction.Hook semantics: SAE surgery at the block output operates on the true block output
residual + hidden_states(the #198/#202post_blockconsolidation applied to SAE), writing back through the residual summand as a delta so untouched tokens stay bit-identical; the sum is only formed when SAE buffers are attached at the site.Notes for review
SteeringManager.register_configgains optional content-hash row aliasing (identical vector content shares a physical row); implicit aliasing applies to prefill rows only, keeping decode rows 1:1 for the per-row scale/monitor machinery.tests/models/language/generation/test_sae_*_real_weights.py,test_steering.py,tests/v1/core/test_steering_config_reservation.py) still need a pass on GPU hardware.