Skip to content

fix: fix vllm patching for realonly venvs - #8

Merged
soluwalana merged 1 commit into
nmp/customizerfrom
fix-readonly-venv-vllm-patching/anubhutiv
Aug 5, 2026
Merged

fix: fix vllm patching for realonly venvs#8
soluwalana merged 1 commit into
nmp/customizerfrom
fix-readonly-venv-vllm-patching/anubhutiv

Conversation

@anubhutivyas

Copy link
Copy Markdown
Collaborator

What does this PR do ?

_locked_file_patch runs

lock_path = file_path + ".patch_lock"
 lock_fd = open(lock_path, "w")

which creates a new entry in the venv's site-packages/vllm/v1/executor/. That needs write permission on the directory and the venvs are made root-owned in our Dockerfile but the workers run as uid 1000, so it fails.

Context on the fix:
chown -R on the venvs is not the fix, because of what I learned today:

GNU chown dereferences by default, so plain chown -R follows every per-file symlink and chowns the targets inside /opt/uv_cache and overlayfs copies each one up into the new layer, duplicating the cache, and hands the shared training stack to uid 1000.

I found out that chown -Rh avoids that but only fixes the lock, symlink ownership is ignored for access checks. So, in the next few lines after the above code block we call

def write_back(new_content: str):
            with open(file_path, "w") as f:
                f.write(new_content)

It still hits the read-only cache file. And where the -h in chown does let the write through, it writes into the copy shared by both vllm venvs, each of which needs its own sys.executable in it. So, the fix is to stop needing directory write at all and move the lock out of site-packages, what is done in this PR and give each vllm venv a private copy of just the patched files which is done in this PR - NVIDIA-NeMo/nemo-platform#1056

@anubhutivyas
anubhutivyas force-pushed the fix-readonly-venv-vllm-patching/anubhutiv branch from f6109c2 to a7bf687 Compare August 4, 2026 19:21
Signed-off-by: anubhutiv <anubhutiv@nvidia.com>
@anubhutivyas
anubhutivyas force-pushed the fix-readonly-venv-vllm-patching/anubhutiv branch from a7bf687 to 73833ff Compare August 5, 2026 19:44
@soluwalana
soluwalana merged commit edba0f9 into nmp/customizer Aug 5, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants