Skip to content

[image-spec]: nix runner SSH build pattern matching exa-deploy#44

Open
jld-adriano wants to merge 2 commits intomasterfrom
devin/1771707827-runner-based-nix-build
Open

[image-spec]: nix runner SSH build pattern matching exa-deploy#44
jld-adriano wants to merge 2 commits intomasterfrom
devin/1771707827-runner-based-nix-build

Conversation

@jld-adriano
Copy link

@jld-adriano jld-adriano commented Feb 21, 2026

Tracking issue

Related to exa-labs/monorepo#18302

Why are the changes needed?

The previous nix build approach ran builds locally and used nix remote builders for cross-architecture builds, requiring a buildSystem parameter and complex cross-docker package generation in the flake. This differed from how exa-deploy builds images (SSH to dedicated nix runners, build+push on-runner). Aligning the two systems simplifies the flake, removes cross-build complexity, and ensures each architecture builds natively on its own runner.

What changes were proposed in this pull request?

Replaces the local nix run ... #copyTo approach with an SSH-to-runner pattern matching exa-deploy's mkNixBuild/mkNixRun:

  1. _nix_build_on_runner — runs nix build --store ssh-ng://root@{runner} to evaluate locally and build on the remote runner's store (same as exa-deploy's mkNixBuild)
  2. _nix_run_on_runner — SSHs to the runner to execute the built copy-to binary for ECR push (same as exa-deploy's mkNixRun)
  3. _load_nix_runner_ssh_key — fetches nix-runner-ssh-key from AWS Secrets Manager (same as exa-deploy's loadSshKey)

Same runner pool and SSH key as exa-deploy. Removes local system detection (platform.system()/platform.machine()) and cross-build branching since each arch now builds natively on its own runner.

Human review checklist

  • No e2e test yet — needs validation with flyte_aws.py on hephaestus after the companion monorepo PR lands. The previous cross-build approach was validated but this new runner-SSH path has not been.
  • SSH command constructionimage_spec.image_name() and ECR token flow into the SSH remote command string via f-string interpolation, not shlex.quote(). ECR tokens are base64-safe, image names are registry URLs, but worth eyeballing for injection surface.
  • AWS Secrets Manager required for all nix builds_load_nix_runner_ssh_key() is called even for push=False (build-only). Local-only nix builds now require AWS credentials.
  • Hardcoded runner IPs — same IPs as exa-deploy/src/lib.ts, always picks runners[0] with no fallback or health check.

Updates since last revision

  • SSH key file now created with atomic 0o600 permissions via os.open() (no race window)
  • _nix_run_on_runner now logs the redacted SSH command using re.sub for targeted credential masking

How was this patch tested?

Not yet tested end-to-end. The previous cross-build approach (which this replaces) was validated with flyte_aws.py on the hephaestus cluster. This new runner-based approach needs the same validation after the companion monorepo PR removes the buildSystem/cross-docker flake complexity.

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

  • exa-labs/monorepo#18302 — companion PR that removes buildSystem/crossDockerPackages from the flake

Link to Devin run: https://app.devin.ai/sessions/a0791e463dec4cfcacc33e89aa5c1d9f
Requested by: @jld-adriano

Build+push on native runners via SSH instead of local nix run with
remote builders. Each arch builds natively on its own runner:
- _nix_build_on_runner: nix build --store ssh-ng://root@runner
- _nix_run_on_runner: SSH to runner, execute copyTo
- Same runner pool and SSH key (nix-runner-ssh-key) as exa-deploy
- Removes local cross-build complexity (buildSystem, local_system detection)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
Copy link

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant