feat(fleetnode): route miner commands to fleet-node-paired devices

[ankitgoswami]

Summary

Operators can now send existing miner commands (reboot, start/stop mining, blink LED, curtail/uncurtail, set cooling mode, set power target) to miners that sit on a customer LAN behind a fleet node, using the same RPCs and UI as directly-dialed miners. Until now those miners had no command path: the cloud can't dial them. This PR makes routing internal: a fleet-node-paired device transparently resolves to a remote-node miner that pushes the command down that node's ControlStream and waits for the result. No frontend changes.

Builds on the ControlStream command foundation (#389) and the pairing foundation (#388). End-to-end execution works today for the no-secret virtual driver; authenticated drivers route correctly but defer execution until per-org credential provisioning lands (tracked in #402).

How it works

The command pipeline is unchanged; transport is spliced in at one seam: miner resolution.

1. Operator issues a command through an existing RPC/UI. It flows through command.Service to the durable command queue to ExecutionService.executeCommandOnDevice, which asks minerService.GetMiner(deviceID) for a transport-agnostic interfaces.Miner and calls methods like Reboot(ctx) on it. None of this changed.

2. Resolution decides the transport. GetMinerFromDeviceIdentifier now checks fleet-node ownership first via a new query, GetActiveFleetNodeForDevice: the device must be device_pairing = PAIRED and bound (fleet_node_device) to a CONFIRMED, non-deleted fleet node. If so it returns a remotenode.Miner; otherwise it falls through to the existing direct PluginMiner. This lookup is deliberately not cached: it re-resolves per command so an unpair/revoke takes effect immediately (the remote miner holds no live connection, so re-resolving is cheap).

3. The remote-node miner serializes and dispatches. remotenode.Miner implements every interfaces.Miner method. Each builds a MinerCommand (an action oneof plus a MinerConnectionDescriptor: the dial coordinates and opaque credential the node needs, since the node has no device DB), acquires a per-node concurrency slot (PerNodeLimiter, default 8 in flight per node), wraps it in the shared AgentCommand envelope, and calls registry.SendCommand(fleetNodeID, ControlCommand). That pushes the command onto the node's ControlStream and blocks for the matching ControlAck (feat(fleetnode): carry concurrent commands over the ControlStream #389's machinery).

4. The node executes against the LAN miner. handleCommand switches on the AgentCommand kind; the new miner_command arm runs handleMinerCommand: protovalidate the message, validateDialTarget (must be a private/loopback IP with a known URL scheme), build a local SDK device from the descriptor via the node's plugin driver plus a SecretProvider, run the selected action under a 25s timeout (kept below the server's 30s worker timeout), and sendAck exactly once with a terminal code.

5. The ack maps back to a typed error that drives the queue's retry logic: OK is success; BUSY / no-active-stream are retryable (ResourceExhausted / Unavailable); UNAUTHENTICATED is an auth error (and the server evicts the cached handle); UNIMPLEMENTED / AGENT_INCAPABLE are permanent unsupported; anything else is INTERNAL. Only the queue's existing permanent-fail set (Unimplemented/FailedPrecondition) stops retries.

6. Cache coherence on pairing transitions. Pair/unpair, both the manual RPC and the node-reported result path (PersistFleetNodePairResult), evict the device's cached direct handle (InvalidateMinerByID) after commit, so a freshly-bound device stops direct-dialing immediately instead of waiting out the 1-minute cache TTL.

7. Gateway boundary hardening. Inbound ControlAcks are protovalidated at the gateway and dropped if invalid/oversized, bounding what a buggy or hostile node can publish before it reaches the server.

Component / flow

flowchart LR
  Op["Operator (existing RPC / UI)"] --> Cmd["command.Service + durable queue"]
  Cmd --> Exec["ExecutionService.executeCommandOnDevice"]
  Exec --> Resolve["minerService.GetMiner"]
  Resolve -->|"PAIRED and bound to CONFIRMED node"| Remote["remotenode.Miner"]
  Resolve -->|"otherwise (unchanged)"| Direct["direct PluginMiner"]
  Remote --> Gate["PerNodeLimiter (<=8 in flight per node)"]
  Gate --> Reg["control registry.SendCommand (#389)"]
  Reg -->|"ControlCommand over ControlStream"| Node["Fleet node"]
  Node --> Drv["node plugin driver"]
  Drv -->|"dial LAN miner"| Miner["LAN miner"]
  Miner -->|"result"| Drv
  Drv -->|"ControlAck"| Reg
  Reg --> Remote
  Remote -->|"nil or typed error"| Exec

Loading

Command lifecycle (sequence)

sequenceDiagram
  participant Q as "Queue worker (30s budget)"
  participant M as "remotenode.Miner"
  participant G as "PerNodeLimiter"
  participant R as "Control registry"
  participant N as "Fleet node (25s budget)"
  participant D as "LAN miner"
  Q->>M: "Reboot / Curtail / SetCoolingMode ..."
  M->>G: "Acquire(fleetNodeID)"
  G-->>M: "slot granted"
  M->>R: "SendCommand(AgentCommand.miner_command)"
  R->>N: "push over ControlStream"
  N->>N: "protovalidate + validateDialTarget + build device"
  N->>D: "run action via plugin driver"
  D-->>N: "outcome"
  N-->>R: "ControlAck(code)"
  R-->>M: "ControlAck"
  M-->>Q: "nil | retryable error | permanent error"

Loading

Areas of the code involved

Area / file	What changed	Why it matters for review
proto/pairing/v1/pairing.proto	New MinerCommand, MinerConnectionDescriptor, per-action messages; adds miner_command = 3 to the AgentCommand oneof	Wire contract + validation rules (IP, port range, credential size). The envelope's field numbers are coordinated with #388
proto/fleetnodegateway/v1/fleetnodegateway.proto	Adds ACK_CODE_UNAUTHENTICATED = 9, ACK_CODE_UNIMPLEMENTED = 10	New ack outcomes the adapter maps to errors
server/generated/** (*.pb.go, *_pb.ts, sqlc/*)	Regenerated	generated, skip
server/internal/domain/miner/service.go	WithCommandSender enables routing; tryFleetNodeMiner resolution branch (uncached); InvalidateMinerByID	Core resolution seam: where a device becomes "remote" vs "direct" and how the cache stays coherent
server/internal/domain/miner/remotenode/miner.go (new)	interfaces.Miner adapter: builds MinerCommand, dispatches via the registry, maps ControlAck to error; ack-reason clamp	The whole transport adapter; check ack-to-error/retry classification
server/internal/domain/miner/remotenode/gate.go (new)	PerNodeLimiter keyed counting semaphore	Concurrency/back-pressure model per fleet node
server/sqlc/queries/miner_service.sql	New GetActiveFleetNodeForDevice; site_id added to dial-coordinate selects	Routing gate (PAIRED + CONFIRMED node). Note the discovered_device join intentionally does not filter deleted_at (tolerates re-discovered/soft-deleted linkage)
server/cmd/fleetnode/minercommand.go (new)	Node-side executor: validate, validateDialTarget, build device + SecretProvider, run action under timeout, ack; enum converters; error classification	Node execution + the credential seam + SSRF-style dial validation
server/cmd/fleetnode/control.go, run.go	Adds the miner_command case to handleCommand; wires the plugin driver getter + secret provider	Small dispatch + wiring hooks
server/internal/handlers/fleetnode/gateway/handler.go	Protovalidate + drop invalid inbound ControlAck	Trust boundary for node-supplied acks
server/internal/domain/fleetnode/pairing/{service.go,pair_discovered.go}	WithMinerInvalidator; invalidate after pair (manual + node-reported) and unpair	Cache coherence on pairing transitions
server/cmd/fleetd/main.go	Wires WithCommandSender(registry) and WithMinerInvalidator(InvalidateMinerByID)	Composition root: confirms routing is actually enabled

Key technical decisions & trade-offs

• Splice at the resolution seam, not a parallel command path. A device resolves to a remotenode.Miner implementing the same interfaces.Miner as the direct miner, so the entire command service / queue / status / authz / curtailment pipeline is reused unchanged. Alternative: a separate fleet-node command code path (more surface, drift risk).
• Resolution is uncached. Re-resolving per command makes unpair/revoke take effect immediately. Alternative: cache the remote handle for speed, rejected because revocation latency matters more than a cheap DB lookup.
• Per-node concurrency gate in the server (limit 8 < node worker pool). A large batch paces in the server while the durable queue holds the backlog, instead of oversubscribing the node and getting rejected BUSY. Alternative: rely on node-side BUSY + retries (wastes round-trips).
• Offline/BUSY are retryable, not permanent. Unavailable (no active stream) and ResourceExhausted (BUSY) stay out of the queue's permanent-fail set, so a node mid-reconnect or momentarily saturated re-attempts rather than dropping the command.
• Credential boundary (per RFC 0001). The descriptor carries opaque ciphertext; the node decrypts just-in-time with its per-org key. The cloud never ships plaintext. The default SecretProvider returns an empty bundle, so only no-secret drivers (virtual) execute end-to-end today; authenticated drivers (proto, antminer) route correctly but can't execute until pairing provisions keys (deferred, Fleet-node authenticated-driver credential delivery (post #390) #402).
• Node-side dial validation. validateDialTarget requires a private/loopback IP and a known URL scheme, so a malformed/hostile descriptor can't make the node dial arbitrary hosts. Proto-level constraints (IP format, port 1..65535, credential up to 8 KiB) back this up.
• 25s node budget < 30s server worker budget. Bounds node execution below the server's command timeout. A residual large-batch + slow-miner double-execution window exists but is benign for today's idempotent commands; the deadline-propagation fix is deferred alongside Phase-3 non-idempotent commands (Fleet-node authenticated-driver credential delivery (post #390) #402).
• discovered_device join does not filter deleted_at. Because the discovery uniqueness index is partial on deleted_at IS NULL, re-discovering a soft-deleted identifier inserts a fresh row and leaves the device's FK on the old one; filtering would drop a still-PAIRED device. (A stale-coordinate guard via resolve-by-live-identifier is deferred to Fleet-node authenticated-driver credential delivery (post #390) #402.)

Testing & validation

Automated (added):

• Adapter unit tests: ackToError mapping per code, no-active-stream is retryable, ack-reason clamping; PerNodeLimiter concurrency/release.
• Node unit tests: validateDialTarget (accept private/loopback + known scheme, reject public/empty/unknown), enum converters reject undefined values with BAD_REQUEST, AgentCommand decode/lane routing.
• Node integration: command worker pool acks BUSY only at the pool ceiling.
• DB-backed resolution: a PAIRED device bound to a CONFIRMED node resolves to a remotenode.Miner and dispatches; a node-bound-but-not-PAIRED device does not route; a device whose linked discovered_device is soft-deleted still resolves (regression guard).
• DB-backed pairing: pair/unpair (manual) and PersistFleetNodePairResult (node-reported PAIRED) evict the miner cache; non-PAIRED outcomes don't.
• Gateway: invalid/oversized inbound ControlAck is dropped, valid acks pass through.

Verified locally: go build ./..., full-module golangci-lint, and the above suites (DB-backed against the dev Postgres) pass.

Not covered / out of scope:

• Authenticated-driver execution end-to-end: only the virtual driver runs through; proto/antminer need credential provisioning (Fleet-node authenticated-driver credential delivery (post #390) #402).
• Heavy commands (firmware update, log download, password change, unpair reconciliation): deferred to Phase 3.
• Frontend: none needed; commands reuse existing RPCs.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (75933fce45c269e41e0ba2b30cd33825b0389fa7...97d7288164591372dc1e30485d7b3af94ea076e4, exact PR three-dot diff)
• Model: gpt-5.5

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: HIGH

Findings

[HIGH] Fleet-node miner commands are dispatched without authentication material

• Category: Auth
• Location: server/internal/domain/miner/service.go:203
• Description: tryFleetNodeMiner now routes every paired fleet-node device through remotenode.New, but the descriptor is built without any credential or signer material. The new SQL query also does not select miner_credentials, and the fleet node’s default nodeSecretProvider returns an empty sdk.SecretBundle. Real drivers do not accept that: Proto requires a BearerToken, and Antminer requires UsernamePassword.
• Impact: Successfully paired fleet-node miners cannot execute reboot/start/stop/curtail/power/cooling commands. Depending on the driver, this is reported as auth/internal failure and burns queue retries instead of executing, making the new remote command path effectively work only for no-auth test/virtual drivers.
• Recommendation: Complete the auth handoff before enabling this route for real devices. Include stored encrypted credentials in GetActiveFleetNodeForDevice and implement node-side decryption/secret reconstruction, and generate Proto bearer tokens from the fleet node’s miner-signing private key. Add end-to-end tests for at least one basic-auth driver and one Proto/asymmetric driver.

Notes

No SQL injection, command injection, pool hijacking, or protobuf wire-format breakage was evident in the reviewed diff. The main issue is that the new command routing path crosses the fleet-node/plugin auth boundary without carrying the credentials required by the existing drivers.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7437cc03a8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Copilot]

Pull request overview

This PR enables operator miner commands (reboot/curtail/start/stop/etc.) to reach devices paired via fleet nodes by routing command execution over the fleet node ControlStream, while keeping the existing command queue/permissions/status pipeline unchanged.

Changes:

• Adds pairing.v1.MinerCommand + MinerConnectionDescriptor in the AgentCommand envelope, plus new AckCode values (UNAUTHENTICATED, UNIMPLEMENTED) for better server/node error semantics.
• Adds a server-side remotenode interfaces.Miner adapter and resolution logic (GetActiveFleetNodeForDevice) so fleet-node-owned devices route over the control plane instead of direct-dial.
• Adds node-side miner command execution (cmd/fleetnode/minercommand.go) with protovalidate validation and dial-target constraints, plus gateway-side inbound ControlAck validation/dropping.

Reviewed changes

Copilot reviewed 17 out of 23 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
server/sqlc/queries/miner_service.sql	Adds GetActiveFleetNodeForDevice resolution query; tightens direct-dial queries to ignore soft-deleted discovered_device rows.
server/internal/handlers/fleetnode/gateway/handler.go	Validates inbound ControlAck at the gateway trust boundary and drops invalid/oversized acks.
server/internal/handlers/fleetnode/gateway/handler_controlstream_test.go	Tests that invalid acks are dropped and do not poison command completion.
server/internal/domain/miner/service.go	Adds command routing configuration and fleet-node-aware miner resolution via tryFleetNodeMiner.
server/internal/domain/miner/resolution_test.go	Integration test asserting node-paired devices resolve to the remote-node miner adapter and dispatch via sender.
server/internal/domain/miner/remotenode/miner.go	Implements remote-node Miner adapter, ack→error mapping, and ack error message clamping.
server/internal/domain/miner/remotenode/miner_test.go	Unit tests for encoding, ack mapping, retryability, and clamp behavior.
server/internal/domain/miner/remotenode/gate.go / gate_test.go	Per-fleet-node concurrency limiter to pace large batches and avoid node BUSY rejections.
server/cmd/fleetnode/run.go	Adds seams for driver lookup and secret provisioning used by miner command execution.
server/cmd/fleetnode/minercommand.go / minercommand_test.go	Node-side decoding/validation, device reconstruction, action execution, and ack classification.
server/cmd/fleetnode/control.go / control_test.go	Dispatches AgentCommand.miner_command and tests BUSY behavior at worker pool ceiling.
server/cmd/fleetd/main.go	Wires MinerService.WithCommandSender to enable routing in fleetd.
proto/pairing/v1/pairing.proto	Adds MinerCommand + descriptor into AgentCommand envelope.
proto/fleetnodegateway/v1/fleetnodegateway.proto	Adds new AckCode enum values used for command semantics.
Generated Go/TS (server/generated/..., client/src/.../generated/...)	Regenerated outputs reflecting the proto/sqlc changes.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 77388bdbde

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58b3fec8d1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6b1d691087

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03cc3c227a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07f2977eb3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e1cdb93cf

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bc427d5ff2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[flesher]

LGTM!