[Bug][Windows] Native-main owner fails permanently closed when icacls is slow: 5s ACL-harden budget exhausted at startup, all native requests 503 (root cause of #1120?)

[BeniSisaki]

Summary

On Windows, when icacls.exe takes longer than the 5s OPENCODEX_ACL_TIMEOUT_MS budget during proxy startup, hardenStableLockFile throws a non-busy error inside prepareOwnerDatabase. drive() in src/codex/native-main-owner.ts then publishes {status: "unavailable", reason: "lock-unavailable"} and — unlike the contended (SQLITE_BUSY) branch — never retries. The startup gate latches blocked with reason owner-unavailable, and every native OpenAI request fails for the entire process lifetime with 503 server_is_overloaded / "Native Codex main profile is switching; retry this request", which the Codex app renders as "Selected model is at capacity. Please try a different model." — sending users chasing OpenAI quota for a purely local proxy failure.

This is 100% reproducible on my machine across ocx restart and full cold starts, while the identical acquisition steps succeed in <1s in a standalone process with the same bundled bun binary. It matches #1120's symptom exactly (same error message, same zero-profiles state, same pool mode, Windows) and may be its missing root cause.

Environment

• @bitkyc08/opencodex 2.10.2 (npm), bundled bun runtime
• Windows 11 (10.0.26100), single local user, real-time AV enabled
• providers.openai.codexAccountMode: "pool", authMode: "forward", zero native profiles (GET /api/native-main-profiles → profiles: [])

Instrumented root cause

I temporarily patched drive()'s catch to log the swallowed error, and defaultAsyncIcaclsRunner to log spawn/exit timings. Captured during a cold ocx start:

21:54:46.286 spawn icacls .../.opencodex-native-main.owner.sqlite /grant:r WORKGROUP\user:(F) budgetMs=5000
21:54:48.247 exited code=0                     // first icacls took ~2.0s
21:54:48.259 spawn icacls .../inheritance:r budgetMs=3027   // only the REMAINING budget
21:54:51.xxx  TIMEOUT kill                      // remaining budget exhausted
21:54:59.425 exited code=0                     // second icacls needed ~11.2s wall time

owner publish: Error: ACL hardening timed out (ETIMEDOUT) — transient icacls stall;
  the volume may still support per-user NTFS ACLs; ACL state unverified (budget exhausted)
    at hardenEntryAsync (src/lib/windows-secret-acl.ts:672)
    at hardenStableLockFile → prepareOwnerDatabase → drive

Both icacls invocations eventually exit 0 — they are just slow inside the daemon's startup window (likely AV scanning + startup I/O contention; the identical harden completes in ~230ms in a light standalone bun process, even when spawned detached with windowsHide). Two design choices turn this transient local slowness into a permanent service outage:

1. One 5s deadline is shared across both icacls calls and both retry attempts (deadline = nowFn() + resolveHardenDeadlineMs() in hardenEntryAsync), so a single slow spawn dooms everything; the second call only gets the leftover budget.
2. The failure latches forever. drive() only calls scheduleRetry for isBusy(error); any other error (including this timeout) publishes terminal unavailable. observeOwner maps that to gate reason owner-unavailable, so all native traffic 503s until process exit. ocx restart does not help because startup load reproduces the timeout every time.

Additional wrinkle: the 503 is emitted in single-digit ms with Retry-After: 1 and code server_is_overloaded, which Codex renders as an upstream "model at capacity" message (also called out in #1120, proposal 4). My install logged 37/37 failed requests (35×503 native, 2×400 mimo) with zero successes while ocx status and /healthz reported everything green — health checks never exercise this gate.

Reproduction

1. Windows host where icacls on %USERPROFILE%\.codex\.opencodex-native-main.owner.sqlite collectively takes >5s during daemon startup (AV-heavy machines).
2. ocx start → any native-routed request immediately 503s with server_is_overloaded / "Native Codex main profile is switching".
3. GET /api/native-main-profiles/doctor → {"code":"NATIVE_MAIN_OWNER_UNAVAILABLE","error":"This process cannot establish native-main ownership for the effective CODEX_HOME.","retryable":true} (note: retryable:true, but nothing ever retries).
4. Control: running the exact same retainNativeMainOwner flow from a standalone bun process on the same host converges to held → gate ready in <1s.

Workaround (verified end-to-end)

setx OPENCODEX_ACL_TIMEOUT_MS 60000
ocx restart

After this, doctor reports healthy (authStatus: ok) and a real streaming gpt-5.6-luna request through the proxy returns HTTP 200. (#1120's ocx provider account-mode direct workaround also bypasses this layer.)

Suggested fixes

1. Give each icacls invocation its own budget instead of sharing one 5s deadline across both calls and both attempts; and/or raise the default with realistic headroom for Windows AV machines.
2. Classify ACL-harden timeout as retryable (scheduleRetry, or re-drive with backoff) — never permanently fence all native traffic for transient local slowness. At minimum, honor the advertised retryable: true.
3. With zero registered profiles, consider skipping the owner lock / ACL hardening entirely so the subsystem is inert (Native-profile lock deadlock: all native requests 503 with zero profiles registered, and account main recover cannot clear it #1120 proposal 1).
4. Emit a distinct, opencodex-named error for locally generated failures instead of server_is_overloaded, so clients don't misrender it as an upstream capacity problem (Native-profile lock deadlock: all native requests 503 with zero profiles registered, and account main recover cannot clear it #1120 proposal 4).
5. Surface gate/owner state in ocx status/ocx doctor — both reported "ok" while 100% of data-plane requests failed.

Related: #1120 (same symptom), #1149 (separate Windows ACL-hardening failure mode). My instrumentation patches were reverted after capture; timings/logs above are verbatim from the captured run.

[Wibias]

Closing as completed by #1197 together with the existing owner recovery path.

The investigation corrected one assumption in this report: owner recovery already retries with a fresh hardening budget. The remaining reproduced defect was the 5-second ACL envelope being too short for a complete icacls sequence on slow Windows hosts. #1197 raises the default envelope to 30 seconds while retaining the bounded, fail-closed behaviour.

If a current build can still reproduce the same failure with the new 30-second policy, that should be reported as a new focused issue with fresh timings rather than keeping the outdated no-retry diagnosis open.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}