diff --git a/docs-site/src/content/docs/guides/grok-build.md b/docs-site/src/content/docs/guides/grok-build.md index 9b1fc1bf64..958b25a30b 100644 --- a/docs-site/src/content/docs/guides/grok-build.md +++ b/docs-site/src/content/docs/guides/grok-build.md @@ -127,11 +127,12 @@ the id `grok-4.5`. Generated aliases avoid dots entirely for this reason. rejects unknown event types, so a manually configured `api_backend = "responses"` model can fail mid-turn on slow upstreams. The auto-registered entries pin `api_backend = "chat_completions"`, which never surfaces raw heartbeat frames. -- **Service-installed `ocx restart`:** when opencodex runs under a service manager, - `ocx restart` currently stops the service and replaces it with an unmanaged process — - service persistence (auto-restart, start-at-login) is lost until the next - `ocx service` setup, and if that unmanaged process dies the managed block can point at - a dead proxy until the next `ocx start`/`ocx ensure` refreshes it. +- **Service-installed `ocx restart`:** the running proxy owns restart authorization and drain + coordination, while the installed service manager launches the replacement after the old process + exits. Service supervision remains installed. On loopback auto-registration, the managed block + also remains in place across the handoff; non-loopback deployments use manually managed Grok + configuration instead. The command succeeds only after a different, identity-verified process is + healthy on the same port. - **Config read timing:** start opencodex first, then launch `grok` for the most predictable results. Grok Build watches `~/.grok/config.toml` and reloads when the `[model]` table actually changes (roughly a one-second debounce, compared by content), so diff --git a/docs-site/src/content/docs/ja/guides/grok-build.md b/docs-site/src/content/docs/ja/guides/grok-build.md index b7c18f3cf4..6dc6b2f2e4 100644 --- a/docs-site/src/content/docs/ja/guides/grok-build.md +++ b/docs-site/src/content/docs/ja/guides/grok-build.md @@ -78,8 +78,7 @@ api_key = "your-OPENCODEX_API_AUTH_TOKEN" - **バックエンドとキープアライブの応答:** opencodex は `response.heartbeat` キープアライブを発行します アップストリーム沈黙中の `/v1/responses` ストリーム。 Grok Build の Responses デコーダは未知のイベント タイプを拒否するため、手動で構成された `api_backend = "responses"` モデルは低速なアップストリームではターン中に失敗する可能性があります。自動登録されたエントリは `api_backend = "chat_completions"` をピン留めしますが、生のハートビート フレームが表示されることはありません。 -- **サービスでインストールされた `ocx restart`:** opencodex がサービス マネージャーの下で実行される場合、 -現在、`ocx restart` はサービスを停止し、アンマネージド プロセスに置き換えます。サービスの永続性 (自動再起動、ログイン時開始) は、次の `ocx service` セットアップまで失われます。また、そのアンマネージド プロセスが終了した場合、次の `ocx start`/`ocx ensure` が更新するまで、マネージド ブロックは無効なプロキシを指す可能性があります。 +- **サービスでインストールされた `ocx restart`:** 実行中のプロキシが再起動の認可とドレインの調整を担当し、古いプロセスの終了後はインストール済みのサービス マネージャーが置換プロセスを起動します。サービス監視は維持されます。ループバックの自動登録を使用している場合に限り、マネージド ブロックもハンドオフ中に維持されます。非ループバック構成では Grok 設定を手動管理します。同じポートで、別の ID 検証済みプロセスが正常になったことを確認した場合にのみ成功します。 - **構成読み取りタイミング:** 最初に opencodex を起動し、その後 `grok` を起動します。 予測可能な結果。 Grok Build は `~/.grok/config.toml` を監視し、`[model]` テーブルが実際に変更されると (内容で比較すると約 1 秒のデバウンス) 再ロードするため、更新されたブロックは再起動せずに開いているセッションに到達します。 Grok が解析した内容を確認するには、`grok inspect` を実行します。ロードされた設定ソースがリストされ、拒否されたフィールドについて警告が表示されます。解決されたモデルのリストは出力されません。単一の TOML エラーがユーザー設定レイヤー「全体」を無効にすることに注意してください。これが、opencodex がファイルをアトミックに書き込む理由です。Grok は書きかけの設定を決して認識しません。 - **カタログの更新:** フェンスで囲まれたブロックには、射出時のカタログが反映されます。後 diff --git a/docs-site/src/content/docs/ja/reference/cli/lifecycle.md b/docs-site/src/content/docs/ja/reference/cli/lifecycle.md index 5f6a049987..4477201701 100644 --- a/docs-site/src/content/docs/ja/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/ja/reference/cli/lifecycle.md @@ -28,7 +28,9 @@ ocx start --port 8080 ### `ocx restart` -`stop` に続いて `ensure` を実行します。プロキシ/サービスを停止し、ネイティブ Codex を復元し、バックグラウンドでプロキシを起動し、ライブ ポートを Codex に同期します。 +プロキシが実行中の場合、検証済みの正確な PID とポートに対して in-place 再起動を要求し、通常のドレインを待ってから、同じポートに別のランタイム PID が起動したことを確認します。管理対象ルーティングとサービス監視は維持され、不確実な要求を別の stop/start として再実行しません。実行中のプロキシがない場合に限り、通常の `ensure` 起動にフォールバックします。 + +稼働中のリスナーをランタイム PID に結び付けて検証できない場合(更新前のプロキシを含む)、`ensure` や stop/start へのフォールバックは行わず安全側で失敗します。所有権を確認してから `ocx stop`、`ocx start` の順に一度実行してください。 ### `ocx ensure` diff --git a/docs-site/src/content/docs/ja/troubleshooting/windows-memory.md b/docs-site/src/content/docs/ja/troubleshooting/windows-memory.md index eca5b46fd1..6d25a17633 100644 --- a/docs-site/src/content/docs/ja/troubleshooting/windows-memory.md +++ b/docs-site/src/content/docs/ja/troubleshooting/windows-memory.md @@ -26,7 +26,7 @@ Windows では、#32111 クラッシュを回避するために、opencodex は - **`ocx doctor`** — 「メモリ / ランタイム」セクションには *サービス* が表示されます プロセスの Bun バージョン、RSS、外部/ArrayBuffers カウンター、JS ヒープ コンテキスト、およびストリーム モードの決定。バンドルされている Bun 1.3.14 ランタイムでは、`heapUsed` / `jscHeap` 単独ではリーク識別子ではありません。アプリレベルのリークを割り当てる前に、観察されたメモリを `responseState` および繰り返しサンプルと比較します。 - **`GET /api/system/memory`** — 認証済みの同じデータ -ダッシュボードまたはスクリプトの管理 API。 RSS/ヒープ/外部カウンターとともに、プロキシのメモリ内 `previous_response_id` 継続ストアのスカラー `responseState` ブロック (エントリ数、シリアル化された合計/最大バイト数、最も古いエントリの経過時間) を報告します。これはさらに成長に起因します。観察された記憶の上昇下での `responseState.totalBytes` の上昇は会話の保持を指します (長い `store:false` チェーンはターンごとに再拡張します)。一方、観察された記憶の上昇の下での横ばいの `responseState` はそのストアから遠ざかることを示します。値はスカラーのみであり、リクエスト本文、トークン、パス、アカウント識別子はありません。また、読み取りには副作用はありません (プルーニングや削除は行われません)。ダッシュボードの **メモリ可観測性** カードは同じフィールドをレンダリングし、確認ゲート付き **ドレインと再起動** アクションを提供します。現在のアクティブ ターン数を表示し、アクティブ ターンを最大 60 秒待機し (既存の 503 + `Retry-After` ドレインを再利用)、残りのターンを中止し、ライブ ポート (または障害専用サービス スーパーバイザ) 上の `ocx start` 経由でプロキシを再起動します。 respawn)Codex インジェクションを破棄せずに。これは、`POST /api/stop` の短いドレインよりも長く、情報に基づいたリサイクルです。 +ダッシュボードまたはスクリプトの管理 API。 RSS/ヒープ/外部カウンターとともに、プロキシのメモリ内 `previous_response_id` 継続ストアのスカラー `responseState` ブロック (エントリ数、シリアル化された合計/最大バイト数、最も古いエントリの経過時間) を報告します。これはさらに成長に起因します。観察された記憶の上昇下での `responseState.totalBytes` の上昇は会話の保持を指します (長い `store:false` チェーンはターンごとに再拡張します)。一方、観察された記憶の上昇の下での横ばいの `responseState` はそのストアから遠ざかることを示します。値はスカラーのみであり、リクエスト本文、トークン、パス、アカウント識別子はありません。また、読み取りには副作用はありません (プルーニングや削除は行われません)。ダッシュボードの **メモリ可観測性** カードは同じフィールドをレンダリングし、確認ゲート付き **ドレインと再起動** アクションを提供します。現在のアクティブ ターン数を表示し、アクティブ ターンを最大 60 秒待機し (既存の 503 + `Retry-After` ドレインを再利用)、残りのターンを中止します。実行中のプロキシは再起動の認可とドレイン調整を担当して終了し、サービス管理下ではインストール済みサービス マネージャーが置換プロセスを起動します。同じポートで、別の ID 検証済みプロセスが正常であることを確認した場合にのみ成功し、Codex インジェクションは破棄しません。これは、`POST /api/stop` の短いドレインよりも長く、情報に基づいたリサイクルです。 - **ゲートされた代替ストリーム パス** — 制限された単一リーダー リレー。 境界のないバッファリング形状を完全に削除します。 Windows では、バンドルされた Bun リリースに #32111 修正が確実に適用されると、これが自動的にデフォルトになります。現在はオプトインのみとなっています (以下を参照)。 macOS では、そのようなリリース後でもオプトインのままになります。macOS `auto` を切り替えるかどうかは別の決定となります。 diff --git a/docs-site/src/content/docs/ko/guides/grok-build.md b/docs-site/src/content/docs/ko/guides/grok-build.md index a8fe6e10e3..79bd048367 100644 --- a/docs-site/src/content/docs/ko/guides/grok-build.md +++ b/docs-site/src/content/docs/ko/guides/grok-build.md @@ -73,6 +73,6 @@ api_key = "your-OPENCODEX_API_AUTH_TOKEN" ## 알려진 제한 - **Responses 백엔드와 keep-alive:** 상위 업스트림이 조용한 동안 opencodex는 `/v1/responses` 스트림에 `response.heartbeat` keep-alive를 보냅니다. Grok Build의 Responses 디코더는 알 수 없는 이벤트 타입을 거부하므로, 수동으로 설정한 `api_backend = "responses"` 모델은 느린 업스트림에서 턴 도중 실패할 수 있습니다. 자동 등록된 항목은 `api_backend = "chat_completions"`로 고정되며, 원시 heartbeat 프레임을 노출하지 않습니다. -- **서비스 설치된 `ocx restart`:** opencodex가 서비스 관리자 아래에서 실행될 때 `ocx restart`는 현재 서비스를 멈추고 unmanaged 프로세스로 바꿉니다. 서비스 지속성(auto-restart, start-at-login)은 다음 `ocx service` 설정 전까지 사라지며, 그 unmanaged 프로세스가 죽으면 다음 `ocx start`/`ocx ensure`가 갱신하기 전까지 관리 블록이 죽은 프록시를 가리킬 수 있습니다. +- **서비스 설치된 `ocx restart`:** 실행 중인 프록시는 재시작 권한 확인과 드레인 조정을 담당하고, 기존 프로세스가 종료된 뒤 설치된 서비스 관리자가 교체 프로세스를 시작합니다. 서비스 감독은 그대로 유지됩니다. 루프백 자동 등록을 사용하는 경우에만 관리 블록도 핸드오프 동안 유지되며, 비루프백 배포에서는 Grok 설정을 수동으로 관리합니다. 같은 포트에서 신원이 확인된 다른 프로세스가 정상 상태가 된 뒤에만 명령이 성공합니다. - **설정 읽기 시점:** 가장 예측 가능한 결과를 얻으려면 opencodex를 먼저 시작하고 그다음 `grok`를 실행합니다. Grok Build는 `~/.grok/config.toml`을 감시하다가 `[model]` 테이블이 실제로 바뀔 때 다시 불러옵니다(내용을 기준으로 비교하는 약 1초 디바운스). 그래서 새로 고친 블록은 재시작 없이 열린 세션에도 들어갑니다. Grok가 무엇을 파싱했는지 확인하려면 `grok inspect`를 실행합니다. 이 명령은 로드한 설정 원본을 나열하고 거부한 필드가 있으면 경고합니다. 해석된 모델 목록은 출력하지 않습니다. TOML 오류 하나만으로도 사용자 설정 레이어 전체가 무효가 되므로, opencodex가 파일을 원자적으로 쓰는 이유도 여기에 있습니다. Grok는 절반만 써진 설정을 보지 않습니다. - **카탈로그 업데이트:** 펜스 블록은 주입 시점의 카탈로그를 반영합니다. 공급자나 모델을 추가한 뒤에는 `ocx ensure`를 실행하거나 프록시를 재시작해 갱신합니다. diff --git a/docs-site/src/content/docs/ko/reference/cli/lifecycle.md b/docs-site/src/content/docs/ko/reference/cli/lifecycle.md index 032a13d8ff..881dc9bc2d 100644 --- a/docs-site/src/content/docs/ko/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/ko/reference/cli/lifecycle.md @@ -36,8 +36,13 @@ ocx start --port 8080 ### `ocx restart` -`stop` 다음에 `ensure`를 실행합니다. 즉, 프록시/서비스를 중지하고 기본 Codex를 복원한 뒤, -프록시를 백그라운드에서 다시 시작하고 살아 있는 포트를 Codex에 다시 동기화합니다. +프록시가 실행 중이면 확인된 정확한 PID와 포트에 in-place 재시작을 요청하고, 정상 드레인을 +기다린 뒤 같은 포트에 다른 런타임 PID가 올라왔는지 확인합니다. 이 과정에서 관리형 라우팅과 +서비스 감시는 유지되며, 요청 결과가 불확실해도 별도의 stop/start로 재실행하지 않습니다. +실행 중인 프록시가 없을 때만 일반 `ensure` 시작 동작으로 전환합니다. +실행 중인 리스너를 런타임 PID로 증명할 수 없으면(업데이트 전 프록시 포함) `ensure`나 +stop/start 대체 동작 없이 안전하게 실패합니다. 소유권을 확인한 뒤 `ocx stop`과 `ocx start`를 +순서대로 한 번 실행하세요. ### `ocx ensure` diff --git a/docs-site/src/content/docs/ko/troubleshooting/windows-memory.md b/docs-site/src/content/docs/ko/troubleshooting/windows-memory.md index 4e53e3d5f3..3feb6b682a 100644 --- a/docs-site/src/content/docs/ko/troubleshooting/windows-memory.md +++ b/docs-site/src/content/docs/ko/troubleshooting/windows-memory.md @@ -23,7 +23,7 @@ Windows에서는 opencodex가 #32111 충돌을 피하기 위해 스트리밍 응 - **메모리 감시기** - 프록시는 1분마다 자체 메모리를 샘플링하고, 관측된 메모리가 4 GiB를 넘으면 속도 제한이 걸린 경고를 기록합니다. 관측된 메모리는 RSS, `external`, `arrayBuffers`의 합이 아니라 그중 가장 큰 값입니다. Windows의 working-set/RSS 카운터가 커밋된 external 잔존량을 낮게 잡을 수 있기 때문입니다. - **`ocx doctor`** - "Memory / runtime" 섹션에서 *서비스* 프로세스의 Bun 버전, RSS, external/ArrayBuffers 카운터, JS 힙 문맥, 스트림 모드 결정을 보여줍니다. 번들된 Bun 1.3.14 런타임에서는 `heapUsed` / `jscHeap`만으로 누수를 판별할 수 없습니다. 애플리케이션 수준 누수로 단정하기 전에 관측된 메모리, `responseState`, 반복 샘플을 함께 보아야 합니다. -- **`GET /api/system/memory`** - 대시보드나 스크립트에서 쓸 수 있도록 같은 데이터를 인증된 관리 API로 제공합니다. RSS/heap/external 카운터와 함께, 프록시의 메모리 내 `previous_response_id` 이어받기 저장소에 대한 스칼라 `responseState` 블록(항목 수, 직렬화된 총/최대 바이트, 가장 오래된 항목의 경과 시간)을 보고합니다. 이를 통해 증가 원인을 더 잘 구분할 수 있습니다. 관측된 메모리가 함께 증가하면서 `responseState.totalBytes`도 늘면 대화 보존(long `store:false` 체인이 매 턴 다시 확장되는 경우)을 가리키고, 관측된 메모리는 늘지만 `responseState`는 평평하면 그 저장소와는 무관한 원인을 가리킵니다. 값은 스칼라만 포함하며 요청 본문, 토큰, 경로, 계정 식별자는 포함하지 않습니다. 또한 읽기 동작은 부작용이 없습니다. 절대 prune하거나 evict하지 않습니다. 대시보드의 **Memory observability** 카드는 같은 필드를 렌더링하고, 확인을 거쳐야 하는 **Drain & restart** 동작도 제공합니다. 현재 활성 턴 수를 보여주고, 기존 503 + `Retry-After` 드레인과 같은 방식으로 최대 60초 동안 활성 턴을 기다린 뒤, 남아 있는 턴을 강제로 중단하고 Codex 주입을 해제하지 않은 채 라이브 포트의 `ocx start`(또는 실패했을 때만 동작하는 서비스 슈퍼바이저 재기동)를 통해 프록시를 재시작합니다. 이는 `POST /api/stop`의 짧은 드레인보다 더 길고, 더 많은 정보를 반영한 재순환입니다. +- **`GET /api/system/memory`** - 대시보드나 스크립트에서 쓸 수 있도록 같은 데이터를 인증된 관리 API로 제공합니다. RSS/heap/external 카운터와 함께, 프록시의 메모리 내 `previous_response_id` 이어받기 저장소에 대한 스칼라 `responseState` 블록(항목 수, 직렬화된 총/최대 바이트, 가장 오래된 항목의 경과 시간)을 보고합니다. 이를 통해 증가 원인을 더 잘 구분할 수 있습니다. 관측된 메모리가 함께 증가하면서 `responseState.totalBytes`도 늘면 대화 보존(long `store:false` 체인이 매 턴 다시 확장되는 경우)을 가리키고, 관측된 메모리는 늘지만 `responseState`는 평평하면 그 저장소와는 무관한 원인을 가리킵니다. 값은 스칼라만 포함하며 요청 본문, 토큰, 경로, 계정 식별자는 포함하지 않습니다. 또한 읽기 동작은 부작용이 없습니다. 절대 prune하거나 evict하지 않습니다. 대시보드의 **Memory observability** 카드는 같은 필드를 렌더링하고, 확인을 거쳐야 하는 **Drain & restart** 동작도 제공합니다. 현재 활성 턴 수를 보여주고, 기존 503 + `Retry-After` 드레인과 같은 방식으로 최대 60초 동안 활성 턴을 기다린 뒤, 남아 있는 턴을 강제로 중단하고 Codex 주입을 해제하지 않은 채 확인된 실행 프로세스가 스스로 교체되게 한 다음 같은 포트의 다른 PID를 검증합니다. 이는 `POST /api/stop`의 짧은 드레인보다 더 길고, 더 많은 정보를 반영한 재순환입니다. - **가드된 대체 스트림 경로** - unbounded buffering 형태를 완전히 제거하는 bounded single-reader relay입니다. Windows에서는 번들된 Bun 릴리스가 #32111 수정을 실제로 포함하고 있음이 확인되면 자동으로 기본값이 됩니다. 지금은 아래에서 설명하는 opt-in만 가능합니다. macOS에서는 그런 릴리스 이후에도 계속 opt-in입니다. macOS의 `auto`를 바꾸는 것은 별도의 결정입니다. 이 변경들로 실제 RSS가 얼마나 좋아지는지는 **Windows 사용자의 검증을 기다리고 있습니다**. 아직 이 누수가 해결되었다고 말하지는 않습니다. diff --git a/docs-site/src/content/docs/reference/cli/lifecycle.md b/docs-site/src/content/docs/reference/cli/lifecycle.md index 80e0a1d789..2751235286 100644 --- a/docs-site/src/content/docs/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/reference/cli/lifecycle.md @@ -36,8 +36,14 @@ The same action is available from the web dashboard's **Stop** button (`POST /ap ### `ocx restart` -Run `stop` followed by `ensure`: stop the proxy/service, restore native Codex, start the proxy in the -background, and sync the live port back into Codex. +When a proxy is running, ask that exact attested PID and port to restart in place, wait for its +normal drain, and verify a different runtime PID on the same port. Managed routing and service +supervision stay installed throughout; an uncertain request is observed rather than replayed as a +separate stop/start. If no proxy is running, the command falls back to the normal `ensure` start. +If a live listener cannot be attested to a runtime PID (including a pre-update proxy), restart fails +closed without an `ensure` or stop/start fallback. After confirming ownership, use `ocx stop` then +`ocx start` for a standalone proxy. For a service-managed proxy, use `ocx stop` followed by +`ocx service start` so supervision is restored. ### `ocx ensure` diff --git a/docs-site/src/content/docs/ru/guides/grok-build.md b/docs-site/src/content/docs/ru/guides/grok-build.md index d04473c609..096bb7aa39 100644 --- a/docs-site/src/content/docs/ru/guides/grok-build.md +++ b/docs-site/src/content/docs/ru/guides/grok-build.md @@ -113,11 +113,9 @@ api_key = "your-OPENCODEX_API_AUTH_TOKEN" `api_backend = "responses"` может оборваться посреди хода на медленных upstream. Автоматически зарегистрированные записи жёстко используют `api_backend = "chat_completions"`, где сырые heartbeat-кадры никогда не видны. -- **`ocx restart`, установленный как service:** когда opencodex работает под service manager, - `ocx restart` сейчас останавливает службу и заменяет её неуправляемым процессом — persistence - службы (автоперезапуск, старт при логине) теряется до следующего `ocx service`, а если этот - неуправляемый процесс погибнет, managed block может указывать на уже мёртвый прокси, пока - следующий `ocx start`/`ocx ensure` не обновит его. +- **`ocx restart` при установленной службе:** работающий прокси сам управляет drain и заменой, + поэтому supervision службы и managed block сохраняются. Команда завершается успешно только после + того, как на том же порту станет здоровым другой процесс с проверенной идентичностью. - **Время чтения конфигурации:** для наиболее предсказуемого поведения сначала запускайте opencodex, а затем `grok`. Grok Build отслеживает `~/.grok/config.toml` и перезагружает его, когда секция `[model]` действительно меняется (порядка секунды debounce, сравнение по diff --git a/docs-site/src/content/docs/ru/reference/cli/lifecycle.md b/docs-site/src/content/docs/ru/reference/cli/lifecycle.md index ded9943ad4..8d82fead7c 100644 --- a/docs-site/src/content/docs/ru/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/ru/reference/cli/lifecycle.md @@ -40,8 +40,13 @@ ocx start --port 8080 ### `ocx restart` -Выполнить `stop`, затем `ensure`: остановить прокси/службу, восстановить native Codex, поднять -прокси в фоне и синхронизировать живой порт обратно в Codex. +Если прокси уже работает, команда запрашивает перезапуск на месте у точно аттестованных PID и +порта, ждёт штатного drain и проверяет появление другого runtime PID на том же порту. Управляемая +маршрутизация и supervision службы сохраняются; неопределённый результат запроса не повторяется как +отдельный stop/start. Если прокси не запущен, используется обычный запуск через `ensure`. +Если работающий слушатель нельзя подтвердить привязкой к runtime PID (включая прокси до обновления), +перезапуск завершается безопасным отказом без `ensure` или stop/start. После проверки владения +выполните `ocx stop`, затем один раз `ocx start`. ### `ocx ensure` diff --git a/docs-site/src/content/docs/ru/troubleshooting/windows-memory.md b/docs-site/src/content/docs/ru/troubleshooting/windows-memory.md index 11fff3de88..7dd47d8da1 100644 --- a/docs-site/src/content/docs/ru/troubleshooting/windows-memory.md +++ b/docs-site/src/content/docs/ru/troubleshooting/windows-memory.md @@ -49,8 +49,8 @@ opencodex поставляет рантайм Bun (сейчас это **1.3.14* prune'ится и не evict'ится. Карточка **Memory observability** в дашборде показывает те же поля и даёт confirm-gated действие **Drain & restart**: она показывает текущее число активных ходов, ждёт до 60 секунд, пока они закончатся (используя уже существующий drain через 503 + - `Retry-After`), затем прерывает оставшиеся ходы и перезапускает прокси через `ocx start` на - текущем порту (или через respawn service supervisor только при аварии), не снимая внедрение + `Retry-After`), затем прерывает оставшиеся ходы, просит аттестованный живой процесс заменить себя + и проверяет другой PID на том же порту, не снимая внедрение в Codex. Это более длинный и осознанный recycle, чем короткий drain у `POST /api/stop`. - **Альтернативный stream path под флагом** — bounded single-reader relay, полностью убирающий форму неограниченной буферизации. На Windows он станет значением по умолчанию автоматически, diff --git a/docs-site/src/content/docs/troubleshooting/windows-memory.md b/docs-site/src/content/docs/troubleshooting/windows-memory.md index 54a9492d21..8bbcd8214b 100644 --- a/docs-site/src/content/docs/troubleshooting/windows-memory.md +++ b/docs-site/src/content/docs/troubleshooting/windows-memory.md @@ -53,9 +53,11 @@ runtime the leak itself remains an upstream problem: evicts). The dashboard's **Memory observability** card renders the same fields and offers a confirm-gated **Drain & restart** action: it shows the current active-turn count, waits up to 60s for active turns (reusing - the existing 503 + `Retry-After` drain), then aborts any remaining turns and - restarts the proxy via `ocx start` on the live port (or a failure-only - service supervisor respawn) without tearing down Codex injection. That is a + the existing 503 + `Retry-After` drain), then aborts any remaining turns. + The running proxy owns restart authorization and drain coordination, then + exits; an installed service manager launches the replacement when applicable. + The action reports success only after a different, identity-verified process + is healthy on the same port, without tearing down Codex injection. That is a longer, informed recycle than the short drain on `POST /api/stop`. - **A gated alternative stream path** — a bounded single-reader relay that removes the unbounded buffering shape entirely. On Windows it becomes the diff --git a/docs-site/src/content/docs/zh-cn/guides/grok-build.md b/docs-site/src/content/docs/zh-cn/guides/grok-build.md index 5e014c851e..ffd9e43254 100644 --- a/docs-site/src/content/docs/zh-cn/guides/grok-build.md +++ b/docs-site/src/content/docs/zh-cn/guides/grok-build.md @@ -73,6 +73,6 @@ api_key = "your-OPENCODEX_API_AUTH_TOKEN" ## 已知限制 - **Responses 后端与保活:** opencodex 在 `/v1/responses` 流上、上游静默期间会发送 `response.heartbeat` 保活事件。Grok Build 的 Responses 解码器会拒绝未知事件类型,因此手动配置为 `api_backend = "responses"` 的模型在上游较慢时可能会在对话中途失败。自动注册的条目会固定为 `api_backend = "chat_completions"`,这样就不会暴露原始的心跳帧。 -- **服务安装后的 `ocx restart`:** 当 opencodex 运行在服务管理器下时,`ocx restart` 目前会停止该服务,并将其替换为一个非受管进程——服务持久化能力(自动重启、登录时启动)会丢失,直到下一次 `ocx service` 设置完成;如果这个非受管进程退出,受管理区块可能会指向一个已失效的代理,直到下一次 `ocx start`/`ocx ensure` 刷新它。 +- **服务安装后的 `ocx restart`:** 运行中的代理负责重启授权和排空协调;旧进程退出后,由已安装的服务管理器启动替换进程。服务监督始终保留。仅在 loopback 自动注册模式下,受管理区块也会在交接期间保留;非 loopback 部署使用手动管理的 Grok 配置。只有确认同一端口上出现另一个经过身份验证且健康的进程后,命令才会成功。 - **配置读取时机:** 先启动 opencodex,再启动 `grok`,结果最可预测。Grok Build 会监视 `~/.grok/config.toml`,并在 `[model]` 表实际发生变化时重新加载(大约一秒的防抖,按内容比较),因此刷新后的区块可以在无需重启的情况下进入已打开的会话。要确认 Grok 解析到了什么,可以运行 `grok inspect`:它会列出已加载的配置来源,并提示被拒绝的字段,但不会打印最终解析出的模型列表。注意,单个 TOML 错误会使*整个*用户配置层失效,这也是 opencodex 以原子方式写入文件的原因——Grok 不会看到半写入的配置。 - **目录更新:** 有边界线的区块反映的是注入时的目录状态。添加提供方或模型后,运行 `ocx ensure`(或重启代理)以刷新它。 diff --git a/docs-site/src/content/docs/zh-cn/reference/cli/lifecycle.md b/docs-site/src/content/docs/zh-cn/reference/cli/lifecycle.md index e2a4605057..a8d64d852d 100644 --- a/docs-site/src/content/docs/zh-cn/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/zh-cn/reference/cli/lifecycle.md @@ -28,7 +28,9 @@ ocx start --port 8080 ### `ocx restart` -执行 `stop` 然后执行 `ensure`:停止代理/服务,恢复原生 Codex,在后台启动代理,并将当前端口重新同步回 Codex。 +代理正在运行时,请求经过验证的准确 PID 和端口执行原位重启,等待正常排空,并确认同一端口上出现不同的运行时 PID。整个过程保留托管路由和服务监督;若请求结果不确定,也不会将其重放为单独的 stop/start。只有没有代理运行时,命令才回退到常规的 `ensure` 启动。 + +如果无法将正在运行的监听器验证为对应的运行时 PID(包括升级前的代理),重启会安全失败,不会回退到 `ensure` 或 stop/start。确认所有权后,请依次运行一次 `ocx stop` 和 `ocx start`。 ### `ocx ensure` diff --git a/docs-site/src/content/docs/zh-cn/troubleshooting/windows-memory.md b/docs-site/src/content/docs/zh-cn/troubleshooting/windows-memory.md index 8d3ae224d8..7734ebe23d 100644 --- a/docs-site/src/content/docs/zh-cn/troubleshooting/windows-memory.md +++ b/docs-site/src/content/docs/zh-cn/troubleshooting/windows-memory.md @@ -23,7 +23,7 @@ opencodex 打包了 Bun 运行时(当前为 **1.3.14**)。这类内存增长 - **内存监视器** — 代理每分钟采样一次自身内存,并在观测到的内存超过 4 GiB 时记录限频告警。观测到的内存取 RSS、`external` 和 `arrayBuffers` 三者中的最大值(不是它们的总和),因为 Windows 的工作集/RSS 计数可能低报已提交的外部保留。 - **`ocx doctor`** — `"Memory / runtime"` 部分会显示*服务*进程的 Bun 版本、RSS、external/ArrayBuffers 计数、JS 堆上下文以及流模式决策。在捆绑的 Bun 1.3.14 运行时上,单看 `heapUsed` / `jscHeap` 不能作为泄漏判据;在认定为应用层泄漏之前,应把观测到的内存与 `responseState` 以及多次采样一起比较。 -- **`GET /api/system/memory`** — 通过已认证的管理 API 提供同样的数据,便于仪表板或脚本使用。除了 RSS/heap/external 计数之外,它还会报告一个标量的 `responseState` 块(条目数、序列化总字节数/最大字节数、最老条目的年龄),对应代理内存中的 `previous_response_id` 续接存储。这能进一步归因增长:在观测到的内存上升时,如果 `responseState.totalBytes` 也在上升,说明是对话保留在增长(较长的 `store:false` 链在每轮中重新扩张);而在观测到的内存上升时,如果 `responseState` 保持平稳,则更像不是这个存储造成的。返回值只包含标量,不包含请求正文、token、路径或账户标识,而且读取没有副作用(不会执行 prune,也不会 evict)。仪表板中的 **Memory observability** 卡片会渲染相同字段,并提供一个需要确认的 **Drain & restart** 操作:它会显示当前活动轮次数量,最多等待 60 秒让活动轮次结束(复用现有的 503 + `Retry-After` 排空机制),然后中止剩余轮次,并通过 `ocx start` 在当前端口重启代理(或者在仅故障时由服务监督程序重新拉起),同时不拆除 Codex 注入。这是一种比 `POST /api/stop` 的短排空更长、更知情的回收方式。 +- **`GET /api/system/memory`** — 通过已认证的管理 API 提供同样的数据,便于仪表板或脚本使用。除了 RSS/heap/external 计数之外,它还会报告一个标量的 `responseState` 块(条目数、序列化总字节数/最大字节数、最老条目的年龄),对应代理内存中的 `previous_response_id` 续接存储。这能进一步归因增长:在观测到的内存上升时,如果 `responseState.totalBytes` 也在上升,说明是对话保留在增长(较长的 `store:false` 链在每轮中重新扩张);而在观测到的内存上升时,如果 `responseState` 保持平稳,则更像不是这个存储造成的。返回值只包含标量,不包含请求正文、token、路径或账户标识,而且读取没有副作用(不会执行 prune,也不会 evict)。仪表板中的 **Memory observability** 卡片会渲染相同字段,并提供一个需要确认的 **Drain & restart** 操作:它会显示当前活动轮次数量,最多等待 60 秒让活动轮次结束(复用现有的 503 + `Retry-After` 排空机制),然后中止剩余轮次。运行中的代理负责重启授权和排空协调并退出;如果由服务管理器托管,则由已安装的服务管理器启动替换进程。只有确认同一端口上另一个经过身份验证且健康的进程后才会报告成功,同时不会拆除 Codex 注入。这是一种比 `POST /api/stop` 的短排空更长、更知情的回收方式。 - **有门控的替代流路径** — 一种有界的单读者中继,能彻底消除无界缓冲的形态。在 Windows 上,一旦某个捆绑的 Bun 版本可验证地包含 #32111 的修复,它就会自动成为默认路径;目前它仅支持显式启用(见下文)。在 macOS 上,即使到了那样的版本,它也仍然保持显式启用状态;切换 macOS 的 `auto` 是另一项独立决定。 这些改动带来的真实世界 RSS 改善,仍在等待 Windows 用户验证,我们并不宣称泄漏已经修复。 diff --git a/src/cli/index.ts b/src/cli/index.ts index ace0ee130f..5a995987dd 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -21,7 +21,15 @@ import { } from "../config"; import { collectStatus } from "./status"; import { dispatchInternalCliCommand, type InternalCliCommand } from "./internal-dispatch"; -import { runTrayProxyRestart, runTrayProxyStart } from "./tray-proxy"; +import { + discoverStableProxyForRestart, + isProxyReplacement, + runProxyRestart, + runTrayProxyStart, + type ProxyRestartLive, + type ProxyRestartResult, +} from "./tray-proxy"; +import { requestBoundSystemRestart } from "./system-restart-client"; import { installCrashGuards } from "../lib/crash-guard"; import { hasHelpFlag, printSubcommandUsage, printUsage, printVersion } from "./help"; import { findAvailablePort, isAddrInUse, PortUnavailableError, shouldPersistSelectedPort, waitForPortAvailable } from "../server/ports"; @@ -50,6 +58,7 @@ import { removeOwnedConfigState } from "../lib/config-ownership"; import { withProcessRuntimeProvenance } from "../lib/bun-runtime"; import { initializeNodeLauncherContext } from "./launcher-context"; import { createLocalAttestationSecret } from "../lib/local-management-attestation"; +import { MEMORY_DRAIN_RESTART_MS, REPLACEMENT_READY_TIMEOUT_MS } from "../lib/system-restart-contract"; initializeNodeLauncherContext(); const args = process.argv.slice(2); @@ -298,7 +307,8 @@ async function handleStart(options: { block?: boolean } = {}) { } removePid(process.pid); removeRuntimePort(process.pid); - if (!recycling && !process.env.OCX_SERVICE && !currentExternalCodexModelProvider()) { + const preserveRouting = process.env.OCX_SERVICE === "1"; + if (!recycling && !preserveRouting && !currentExternalCodexModelProvider()) { try { const restored = restoreNativeCodex(); if (!restored.success) { @@ -314,7 +324,7 @@ async function handleStart(options: { block?: boolean } = {}) { // Grok fence is shared state we must not remove — that service keeps running and would be // left pointing nowhere. This guard also covers signal-driven exits, which is the path that // would otherwise bypass handleStop's gate entirely. - if (!recycling && !process.env.OCX_SERVICE && serviceEnvironmentOwnedHere()) { + if (!recycling && !preserveRouting && serviceEnvironmentOwnedHere()) { try { stripGrokConfig(); } catch { /* best-effort restore */ } } return cleanupSucceeded; @@ -419,15 +429,27 @@ async function handleStart(options: { block?: boolean } = {}) { } } -async function handleEnsure() { +function detachedStartEnvironment(): NodeJS.ProcessEnv { + const env: NodeJS.ProcessEnv = { ...process.env }; + // Only a real service wrapper may claim supervision. A detached ensure/tray child + // is an ordinary owner: while live it maintains routing, and on exit it restores it. + delete env.OCX_SERVICE; + return withProcessRuntimeProvenance(env); +} + +async function handleEnsure(options: { existingIsSuccess?: boolean } = {}): Promise { if (!currentExternalCodexModelProvider()) reconcileJournal(); const config = loadConfig(); if (!codexAutoStartEnabled(config)) { console.log("Codex autostart is disabled."); - return; + return false; } const live = await findLiveProxy(); if (live) { + if (options.existingIsSuccess === false) { + console.error("Proxy appeared while restart was confirming absence; no start was attempted."); + return false; + } const synced = await syncModelsToCodex(live.port).catch(e => { console.error(`⚠️ Model sync skipped: ${e instanceof Error ? e.message : String(e)}`); return null; @@ -444,7 +466,7 @@ async function handleEnsure() { else if (!g.ok) console.error(`⚠️ ${g.message}`); } catch (err) { console.error(`⚠️ ${grokSyncFailureMessage(err)}`); } console.log(`✅ Proxy running on port ${live.port}`); - return; + return true; } const pinPort = config.port ?? 10100; @@ -452,14 +474,15 @@ async function handleEnsure() { detached: true, stdio: "ignore", windowsHide: true, - env: withProcessRuntimeProvenance({ ...process.env, OCX_SERVICE: "1" }), + env: detachedStartEnvironment(), }); child.unref(); const port = (await waitForProxy())?.port; if (!port) { console.error("❌ Proxy did not become healthy after starting."); - process.exit(1); + process.exitCode = 1; + return false; } // Deterministic fence guarantee: the spawned child injects late in its own startup, but // this parent returns as soon as /healthz responds — inject here too (idempotent block @@ -478,12 +501,14 @@ async function handleEnsure() { }); if (synced?.status === "skipped") console.log(" Codex integration OFF; startup left Codex native."); console.log(`✅ Proxy running on port ${port}`); + return true; } /** Fixed tray action: start the proxy without depending on codexAutoStart. */ -async function handleTrayProxyStart(): Promise { +async function handleTrayProxyStart(existingIsSuccess = true): Promise { const ok = await runTrayProxyStart({ findLive: findLiveProxy, + existingIsSuccess, diagnoseService: () => { const service = diagnoseService(); return { installed: service.installed, startable: serviceStartableFromTray(service), summary: service.summary }; @@ -496,29 +521,117 @@ async function handleTrayProxyStart(): Promise { detached: true, stdio: "ignore", windowsHide: true, - env: withProcessRuntimeProvenance({ ...process.env, OCX_SERVICE: "1" }), + env: detachedStartEnvironment(), }); child.unref(); }, - waitForProxy, + // serviceCommand("start") already spends up to 20s confirming the supervised + // child. Slow Windows hosts can still be publishing native-main state after that + // first window, so keep one shared follow-up budget instead of returning a false + // failure while Task Scheduler is still starting the approved child. + waitForProxy: () => waitForProxy(40_000), info: message => console.log(message), error: message => console.error(message), }); - if (!ok) process.exitCode = 1; + // serviceCommand("start") can set exitCode=1 after its own 20s probe, while + // the coordinator's bounded follow-up observes the same service become live. + // The final observed state, not the earlier probe, owns this command result. + process.exitCode = ok ? 0 : 1; + return ok; } -async function handleTrayProxyRestart(): Promise { - const ok = await runTrayProxyRestart({ - stop: async () => { - await handleStop(); - return !process.exitCode || process.exitCode === 0; - }, - start: async () => { - await handleTrayProxyStart(); - return !process.exitCode || process.exitCode === 0; - }, +const PROXY_RESTART_OBSERVE_MS = MEMORY_DRAIN_RESTART_MS + REPLACEMENT_READY_TIMEOUT_MS + 15_000; + +async function waitForProxyReplacement( + previous: ProxyRestartLive, + deadlineAt: number, +): Promise { + while (Date.now() < deadlineAt) { + const live = await findLiveProxy({ deadlineAt }); + if (Date.now() >= deadlineAt) return null; + // Modern /healthz publishes a PID. Require a different, identity-verified process; + // merely seeing the old port online again is not proof that restart completed. + if (isProxyReplacement(previous, live)) { + return live; + } + const remainingMs = deadlineAt - Date.now(); + if (remainingMs > 0) await Bun.sleep(Math.min(250, remainingMs)); + } + return null; +} + +function reportRestartFailure(result: Extract): void { + if (result.phase === "identity") { + console.error("❌ Refusing to restart because the running proxy identity could not be attested."); + } else if (result.phase === "request") { + const code = result.error instanceof Error ? result.error.message : ""; + if (code === "restart_capability_unsupported") { + console.error("❌ The running proxy predates process-bound restart support; no unsafe fallback was attempted."); + console.error(" After confirming this home owns the proxy, run `ocx stop` and then `ocx start` once."); + } else { + console.error("❌ Proxy restart request could not be confirmed; no fallback stop/start was attempted."); + } + } else if (result.phase === "replacement") { + console.error("❌ Proxy restart was accepted, but no identity-verified replacement became healthy in time."); + } else { + console.error("❌ Proxy was not running and the fallback start did not become healthy."); + } +} + +async function handleProxyRestart( + startWhenStopped: () => Promise, +): Promise { + const deadlineAt = Date.now() + PROXY_RESTART_OBSERVE_MS; + const result = await runProxyRestart({ + findLive: () => discoverStableProxyForRestart({ + findLive: () => findLiveProxy({ deadlineAt, attempts: 2 }), + expired: () => Date.now() >= deadlineAt, + }), + startWhenStopped, + requestInPlaceRestart: previous => requestBoundSystemRestart(previous, deadlineAt), + waitForReplacement: previous => waitForProxyReplacement(previous, deadlineAt), }); - if (!ok) process.exitCode = 1; + if (!result.ok) reportRestartFailure(result); + process.exitCode = result.ok ? 0 : 1; + return result.ok; +} + +async function handleTrayProxyRestart(): Promise { + await handleProxyRestart(() => handleTrayProxyStart(false)); +} + +async function handleRestartStartWhenStopped(): Promise { + if (!codexAutoStartEnabled(loadConfig())) { + console.log("Codex autostart is disabled; no proxy was started."); + return "skipped"; + } + return handleEnsure({ existingIsSuccess: false }); +} + +async function restoreSharedClientStateAfterStop(): Promise { + let restored = true; + try { + const result = await restoreNativeCodexAsync(); + if (result.success) console.log(`↩️ ${result.message}`); + else { + restored = false; + console.error(`⚠️ ${result.message}`); + } + } catch (error) { + restored = false; + console.error(`⚠️ Native Codex restore failed: ${error instanceof Error ? error.message : String(error)}`); + } + + // A refused or thrown Grok strip is actionable because it would point Grok at a dead proxy. + try { + const grok = stripGrokConfig(); + if (grok.changed) console.log(`↩️ ${grok.message}`); + else if (!grok.ok) { restored = false; console.error(`⚠️ ${grok.message}`); } + } catch (error) { + restored = false; + console.error(`⚠️ Grok config restore failed: ${error instanceof Error ? error.message : String(error)}`); + } + return restored; } async function handleStop() { @@ -592,26 +705,11 @@ async function handleStop() { removeRuntimePortIfPidIs(staleRuntimePid); } } - if (!ownershipBlocked) { - const r = await restoreNativeCodexAsync(); - if (r.success) console.log(`↩️ ${r.message}`); - else { - stopFailed = true; - console.error(`⚠️ ${r.message}`); - } - } - // revertSystemEnv is NOT gated: it carries its own ownership check and concerns launchctl - // user env, not CODEX_HOME. Safety net for when the daemon's syncCleanup didn't run (SIGKILL). + // Environment ownership is independent from service ownership. Always roll back + // current-home variables; the helper refuses foreign markers on its own. try { revertSystemEnv(); } catch { /* best-effort */ } if (!ownershipBlocked) { - // Same safety net for the Grok Build managed block (marker-owned, idempotent). - try { - const g = stripGrokConfig(); - if (g.changed) console.log(`↩️ ${g.message}`); - // A refused strip (e.g. orphaned marker) leaves the fence pointing at a dead proxy — - // reporting success there hides a broken end state. - else if (!g.ok) { stopFailed = true; console.error(`⚠️ ${g.message}`); } - } catch { /* best-effort */ } + if (!await restoreSharedClientStateAfterStop()) stopFailed = true; } // Set the code rather than exiting inline: `restart` and the tray coordinator call this // function and need it to RETURN so they can decide what to do next. @@ -1095,7 +1193,7 @@ switch (command) { case "__tray-restart": case "__startup-health": await dispatchInternalCliCommand(command as InternalCliCommand, { - trayStart: handleTrayProxyStart, + trayStart: async () => { await handleTrayProxyStart(); }, trayRestart: handleTrayProxyRestart, startupHealth: async () => { const { collectStartupHealth } = await import("../codex/autostart-health"); @@ -1116,10 +1214,9 @@ switch (command) { break; } case "restart": { - // A failed stop must not be followed by a re-inject: with a foreign service still running - // (ownership mismatch) we would rewrite shared config we just declined to touch. - if (await handleStop()) await handleEnsure(); - else console.error("↩️ Restart aborted: the proxy was not stopped cleanly."); + // The running proxy owns its drain and replacement through /api/system/restart. + // If nothing is live, restart degrades to the documented `ensure` start behavior. + await handleProxyRestart(handleRestartStartWhenStopped); break; } case "health": { diff --git a/src/cli/system-restart-client.ts b/src/cli/system-restart-client.ts new file mode 100644 index 0000000000..b059f07076 --- /dev/null +++ b/src/cli/system-restart-client.ts @@ -0,0 +1,146 @@ +import { readRuntimePort } from "../config"; +import { + LOCAL_ATTESTATION_CHALLENGE_HEADER, + LOCAL_ATTESTATION_PROOF_HEADER, + createLocalAttestationChallenge, + verifyLocalAttestationProof, +} from "../lib/local-management-attestation"; +import { + SYSTEM_RESTART_CAPABILITY_HEADER, + SYSTEM_RESTART_CAPABILITY_VERSION, + SYSTEM_RESTART_EXPECTED_PID_HEADER, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_NONCE_HEADER, + SYSTEM_RESTART_PATH, + createSystemRestartCapability, +} from "../lib/system-restart-contract"; +import { + findLiveProxy, + isOpencodexHealthz, + probeHostname, + type HealthzIdentity, + type LiveProxy, +} from "../server/proxy-liveness"; +import type { ProxyRestartRequestOutcome } from "./tray-proxy"; + +export const SYSTEM_RESTART_REQUEST_TIMEOUT_MS = 5_000; +export const SYSTEM_RESTART_ATTESTATION_TIMEOUT_MS = 4_000; + +export interface BoundSystemRestartDeps { + fetchImpl?: typeof fetch; + readRuntime?: typeof readRuntimePort; + findLive?: typeof findLiveProxy; + createChallenge?: () => string; + now?: () => number; +} + +function rejected(code: string): ProxyRestartRequestOutcome { + return { accepted: false, uncertain: false, error: new Error(code) }; +} + +function uncertain(code: string): ProxyRestartRequestOutcome { + return { accepted: false, uncertain: true, error: new Error(code) }; +} + +function remaining(deadlineAt: number, now: () => number, cap: number): number { + return Math.max(0, Math.min(cap, deadlineAt - now())); +} + +function sameRestartTarget(expected: LiveProxy, observed: LiveProxy | null): boolean { + return expected.pid !== null + && observed?.source === "runtime" + && observed.pid === expected.pid + && observed.port === expected.port; +} + +/** + * Send one restart request to the exact runtime proxy observed by the caller. + * + * No reusable admin credential is sent. After the listener proves possession of + * its per-process runtime secret, the client derives a capability bound to this + * method, path, PID, and port. The expected PID is repeated so a replacement that + * wins the port between proof and POST rejects the request. + */ +export async function requestBoundSystemRestart( + target: LiveProxy, + deadlineAt: number, + deps: BoundSystemRestartDeps = {}, +): Promise { + if (target.source !== "runtime" || target.pid === null) return rejected("restart_target_unattested"); + + const now = deps.now ?? Date.now; + const readRuntime = deps.readRuntime ?? readRuntimePort; + const runtime = readRuntime(target.pid); + if (!runtime?.attestationSecret || runtime.pid !== target.pid || runtime.port !== target.port) { + return rejected("restart_target_runtime_mismatch"); + } + + const attestationBudget = remaining(deadlineAt, now, SYSTEM_RESTART_ATTESTATION_TIMEOUT_MS); + if (attestationBudget <= 0) return rejected("restart_deadline_expired"); + + const fetchImpl = deps.fetchImpl ?? fetch; + const challenge = (deps.createChallenge ?? createLocalAttestationChallenge)(); + const baseUrl = `http://${probeHostname(target.hostname)}:${target.port}`; + let proofResponse: Response; + try { + proofResponse = await fetchImpl(`${baseUrl}/healthz`, { + headers: { [LOCAL_ATTESTATION_CHALLENGE_HEADER]: challenge }, + signal: AbortSignal.timeout(attestationBudget), + }); + } catch { + return rejected("restart_attestation_unreachable"); + } + const body = await proofResponse.json().catch(() => null) as HealthzIdentity | null; + const proof = proofResponse.headers.get(LOCAL_ATTESTATION_PROOF_HEADER); + if ( + !proofResponse.ok + || !isOpencodexHealthz(body) + || body?.pid !== target.pid + || !verifyLocalAttestationProof(runtime.attestationSecret, challenge, target.pid, target.port, proof) + ) { + return rejected("restart_attestation_failed"); + } + if (body.restartCapability !== SYSTEM_RESTART_CAPABILITY_VERSION) { + // A pre-update proxy accepts only the reusable management credential and cannot + // bind the operation to the attested PID. Refuse before POST rather than weakening + // the exact-process contract or replaying a stop/start transaction. + return rejected("restart_capability_unsupported"); + } + + let observed: LiveProxy | null; + try { + observed = await (deps.findLive ?? findLiveProxy)({ deadlineAt, nowFn: now }); + } catch { + return rejected("restart_target_recheck_failed"); + } + if (!sameRestartTarget(target, observed)) return rejected("restart_target_changed"); + + const capability = createSystemRestartCapability( + runtime.attestationSecret, + challenge, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_PATH, + target.pid, + target.port, + ); + if (!capability) return rejected("restart_capability_unavailable"); + + const requestBudget = remaining(deadlineAt, now, SYSTEM_RESTART_REQUEST_TIMEOUT_MS); + if (requestBudget <= 0) return rejected("restart_deadline_expired"); + try { + const response = await fetchImpl(`${baseUrl}${SYSTEM_RESTART_PATH}`, { + method: SYSTEM_RESTART_METHOD, + headers: { + [SYSTEM_RESTART_EXPECTED_PID_HEADER]: String(target.pid), + [SYSTEM_RESTART_NONCE_HEADER]: challenge, + [SYSTEM_RESTART_CAPABILITY_HEADER]: capability, + }, + signal: AbortSignal.timeout(requestBudget), + }); + return response.ok ? { accepted: true } : rejected(`restart_request_http_${response.status}`); + } catch { + // The server may have accepted the restart before the response connection failed. + // The coordinator observes the original PID for replacement instead of replaying. + return uncertain("restart_request_outcome_unknown"); + } +} diff --git a/src/cli/tray-proxy.ts b/src/cli/tray-proxy.ts index 00559fb451..bf1075d8f5 100644 --- a/src/cli/tray-proxy.ts +++ b/src/cli/tray-proxy.ts @@ -8,6 +8,8 @@ export interface TrayProxyServiceState { export interface TrayProxyStartIo { findLive: () => Promise; + /** Normal Start is idempotent; restart fallback refuses a target that reappeared. */ + existingIsSuccess?: boolean; diagnoseService: () => TrayProxyServiceState; startService: () => void | Promise; startDirect: () => void | Promise; @@ -16,10 +18,101 @@ export interface TrayProxyStartIo { error: (message: string) => void; } +export interface ProxyRestartLive { + pid: number | null; + port: number; + hostname?: string; + source: "runtime" | "config"; +} + +export type ProxyRestartRequestOutcome = + | { accepted: true } + | { accepted: false; uncertain: boolean; error?: unknown }; + +export type ProxyRestartResult = + | { ok: true; mode: "started" } + | { ok: true; mode: "skipped" } + | { ok: true; mode: "restarted"; live: ProxyRestartLive } + | { ok: false; phase: "start" | "identity" | "request" | "replacement"; error?: unknown }; + +export type ProxyRestartDiscovery = + | { status: "live"; live: ProxyRestartLive } + | { status: "absent" } + | { status: "uncertain"; error?: unknown }; + +export interface ProxyRestartDiscoveryIo { + findLive: () => Promise; + waitBetweenChecks?: () => Promise; + expired?: () => boolean; +} + +export interface ProxyRestartIo { + findLive: () => Promise; + startWhenStopped: () => boolean | "skipped" | Promise; + requestInPlaceRestart: ( + previous: ProxyRestartLive, + ) => ProxyRestartRequestOutcome | Promise; + waitForReplacement: (previous: ProxyRestartLive) => Promise; +} + +/** + * Confirm absence twice before restart is allowed to select a start-only path. + * A live target appearing during confirmation is uncertainty, not success: the + * caller must not claim it restarted a process it never asked to restart. + */ +export async function discoverStableProxyForRestart( + io: ProxyRestartDiscoveryIo, +): Promise { + let first: ProxyRestartLive | null; + try { + first = await io.findLive(); + } catch (error) { + return { status: "uncertain", error }; + } + if (first) return { status: "live", live: first }; + if (io.expired?.()) { + return { status: "uncertain", error: new Error("restart_discovery_deadline_expired") }; + } + + await (io.waitBetweenChecks ?? (() => Bun.sleep(100)))(); + let second: ProxyRestartLive | null; + try { + second = await io.findLive(); + } catch (error) { + return { status: "uncertain", error }; + } + if (second) { + return { + status: "uncertain", + error: new Error("restart_target_appeared_during_absence_confirmation"), + }; + } + if (io.expired?.()) { + return { status: "uncertain", error: new Error("restart_discovery_deadline_expired") }; + } + return { status: "absent" }; +} + +export function isProxyReplacement( + previous: ProxyRestartLive, + candidate: ProxyRestartLive | null, +): candidate is ProxyRestartLive & { pid: number } { + return previous.pid !== null + && candidate?.pid !== null + && candidate?.pid !== undefined + && candidate.source === "runtime" + && candidate.port === previous.port + && candidate.pid !== previous.pid; +} + /** Side-effect coordinator for the tray's fixed proxy-start action. */ export async function runTrayProxyStart(io: TrayProxyStartIo): Promise { const live = await io.findLive(); if (live) { + if (io.existingIsSuccess === false) { + io.error("Proxy appeared while restart was confirming absence; no start was attempted."); + return false; + } io.info(`Proxy already running on port ${live.port}.`); return true; } @@ -43,10 +136,64 @@ export async function runTrayProxyStart(io: TrayProxyStartIo): Promise return true; } -export async function runTrayProxyRestart(io: { - stop: () => boolean | Promise; - start: () => boolean | Promise; -}): Promise { - if (!await io.stop()) return false; - return io.start(); +/** + * Shared restart transaction for both `ocx restart` and the Windows tray. + * + * A live proxy restarts itself through POST /api/system/restart. That lifecycle owns + * drain, supervisor handoff, exact replacement identity, and managed-routing + * preservation. Re-implementing restart as `stop` + `start` here races a late service + * child and lets ordinary /api/stop restore native routing between the two halves. + * When no proxy is live there is nothing to recycle, so restart degrades to the + * caller's normal start path. + */ +export async function runProxyRestart(io: ProxyRestartIo): Promise { + let discovery: ProxyRestartDiscovery; + try { + discovery = await io.findLive(); + } catch (error) { + return { ok: false, phase: "request", error }; + } + + if (discovery.status === "uncertain") { + return { ok: false, phase: "request", error: discovery.error }; + } + + if (discovery.status === "absent") { + try { + const started = await io.startWhenStopped(); + if (started === "skipped") return { ok: true, mode: "skipped" }; + return started ? { ok: true, mode: "started" } : { ok: false, phase: "start" }; + } catch (error) { + return { ok: false, phase: "start", error }; + } + } + + const previous = discovery.live; + + if (previous.pid === null || previous.source !== "runtime") { + return { ok: false, phase: "identity" }; + } + + let request: ProxyRestartRequestOutcome; + try { + request = await io.requestInPlaceRestart(previous); + } catch (error) { + // The request may have reached the proxy before the response connection failed. + // Keep observing the original identity; never replay or fall back to stop/start. + request = { accepted: false, uncertain: true, error }; + } + + if (!request.accepted && !request.uncertain) { + return { ok: false, phase: "request", error: request.error }; + } + + try { + const replacement = await io.waitForReplacement(previous); + if (replacement) return { ok: true, mode: "restarted", live: replacement }; + return request.accepted + ? { ok: false, phase: "replacement" } + : { ok: false, phase: "request", error: request.error }; + } catch (error) { + return { ok: false, phase: "replacement", error }; + } } diff --git a/src/lib/system-restart-contract.ts b/src/lib/system-restart-contract.ts new file mode 100644 index 0000000000..2ec6ff8544 --- /dev/null +++ b/src/lib/system-restart-contract.ts @@ -0,0 +1,73 @@ +import { createHmac, timingSafeEqual } from "node:crypto"; +import { isLocalAttestationSecret } from "./local-management-attestation"; + +export const SYSTEM_RESTART_METHOD = "POST"; +export const SYSTEM_RESTART_PATH = "/api/system/restart"; +export const SYSTEM_RESTART_CAPABILITY_VERSION = "v1"; +export const SYSTEM_RESTART_EXPECTED_PID_HEADER = "x-opencodex-restart-expected-pid"; +export const SYSTEM_RESTART_NONCE_HEADER = "x-opencodex-restart-nonce"; +export const SYSTEM_RESTART_CAPABILITY_HEADER = "x-opencodex-restart-capability"; + +/** Fixed drain and replacement budgets shared by the server, CLI, and tray. */ +export const MEMORY_DRAIN_RESTART_MS = 60_000; +export const REPLACEMENT_READY_TIMEOUT_MS = 70_000; + +const BASE64URL_256 = /^[A-Za-z0-9_-]{43}$/; + +export type ExpectedSystemRestartPid = + | { kind: "absent" } + | { kind: "invalid" } + | { kind: "present"; pid: number }; + +export function parseExpectedSystemRestartPid(value: string | null): ExpectedSystemRestartPid { + if (value === null) return { kind: "absent" }; + if (!/^[1-9]\d*$/.test(value)) return { kind: "invalid" }; + const pid = Number(value); + return Number.isSafeInteger(pid) ? { kind: "present", pid } : { kind: "invalid" }; +} + +function restartCapabilityPayload( + nonce: string, + method: string, + path: string, + pid: number, + port: number, +): string | null { + if (!BASE64URL_256.test(nonce)) return null; + if (method !== SYSTEM_RESTART_METHOD || path !== SYSTEM_RESTART_PATH) return null; + if (!Number.isSafeInteger(pid) || pid <= 0) return null; + if (!Number.isInteger(port) || port <= 0 || port > 65535) return null; + return `opencodex-system-restart-v1\n${nonce}\n${method}\n${path}\n${pid}\n${port}`; +} + +/** Process-scoped, operation-only authorization. It is not a reusable management credential. */ +export function createSystemRestartCapability( + secret: string, + nonce: string, + method: string, + path: string, + pid: number, + port: number, +): string | null { + if (!isLocalAttestationSecret(secret)) return null; + const payload = restartCapabilityPayload(nonce, method, path, pid, port); + if (!payload) return null; + return createHmac("sha256", secret).update(payload).digest("base64url"); +} + +export function verifySystemRestartCapability( + secret: string, + nonce: string | null, + method: string, + path: string, + pid: number, + port: number, + capability: string | null, +): boolean { + if (!nonce || !capability || !BASE64URL_256.test(capability)) return false; + const expected = createSystemRestartCapability(secret, nonce, method, path, pid, port); + if (!expected) return false; + const expectedBytes = Buffer.from(expected); + const actualBytes = Buffer.from(capability); + return expectedBytes.length === actualBytes.length && timingSafeEqual(expectedBytes, actualBytes); +} diff --git a/src/lib/winsw.ts b/src/lib/winsw.ts index d84854719a..ea7558fb20 100644 --- a/src/lib/winsw.ts +++ b/src/lib/winsw.ts @@ -328,7 +328,23 @@ export async function installWinswService(entry: WinswEntry, deps: WinswInstallD } export function startWinswService(): void { runWinsw(["start"]); } -export function stopWinswService(): void { try { runWinsw(["stopwait"]); } catch { /* not running */ } } + +/** + * Stop the native service and prove it is no longer running. `stopwait` can fail both + * for the benign already-stopped case and for real access/timeout failures, so a bare + * catch cannot decide whether it is safe for lifecycle callers to continue. Re-read + * SCM state and only accept the two states that cannot still own the proxy listener. + */ +export function stopWinswService(): void { + try { runWinsw(["stopwait"]); } catch { /* classify by verified state below */ } + const status = statusWinswRaw(); + if (status === "stopped" || status === "nonexistent") return; + if (status === "unknown") { + throw new Error("Native service stop could not be verified."); + } + throw new Error("Native service is still running after stop."); +} + export function uninstallWinswService(): void { if (!existsSync(winswExePath())) { // The binary is gone but the SCM registration can outlive it (quarantine, partial @@ -378,4 +394,4 @@ export function winswStatusSummary(): string { export function defaultWinswEntry(cliDir: string): WinswEntry { const runtime = durableBunRuntime(); return { bun: runtime.path, bunRuntimeSource: runtime.source, cli: join(cliDir, "cli", "index.ts") }; -} +} \ No newline at end of file diff --git a/src/server/index.ts b/src/server/index.ts index 07a900e1b7..65cb50c1cf 100644 --- a/src/server/index.ts +++ b/src/server/index.ts @@ -186,6 +186,7 @@ import { createLocalAttestationProof, createLocalAttestationSecret, } from "../lib/local-management-attestation"; +import { SYSTEM_RESTART_CAPABILITY_VERSION } from "../lib/system-restart-contract"; import { createReadinessGate, type ReadinessGate } from "./readiness"; const MAX_WS_FRAME_BYTES = 50 * 1024 * 1024; @@ -702,7 +703,15 @@ export function startServer(port?: number, deps: StartServerDeps = {}): Server((resolve, reject) => { let child: ReturnType; try { + const env: NodeJS.ProcessEnv = { ...process.env }; + delete env.OCX_SERVICE; child = spawn(process.execPath, args, { detached: true, stdio: "ignore", windowsHide: true, - env: withProcessRuntimeProvenance({ ...process.env, OCX_SERVICE: "1" }), + env: withProcessRuntimeProvenance(env), }); } catch (err) { reject(err); @@ -411,8 +413,8 @@ export function acceptSystemRestart(io: SystemRestartIo = restartIo): { `⚠️ Drain-and-restart spawn failed (${spawnFailureCode(err)}); exiting without replacement`, ); // Listen socket is already stopped; do not markRecycling — no child to inherit fences. - // ensure/tray children inherit OCX_SERVICE=1 without an installed service; clear it so - // syncCleanup can restore Codex/Grok fences instead of leaving clients pointed at a dead port. + // No replacement inherited the routing. Clear a stale service marker so + // this unsupervised parent restores clients after the failed handoff. delete process.env.OCX_SERVICE; exitProcess(1); return; diff --git a/src/server/management/system-routes.ts b/src/server/management/system-routes.ts index 6defbbc5ae..98fb922e56 100644 --- a/src/server/management/system-routes.ts +++ b/src/server/management/system-routes.ts @@ -27,6 +27,10 @@ import { getActiveTurnCount, isDraining } from "../lifecycle"; import { getActiveMemoryWatchdog, observedMemoryCounter } from "../memory-watchdog"; import { responseStateMetrics } from "../../responses/state"; import { appOwnedBytesSnapshot } from "../../lib/app-owned-memory"; +import { + SYSTEM_RESTART_EXPECTED_PID_HEADER, + parseExpectedSystemRestartPid, +} from "../../lib/system-restart-contract"; import { jsonResponse } from "../auth-cors"; import { getInspectionCounters } from "../relay"; import type { ManagementContext } from "./context"; @@ -104,6 +108,22 @@ export async function handleSystemRoutes(ctx: ManagementContext): Promise string; uninstall: () => void; }; +type ServiceInstallCleanupOps = { + status: () => string | null; + stop: () => void; +}; + function platformOps(backend: ServiceBackend = "scheduler"): ServiceOps | null { if (process.platform === "darwin") return { install: installLaunchd, start: startLaunchd, stop: stopLaunchd, status: statusLaunchd, uninstall: uninstallLaunchd }; @@ -2202,6 +2207,67 @@ function platformOps(backend: ServiceBackend = "scheduler"): ServiceOps | null { return null; } +/** + * Install-only manager operations. Unlike the ordinary status/stop helpers, these + * distinguish confirmed absence from a failed manager query and propagate every + * non-benign stop failure. Installing new assets is unsafe while either answer is + * unknown because an old manager may still respawn a listener on the target port. + */ +function platformServiceInstallCleanupOps(backend: ServiceBackend): ServiceInstallCleanupOps | null { + if (process.platform === "darwin") { + return { + status: () => { + const listing = sh("launchctl list"); + return listing.split("\n").some(line => line.includes(LABEL)) ? listing : null; + }, + stop: () => { sh(`launchctl unload "${plistPath()}"`); }, + }; + } + if (process.platform === "win32") { + if (backend === "native") { + return { + status: () => { + const status = statusWinswRaw(); + if (status === "unknown") throw new Error("Native service status could not be verified."); + return status === "nonexistent" ? null : status; + }, + stop: stopWinswService, + }; + } + return { + status: () => { + const probe = probeWindowsSchedulerTask(TASK); + if (probe.status === "unknown") throw new Error(`Task Scheduler status could not be verified: ${probe.detail}`); + return probe.status === "present" ? "present" : null; + }, + stop: () => { + try { + schtasks(["/end", "/tn", TASK]); + } catch (error) { + if (!isWindowsSchedulerEndBenign(error)) throw error; + } + }, + }; + } + if (process.platform === "linux") { + return { + status: () => { + // `list-unit-files ` exits non-zero when the unit has never been + // installed, which made a clean first install look like an unknown manager + // failure. `show LoadState` gives us the tri-state we actually need: a + // healthy user manager returns `not-found` for a missing unit, while an + // unreachable/permission-denied manager still makes `sh()` throw and the + // caller therefore fails closed. + const loadState = sh(`systemctl --user show ${TASK} --property=LoadState --value`).trim().toLowerCase(); + if (!loadState) throw new Error("systemd service status could not be verified."); + return loadState === "not-found" ? null : loadState; + }, + stop: () => { sh(`systemctl --user stop ${TASK}`); }, + }; + } + return null; +} + type TrackedProxyCleanupResult = "none" | "stale" | "stopped"; function verifiedKillTarget(pid: number | null | undefined): number | null { @@ -2297,6 +2363,60 @@ async function stopTrackedProxyForServiceCommand(): Promise ServiceDiagnostic; + managerOps?: (backend: ServiceBackend) => ServiceInstallCleanupOps | null; + stopTrackedProxy?: () => Promise; + platform?: NodeJS.Platform; +} + +/** + * Stop every manager that could own the install port, then stop the tracked + * standalone listener. Any unknown status or cleanup failure rejects, so callers + * cannot write assets or report success over a surviving old listener. + */ +export async function prepareServiceInstall( + requestedBackend: ServiceBackend, + deps: ServiceInstallPreparationDeps = {}, +): Promise { + const diagnostic = (deps.diagnose ?? diagnoseService)(); + const platform = deps.platform ?? process.platform; + const resolveOps = deps.managerOps ?? platformServiceInstallCleanupOps; + const backends: ServiceBackend[] = []; + const addBackend = (backend: ServiceBackend) => { + if (!backends.includes(backend)) backends.push(backend); + }; + + if (platform === "win32") { + // The recorded backend owns the old installation and must be stopped first. + // A conflicting diagnostic means both managers exist, so stop both even when + // the requested backend happens to match the recorded one. + if (diagnostic.backend === "scheduler" || diagnostic.backend === "native") { + addBackend(diagnostic.backend); + if (diagnostic.conflict) addBackend(diagnostic.backend === "scheduler" ? "native" : "scheduler"); + } + addBackend(requestedBackend); + } else { + addBackend(requestedBackend); + } + + for (const backend of backends) { + const manager = resolveOps(backend); + if (!manager) throw new Error(`Background service manager is unavailable for ${backend}.`); + if (manager.status() !== null) manager.stop(); + } + await (deps.stopTrackedProxy ?? stopTrackedProxyIfRunning)(); +} + +export async function installServiceSafely( + requestedBackend: ServiceBackend, + install: () => void | Promise, + deps: ServiceInstallPreparationDeps = {}, +): Promise { + await prepareServiceInstall(requestedBackend, deps); + await install(); +} + /** * If a service is installed, stop it so the process manager doesn't respawn after `ocx stop`. * Returns true if a service was found and stopped. @@ -2645,7 +2765,19 @@ export async function serviceCommand(...args: (string | undefined)[]): Promise { for (const relative of launchers) { const text = readFileSync(join(import.meta.dir, "..", relative), "utf8"); const spawnCount = (text.match(/spawn\(process\.execPath/g) ?? []).length; - const stampCount = (text.match(/env: withProcessRuntimeProvenance\(/g) ?? []).length; + const directStampCount = (text.match(/env: withProcessRuntimeProvenance\(/g) ?? []).length; + const detachedStartStampCount = relative === "src/cli/index.ts" + ? (text.match(/env: detachedStartEnvironment\(\)/g) ?? []).length + : 0; + if (detachedStartStampCount > 0) { + expect(text).toContain("return withProcessRuntimeProvenance(env)"); + } expect(spawnCount).toBeGreaterThan(0); - expect(stampCount).toBe(spawnCount); + expect(directStampCount + detachedStartStampCount).toBe(spawnCount); } }); }); diff --git a/tests/cli-models.test.ts b/tests/cli-models.test.ts index 8a91f08ce5..e40c0ed2a8 100644 --- a/tests/cli-models.test.ts +++ b/tests/cli-models.test.ts @@ -4,16 +4,21 @@ import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; +import { INTERNAL_DEADLINE_MS } from "./helpers/test-budget"; const repoRoot = dirname(fileURLToPath(new URL("../package.json", import.meta.url))); const cliPath = join(repoRoot, "src", "cli", "index.ts"); function runCli(args: string[], env: Record = {}) { - return spawnSync(process.execPath, [cliPath, ...args], { + const result = spawnSync(process.execPath, [cliPath, ...args], { cwd: repoRoot, env: { ...process.env, ...env }, encoding: "utf8", + timeout: INTERNAL_DEADLINE_MS, + killSignal: "SIGKILL", }); + if (result.error) throw result.error; + return result; } function freshConfig(extra?: Record) { diff --git a/tests/grok-lifecycle.test.ts b/tests/grok-lifecycle.test.ts index e693682319..9bf07ebb7b 100644 --- a/tests/grok-lifecycle.test.ts +++ b/tests/grok-lifecycle.test.ts @@ -43,26 +43,24 @@ describe("Grok fence lifecycle wiring", () => { test("handleStop gates shared teardown on ownership but still reverts system env", () => { const stopFn = sliceFn(CLI_SOURCE, "async function handleStop(", "async function handleUninstall("); + const restoreFn = sliceFn(CLI_SOURCE, "async function restoreSharedClientStateAfterStop(", "async function handleStop("); expect(stopFn).toContain("isServiceOwnershipError(err)"); expect(stopFn).toContain("ownershipBlocked = true"); - - const gateAt = stopFn.indexOf("if (!ownershipBlocked)"); - const stripAt = stopFn.indexOf("stripGrokConfig()"); - const restoreAt = stopFn.indexOf("restoreNativeCodexAsync()"); - const revertAt = stopFn.indexOf("revertSystemEnv()"); - - expect(gateAt).toBeGreaterThan(-1); - expect(stripAt).toBeGreaterThan(gateAt); - expect(restoreAt).toBeGreaterThan(gateAt); - // revertSystemEnv carries its own ownership check and concerns launchctl env, not - // CODEX_HOME — gating it too would be over-broad. - expect(stopFn.slice(revertAt - 200, revertAt)).toContain("NOT gated"); + expect(stopFn).toContain("if (!ownershipBlocked)"); + expect(stopFn).toContain("await restoreSharedClientStateAfterStop()"); + expect(restoreFn).toContain("restoreNativeCodexAsync()"); + expect(restoreFn).not.toContain("revertSystemEnv()"); + expect(restoreFn).toContain("stripGrokConfig()"); + expect(stopFn.indexOf("revertSystemEnv()")).toBeLessThan(stopFn.indexOf("if (!ownershipBlocked)")); }); test("a refused Grok strip makes ocx stop fail instead of reporting success", () => { + const restoreFn = sliceFn(CLI_SOURCE, "async function restoreSharedClientStateAfterStop(", "async function handleStop("); const stopFn = sliceFn(CLI_SOURCE, "async function handleStop(", "async function handleUninstall("); - expect(stopFn).toContain("else if (!g.ok) { stopFailed = true;"); + expect(restoreFn).toContain("else if (!grok.ok) { restored = false;"); + expect(restoreFn).toContain("Grok config restore failed"); + expect(stopFn).toContain("if (!await restoreSharedClientStateAfterStop()) stopFailed = true"); }); test("a refused proxy stop reports WHY, not just that it failed", () => { @@ -80,7 +78,7 @@ describe("Grok fence lifecycle wiring", () => { expect(stopFn.match(/if \(detail\) console\.error\(` \$\{detail\}`\);/g)).toHaveLength(2); }); - test("handleStop returns its outcome so restart and the tray can react", () => { + test("handleStop returns its outcome while both restart surfaces share the in-place lifecycle", () => { const stopFn = sliceFn(CLI_SOURCE, "async function handleStop(", "async function handleUninstall("); // process.exit() inside handleStop would strand runTrayProxyRestart's start() half. expect(stopFn).toContain("process.exitCode = 1"); @@ -88,20 +86,28 @@ describe("Grok fence lifecycle wiring", () => { expect(stopFn).not.toContain("process.exit(1)"); const restartCase = sliceFn(CLI_SOURCE, 'case "restart"', 'case "health"'); - expect(restartCase).toContain("if (await handleStop()) await handleEnsure()"); + expect(restartCase).toContain("await handleProxyRestart(handleRestartStartWhenStopped)"); + const trayRestart = sliceFn(CLI_SOURCE, "async function handleTrayProxyRestart(", "async function restoreSharedClientStateAfterStop("); + const restartHelper = sliceFn(CLI_SOURCE, "async function handleProxyRestart(", "async function handleTrayProxyRestart("); + expect(trayRestart).toContain("await handleProxyRestart(() => handleTrayProxyStart(false))"); + expect(restartHelper).toContain("requestBoundSystemRestart(previous, deadlineAt)"); }); test("handleStop treats an incomplete native Codex restore as a stop failure", () => { + const restoreFn = sliceFn(CLI_SOURCE, "async function restoreSharedClientStateAfterStop(", "async function handleStop("); const stopFn = sliceFn(CLI_SOURCE, "async function handleStop(", "async function handleUninstall("); - expect(stopFn).toContain("if (r.success) console.log"); - expect(stopFn).toContain("stopFailed = true"); - expect(stopFn).toContain("console.error(`⚠️ ${r.message}`)"); + expect(restoreFn).toContain("if (result.success) console.log"); + expect(restoreFn).toContain("restored = false"); + expect(restoreFn).toContain("console.error(`⚠️ ${result.message}`)"); + expect(stopFn).toContain("if (!await restoreSharedClientStateAfterStop()) stopFailed = true"); }); test("the daemon's exit cleanup keeps the OCX_SERVICE exclusion and adds the ownership check", () => { const startFn = sliceFn(CLI_SOURCE, "const syncCleanup = () => {", "let shuttingDown = false;"); // Crash/respawn under a service manager must still keep the fence. - expect(startFn).toContain("!process.env.OCX_SERVICE && serviceEnvironmentOwnedHere()"); + expect(startFn).toContain('process.env.OCX_SERVICE === "1"'); + expect(startFn).not.toContain("OCX_KEEP_ROUTING"); + expect(startFn).toContain("!preserveRouting && serviceEnvironmentOwnedHere()"); }); test("signal shutdown reports and exits nonzero when native Codex restore is incomplete", () => { diff --git a/tests/server-auth.test.ts b/tests/server-auth.test.ts index 849a5a204f..f67a1689e2 100644 --- a/tests/server-auth.test.ts +++ b/tests/server-auth.test.ts @@ -39,6 +39,7 @@ import type { OcxConfig } from "../src/types"; import { fakeChatGptJwt } from "./helpers/fake-chatgpt-jwt"; import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home"; import { configuredAdminToken } from "../src/lib/admin-secrets"; +import { SYSTEM_RESTART_CAPABILITY_VERSION } from "../src/lib/system-restart-contract"; const previousApiToken = process.env.OPENCODEX_API_AUTH_TOKEN; const previousOpencodexHome = process.env.OPENCODEX_HOME; @@ -614,7 +615,7 @@ describe("server local API auth", () => { } }); - test("/api/system/memory rides the management auth gate; /healthz shape unchanged (#314 WP3)", async () => { + test("/api/system/memory stays gated while /healthz exposes only bounded capability metadata (#314 WP3)", async () => { if (existsSync(TEST_DIR)) rmSync(TEST_DIR, { recursive: true }); mkdirSync(TEST_DIR, { recursive: true }); process.env.OPENCODEX_HOME = TEST_DIR; @@ -641,7 +642,16 @@ describe("server local API auth", () => { const health = await fetch(`http://127.0.0.1:${server.port}/healthz`); expect(health.status).toBe(200); const healthBody = await health.json() as Record; - expect(Object.keys(healthBody).sort()).toEqual(["pid", "port", "service", "status", "uptime", "version"]); + expect(Object.keys(healthBody).sort()).toEqual([ + "pid", + "port", + "restartCapability", + "service", + "status", + "uptime", + "version", + ]); + expect(healthBody.restartCapability).toBe(SYSTEM_RESTART_CAPABILITY_VERSION); expect("rss" in healthBody).toBe(false); } finally { await server.stop(true); diff --git a/tests/server-live.test.ts b/tests/server-live.test.ts index 00399a17f6..8197527fbf 100644 --- a/tests/server-live.test.ts +++ b/tests/server-live.test.ts @@ -857,8 +857,9 @@ test("sideband frame log records direction, kind, and U+FFFD context without ful }); // ── /readyz: per-server readiness gate ──────────────────────────────────────── -// /healthz stays byte-for-byte the same immediate liveness signal; /readyz is the -// stricter gate that reflects the post-startup Codex sync outcome. It is exact-GET +// /healthz remains the immediate liveness signal (with only bounded capability +// metadata); /readyz is the stricter gate that reflects the post-startup Codex sync +// outcome. It is exact-GET // and unauthenticated (like /healthz), returns a sanitized body, 503+Retry-After // while pending/failed, and 200 only when ready. Each startServer gets its own // PRIVATE gate via createReadinessGate(); starting/failing a second server in the diff --git a/tests/server-management-auth.test.ts b/tests/server-management-auth.test.ts index 80c3d28503..0d732c9b37 100644 --- a/tests/server-management-auth.test.ts +++ b/tests/server-management-auth.test.ts @@ -11,6 +11,7 @@ import { isProxyAdmissionSecret } from "../src/server/auth-cors"; import { initializeManagementAuthState, issueGuiSession, + managementPrincipal, removeManagementTokenPathBestEffort, requireManagementAuth, } from "../src/server/management-auth"; @@ -28,6 +29,15 @@ import { LOCAL_ATTESTATION_PROOF_HEADER, verifyLocalAttestationProof, } from "../src/lib/local-management-attestation"; +import { + SYSTEM_RESTART_CAPABILITY_HEADER, + SYSTEM_RESTART_EXPECTED_PID_HEADER, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_NONCE_HEADER, + SYSTEM_RESTART_PATH, + createSystemRestartCapability, +} from "../src/lib/system-restart-contract"; +import { setSystemRestartIoForTests } from "../src/server/management/system-restart"; const previousHome = process.env.OPENCODEX_HOME; const previousDataToken = process.env.OPENCODEX_API_AUTH_TOKEN; @@ -80,6 +90,7 @@ beforeEach(() => { }); afterEach(() => { + setSystemRestartIoForTests(); setIcaclsRunnerForTests(null); setPlatformForTests(null); resetHardenedStateForTests(); @@ -109,6 +120,95 @@ describe("management and data-plane credential separation", () => { } }); + test("a process-scoped capability authorizes only the exact restart operation", async () => { + const secret = "A".repeat(43); + const nonce = "B".repeat(43); + let scheduled = 0; + // The capability contract is platform-independent. Avoid making this HTTP + // integration assertion depend on the host's live icacls policy; dedicated + // Windows ACL tests cover that boundary separately. + setPlatformForTests("linux"); + setSystemRestartIoForTests({ + isDraining: () => false, + schedule: () => { scheduled += 1; }, + setDraining: () => {}, + }); + const unavailable = { available: false, reason: "injected unavailable state" } as const; + const server = startServer(0, { + localAttestationSecret: secret, + managementAuthState: unavailable, + }); + try { + const capability = createSystemRestartCapability( + secret, + nonce, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_PATH, + process.pid, + server.port, + ); + const headers = { + [SYSTEM_RESTART_EXPECTED_PID_HEADER]: String(process.pid), + [SYSTEM_RESTART_NONCE_HEADER]: nonce, + [SYSTEM_RESTART_CAPABILITY_HEADER]: capability!, + }; + + const restart = await fetch(new URL(SYSTEM_RESTART_PATH, server.url), { + method: SYSTEM_RESTART_METHOD, + headers, + }); + expect(restart.status).toBe(202); + expect(scheduled).toBe(1); + + const foreignRoute = await fetch(new URL("/api/config", server.url), { + method: "POST", + headers, + }); + expect(foreignRoute.status).toBe(503); + + const tampered = await fetch(new URL(SYSTEM_RESTART_PATH, server.url), { + method: SYSTEM_RESTART_METHOD, + headers: { ...headers, [SYSTEM_RESTART_CAPABILITY_HEADER]: "C".repeat(43) }, + }); + expect(tampered.status).toBe(503); + + const wrongMethod = await fetch(new URL(SYSTEM_RESTART_PATH, server.url), { + method: "DELETE", + headers, + }); + expect(wrongMethod.status).toBe(503); + + const wrongPortCapability = createSystemRestartCapability( + secret, + nonce, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_PATH, + process.pid, + server.port + 1, + ); + const wrongPort = await fetch(new URL(SYSTEM_RESTART_PATH, server.url), { + method: SYSTEM_RESTART_METHOD, + headers: { + ...headers, + [SYSTEM_RESTART_CAPABILITY_HEADER]: wrongPortCapability!, + }, + }); + expect(wrongPort.status).toBe(503); + expect(scheduled).toBe(1); + + const request = new Request(new URL(SYSTEM_RESTART_PATH, server.url), { + method: SYSTEM_RESTART_METHOD, + headers, + }); + const local = { attestationSecret: secret, pid: process.pid, port: server.port }; + expect(requireManagementAuth(request, unavailable, remoteConfig(), local)).toBeNull(); + expect(managementPrincipal(request, unavailable, remoteConfig(), local)) + .toBe("system-restart-capability"); + } finally { + await server.stop(true); + } + }); + test("management-token temp cleanup forgets successful ACL memos and retains failed removals", () => { const temporary = join(testHome, ".admin-token.tmp"); const previousUsername = process.env.USERNAME; diff --git a/tests/service.test.ts b/tests/service.test.ts index 986756ad40..a60ec8f7bd 100644 --- a/tests/service.test.ts +++ b/tests/service.test.ts @@ -3,7 +3,7 @@ import { existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; import { join } from "node:path"; import { saveConfig } from "../src/config"; import { windowsEnvIndirectBatchValue } from "../src/lib/win-paths"; -import { assertServiceAuthEnvironment, assertServiceEnvironmentMatchesInstall, bakedServicePathsDiagnostic, confirmServiceServing, launchdListenPort, systemdListenPort, buildPlist, buildUnit, buildWindowsLauncherVbs, buildWindowsSchtasksCreateArgs, buildWindowsServiceScript, buildWindowsTaskXml, deriveWindowsServiceDiagnostic, launchctlLoadFailed, launchdJobMatchesPlist, normalizeServiceSubcommand, parseServiceInstallState, readWindowsSchedulerXmlState, repairService, resolveServiceListenPort, runLaunchctl, serviceLogPath, serviceStartableFromTray, serviceStatusReport, serviceRetryCommand, serviceStatusSummary, systemdNeedsDaemonReload, windowsListenPort, winswListenPort, startLaunchd, windowsTaskRegistrationHealthy } from "../src/service"; +import { assertServiceAuthEnvironment, assertServiceEnvironmentMatchesInstall, bakedServicePathsDiagnostic, confirmServiceServing, launchdListenPort, systemdListenPort, buildPlist, buildUnit, buildWindowsLauncherVbs, buildWindowsSchtasksCreateArgs, buildWindowsServiceScript, buildWindowsTaskXml, deriveWindowsServiceDiagnostic, installServiceSafely, launchctlLoadFailed, launchdJobMatchesPlist, normalizeServiceSubcommand, parseServiceInstallState, prepareServiceInstall, readWindowsSchedulerXmlState, repairService, resolveServiceListenPort, runLaunchctl, serviceLogPath, serviceStartableFromTray, serviceStatusReport, serviceRetryCommand, serviceStatusSummary, systemdNeedsDaemonReload, windowsListenPort, winswListenPort, startLaunchd, windowsTaskRegistrationHealthy } from "../src/service"; import type { ServiceDiagnostic } from "../src/service"; import { buildWinswXml } from "../src/lib/winsw"; import { serviceApiTokenFilePath } from "../src/lib/service-secrets"; @@ -647,6 +647,60 @@ describe("launchd service plist", () => { }); describe("service lifecycle cleanup ordering", () => { + test("service install stops the recorded backend, requested backend, and standalone before loading assets", async () => { + const calls: string[] = []; + const managerOps = (backend: "scheduler" | "native") => ({ + status: () => { calls.push(`status:${backend}`); return "present"; }, + stop: () => { calls.push(`stop:${backend}`); }, + }); + await installServiceSafely("native", () => { calls.push("install:native"); }, { + platform: "win32", + diagnose: () => ({ supported: true, installed: true, enabled: true, running: true, viable: true, startable: true, stale: false, conflict: false, backend: "scheduler", summary: "test" }), + managerOps, + stopTrackedProxy: async () => { calls.push("stop:standalone"); }, + }); + expect(calls).toEqual([ + "status:scheduler", "stop:scheduler", + "status:native", "stop:native", + "stop:standalone", "install:native", + ]); + }); + + test("service install fails closed before install on manager or standalone cleanup errors", async () => { + for (const failure of ["status", "stop", "standalone"] as const) { + let installed = false; + const run = installServiceSafely("scheduler", () => { installed = true; }, { + platform: "win32", + diagnose: () => ({ supported: true, installed: true, enabled: true, running: true, viable: true, startable: true, stale: false, conflict: false, backend: "scheduler", summary: "test" }), + managerOps: () => ({ + status: () => { + if (failure === "status") throw new Error("status failed"); + return "present"; + }, + stop: () => { + if (failure === "stop") throw new Error("stop failed"); + }, + }), + stopTrackedProxy: async () => { + if (failure === "standalone") throw new Error("standalone failed"); + }, + }); + await expect(run).rejects.toThrow(`${failure} failed`); + expect(installed).toBe(false); + } + }); + + test("conflicting Windows install preparation stops both managers", async () => { + const stopped: string[] = []; + await prepareServiceInstall("scheduler", { + platform: "win32", + diagnose: () => ({ supported: true, installed: true, enabled: true, running: true, viable: false, startable: false, stale: false, conflict: true, backend: "scheduler", summary: "test" }), + managerOps: backend => ({ status: () => "present", stop: () => { stopped.push(backend); } }), + stopTrackedProxy: async () => {}, + }); + expect(stopped).toEqual(["scheduler", "native"]); + }); + test("direct service stop kills the tracked proxy before restoring native Codex", async () => { const service = await readText("src/service.ts"); const stopCase = service.slice(service.indexOf('case "stop":'), service.indexOf('case "status":')); diff --git a/tests/startup-prompt.test.ts b/tests/startup-prompt.test.ts index 2091eb728c..4b61304f54 100644 --- a/tests/startup-prompt.test.ts +++ b/tests/startup-prompt.test.ts @@ -187,7 +187,7 @@ describe("startup star prompt", () => { test("ocx service install gets the prompt too, after the service is up", async () => { const service = await readText("src/service.ts"); - const installIndex = service.indexOf("await ops.install()"); + const installIndex = service.indexOf("await installServiceSafely(backend, ops.install)"); const promptIndex = service.indexOf("await maybeShowStarPrompt()"); expect(installIndex).toBeGreaterThan(-1); diff --git a/tests/system-restart-client.test.ts b/tests/system-restart-client.test.ts new file mode 100644 index 0000000000..3eb9c7e3b4 --- /dev/null +++ b/tests/system-restart-client.test.ts @@ -0,0 +1,255 @@ +import { describe, expect, test } from "bun:test"; +import { + LOCAL_ATTESTATION_CHALLENGE_HEADER, + LOCAL_ATTESTATION_PROOF_HEADER, + createLocalAttestationProof, + createLocalAttestationSecret, +} from "../src/lib/local-management-attestation"; +import { + SYSTEM_RESTART_CAPABILITY_HEADER, + SYSTEM_RESTART_CAPABILITY_VERSION, + SYSTEM_RESTART_EXPECTED_PID_HEADER, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_NONCE_HEADER, + SYSTEM_RESTART_PATH, + verifySystemRestartCapability, +} from "../src/lib/system-restart-contract"; +import { requestBoundSystemRestart } from "../src/cli/system-restart-client"; +import type { LiveProxy } from "../src/server/proxy-liveness"; + +const target: LiveProxy = { + pid: 4242, + port: 10100, + hostname: "127.0.0.1", + source: "runtime", +}; + +function successfulDeps() { + const secret = createLocalAttestationSecret(); + const challenge = "A".repeat(43); + const requests: Array<{ url: string; init?: RequestInit }> = []; + const fetchImpl = (async (input: string | URL | Request, init?: RequestInit) => { + const url = String(input); + requests.push({ url, init }); + if (url.endsWith("/healthz")) { + return successfulDepsResponse(secret, challenge); + } + return new Response(JSON.stringify({ success: true }), { status: 202 }); + }) as typeof fetch; + + return { + secret, + challenge, + requests, + deps: { + fetchImpl, + readRuntime: () => ({ + pid: target.pid!, + port: target.port, + hostname: target.hostname, + attestationSecret: secret, + }), + findLive: async () => target, + createChallenge: () => challenge, + now: () => 1_000, + }, + }; +} + +describe("bound system restart client", () => { + test("rejects an unattested discovery result without network access", async () => { + let fetches = 0; + const outcome = await requestBoundSystemRestart( + { pid: null, port: 10100, source: "config" }, + 10_000, + { fetchImpl: (async () => { fetches += 1; return new Response(); }) as typeof fetch }, + ); + expect(outcome.accepted).toBe(false); + expect(outcome.accepted ? null : outcome.uncertain).toBe(false); + expect(fetches).toBe(0); + }); + + test("rejects missing runtime proof state before fetching", async () => { + let fetches = 0; + const fetchImpl = (async () => { fetches += 1; return new Response(); }) as typeof fetch; + const noRuntime = await requestBoundSystemRestart(target, 10_000, { + fetchImpl, + readRuntime: () => null, + }); + expect(noRuntime.accepted).toBe(false); + expect(fetches).toBe(0); + }); + + test("rejects stale runtime PID or port state before fetching", async () => { + for (const runtime of [ + { ...target, pid: target.pid! + 1 }, + { ...target, port: target.port + 1 }, + ]) { + let fetches = 0; + const outcome = await requestBoundSystemRestart(target, 10_000, { + fetchImpl: (async () => { fetches += 1; return new Response(); }) as typeof fetch, + readRuntime: () => ({ + pid: runtime.pid!, + port: runtime.port, + hostname: runtime.hostname, + attestationSecret: createLocalAttestationSecret(), + }), + }); + expect(outcome.accepted).toBe(false); + expect(fetches).toBe(0); + } + }); + + test("does not send the admin token when attestation fails", async () => { + const setup = successfulDeps(); + setup.deps.fetchImpl = (async (input: string | URL | Request, init?: RequestInit) => { + setup.requests.push({ url: String(input), init }); + return new Response(JSON.stringify({ + status: "ok", + service: "opencodex", + version: "test", + uptime: 1, + pid: target.pid, + }), { status: 200, headers: { "content-type": "application/json" } }); + }) as typeof fetch; + + const outcome = await requestBoundSystemRestart(target, 10_000, setup.deps); + expect(outcome.accepted).toBe(false); + expect(setup.requests).toHaveLength(1); + const headers = new Headers(setup.requests[0]!.init?.headers); + expect(headers.has("X-OpenCodex-API-Key")).toBe(false); + }); + + test("refuses a pre-update proxy before POST instead of weakening PID-bound auth", async () => { + const setup = successfulDeps(); + setup.deps.fetchImpl = (async (input: string | URL | Request, init?: RequestInit) => { + const url = String(input); + setup.requests.push({ url, init }); + if (url.endsWith("/healthz")) { + const response = successfulDepsResponse(setup.secret, setup.challenge); + const body = await response.json() as Record; + delete body.restartCapability; + return new Response(JSON.stringify(body), { + status: 200, + headers: response.headers, + }); + } + throw new Error("POST must not be attempted"); + }) as typeof fetch; + + const outcome = await requestBoundSystemRestart(target, 10_000, setup.deps); + expect(outcome).toMatchObject({ accepted: false, uncertain: false }); + expect(outcome.accepted ? "" : (outcome.error as Error).message) + .toBe("restart_capability_unsupported"); + expect(setup.requests).toHaveLength(1); + }); + + test("refuses to POST when the live target changes after attestation", async () => { + const setup = successfulDeps(); + setup.deps.findLive = async () => ({ ...target, pid: 4343 }); + const outcome = await requestBoundSystemRestart(target, 10_000, setup.deps); + expect(outcome.accepted).toBe(false); + expect(setup.requests).toHaveLength(1); + }); + + test("posts once with a process-scoped capability and no reusable admin token", async () => { + const setup = successfulDeps(); + const outcome = await requestBoundSystemRestart(target, 10_000, setup.deps); + expect(outcome).toEqual({ accepted: true }); + expect(setup.requests).toHaveLength(2); + + const proofHeaders = new Headers(setup.requests[0]!.init?.headers); + expect(proofHeaders.get(LOCAL_ATTESTATION_CHALLENGE_HEADER)).toBe(setup.challenge); + expect(proofHeaders.has("X-OpenCodex-API-Key")).toBe(false); + + const restart = setup.requests[1]!; + expect(restart.url).toBe("http://127.0.0.1:10100/api/system/restart"); + expect(restart.init?.method).toBe("POST"); + const restartHeaders = new Headers(restart.init?.headers); + expect(restartHeaders.has("X-OpenCodex-API-Key")).toBe(false); + expect(restartHeaders.get(SYSTEM_RESTART_EXPECTED_PID_HEADER)).toBe(String(target.pid)); + expect(restartHeaders.get(SYSTEM_RESTART_NONCE_HEADER)).toBe(setup.challenge); + expect(verifySystemRestartCapability( + setup.secret, + restartHeaders.get(SYSTEM_RESTART_NONCE_HEADER), + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_PATH, + target.pid!, + target.port, + restartHeaders.get(SYSTEM_RESTART_CAPABILITY_HEADER), + )).toBe(true); + }); + + test("forwards the absolute deadline into the final target recheck", async () => { + const setup = successfulDeps(); + let observedDeadline: number | undefined; + setup.deps.findLive = async io => { + observedDeadline = io.deadlineAt; + return target; + }; + expect(await requestBoundSystemRestart(target, 10_000, setup.deps)).toEqual({ accepted: true }); + expect(observedDeadline).toBe(10_000); + }); + + test("treats an HTTP rejection as definite and a POST transport loss as uncertain", async () => { + const rejected = successfulDeps(); + rejected.deps.fetchImpl = (async (input: string | URL | Request, init?: RequestInit) => { + const url = String(input); + if (url.endsWith("/healthz")) return successfulDepsResponse(rejected.secret, rejected.challenge); + rejected.requests.push({ url, init }); + return new Response("stale", { status: 409 }); + }) as typeof fetch; + const rejectedOutcome = await requestBoundSystemRestart(target, 10_000, rejected.deps); + expect(rejectedOutcome.accepted).toBe(false); + expect(rejectedOutcome.accepted ? null : rejectedOutcome.uncertain).toBe(false); + + const uncertain = successfulDeps(); + uncertain.deps.fetchImpl = (async (input: string | URL | Request) => { + const url = String(input); + if (url.endsWith("/healthz")) return successfulDepsResponse(uncertain.secret, uncertain.challenge); + throw new Error("connection closed"); + }) as typeof fetch; + const uncertainOutcome = await requestBoundSystemRestart(target, 10_000, uncertain.deps); + expect(uncertainOutcome.accepted).toBe(false); + expect(uncertainOutcome.accepted ? null : uncertainOutcome.uncertain).toBe(true); + }); + + test("treats an unreachable attestation probe as definite and never posts", async () => { + const setup = successfulDeps(); + setup.deps.fetchImpl = (async (input: string | URL | Request) => { + if (String(input).endsWith("/healthz")) throw new Error("connection refused"); + throw new Error("POST must not be attempted"); + }) as typeof fetch; + const outcome = await requestBoundSystemRestart(target, 10_000, setup.deps); + expect(outcome).toMatchObject({ accepted: false, uncertain: false }); + expect(outcome.accepted ? "" : (outcome.error as Error).message) + .toBe("restart_attestation_unreachable"); + }); + + test("enforces the absolute deadline before any fetch", async () => { + const setup = successfulDeps(); + setup.deps.now = () => 10_001; + const outcome = await requestBoundSystemRestart(target, 10_000, setup.deps); + expect(outcome.accepted).toBe(false); + expect(setup.requests).toHaveLength(0); + }); +}); + +function successfulDepsResponse(secret: string, challenge: string): Response { + const proof = createLocalAttestationProof(secret, challenge, target.pid!, target.port); + return new Response(JSON.stringify({ + status: "ok", + service: "opencodex", + version: "test", + uptime: 1, + pid: target.pid, + port: target.port, + restartCapability: SYSTEM_RESTART_CAPABILITY_VERSION, + }), { + status: 200, + headers: { + "content-type": "application/json", + [LOCAL_ATTESTATION_PROOF_HEADER]: proof!, + }, + }); +} diff --git a/tests/system-restart-contract-security.test.ts b/tests/system-restart-contract-security.test.ts new file mode 100644 index 0000000000..337d313c34 --- /dev/null +++ b/tests/system-restart-contract-security.test.ts @@ -0,0 +1,83 @@ +import { describe, expect, test } from "bun:test"; +import { + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_PATH, + createSystemRestartCapability, + verifySystemRestartCapability, +} from "../src/lib/system-restart-contract"; + +const SECRET = "A".repeat(43); +const NONCE = "B".repeat(43); +const PID = 4242; +const PORT = 10100; + +function capability(): string { + const value = createSystemRestartCapability( + SECRET, + NONCE, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_PATH, + PID, + PORT, + ); + if (!value) throw new Error("test capability could not be created"); + return value; +} + +describe("system restart capability security boundary", () => { + test("rejects a capability when the expected runtime pid is stale", () => { + expect(verifySystemRestartCapability( + SECRET, + NONCE, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_PATH, + PID + 1, + PORT, + capability(), + )).toBe(false); + }); + + test("rejects a capability when the nonce is mutated", () => { + const mutatedNonce = `${NONCE.slice(0, -1)}C`; + expect(verifySystemRestartCapability( + SECRET, + mutatedNonce, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_PATH, + PID, + PORT, + capability(), + )).toBe(false); + }); + + test("rejects cross-operation and cross-listener reuse", () => { + const signed = capability(); + expect(verifySystemRestartCapability( + SECRET, + NONCE, + "DELETE", + SYSTEM_RESTART_PATH, + PID, + PORT, + signed, + )).toBe(false); + expect(verifySystemRestartCapability( + SECRET, + NONCE, + SYSTEM_RESTART_METHOD, + "/api/config", + PID, + PORT, + signed, + )).toBe(false); + expect(verifySystemRestartCapability( + SECRET, + NONCE, + SYSTEM_RESTART_METHOD, + SYSTEM_RESTART_PATH, + PID, + PORT + 1, + signed, + )).toBe(false); + }); +}); diff --git a/tests/system-restart.test.ts b/tests/system-restart.test.ts index 836ade8323..1b43ca2592 100644 --- a/tests/system-restart.test.ts +++ b/tests/system-restart.test.ts @@ -12,6 +12,7 @@ import { setSystemRestartIoForTests, waitForReplacementReady, } from "../src/server/management/system-restart"; +import { SYSTEM_RESTART_EXPECTED_PID_HEADER } from "../src/lib/system-restart-contract"; import type { OcxConfig } from "../src/types"; function config(): OcxConfig { @@ -632,7 +633,7 @@ describe("acceptSystemRestart", () => { acceptSystemRestart({ isDraining: () => false, getActiveTurnCount: () => 0, - // ensure/tray: marker set, but no installed service → unsupervised spawn path + // A stale service marker without a viable service must use the unsupervised spawn path. isSupervisedServiceChild: () => false, listenPort: () => 10123, schedule: (fn) => { scheduled = fn; }, @@ -784,5 +785,50 @@ describe("POST /api/system/restart", () => { expect(body.alreadyDraining).toBe(false); expect(body.message.toLowerCase()).toContain("drain"); }); + + test("rejects an invalid expected PID before scheduling restart", async () => { + let scheduled = 0; + setSystemRestartIoForTests({ + schedule: () => { scheduled += 1; }, + }); + const req = new Request("http://127.0.0.1:10100/api/system/restart", { + method: "POST", + headers: { [SYSTEM_RESTART_EXPECTED_PID_HEADER]: "not-a-pid" }, + }); + const res = await handleManagementAPI(req, new URL(req.url), config()); + expect(res?.status).toBe(400); + expect(scheduled).toBe(0); + }); + + test("rejects a stale expected PID before scheduling restart", async () => { + let scheduled = 0; + setSystemRestartIoForTests({ + schedule: () => { scheduled += 1; }, + }); + const stalePid = process.pid === 1 ? 2 : 1; + const req = new Request("http://127.0.0.1:10100/api/system/restart", { + method: "POST", + headers: { [SYSTEM_RESTART_EXPECTED_PID_HEADER]: String(stalePid) }, + }); + const res = await handleManagementAPI(req, new URL(req.url), config()); + expect(res?.status).toBe(409); + expect(scheduled).toBe(0); + }); + + test("accepts a matching expected PID", async () => { + let scheduled = 0; + setSystemRestartIoForTests({ + isDraining: () => false, + schedule: () => { scheduled += 1; }, + setDraining: () => {}, + }); + const req = new Request("http://127.0.0.1:10100/api/system/restart", { + method: "POST", + headers: { [SYSTEM_RESTART_EXPECTED_PID_HEADER]: String(process.pid) }, + }); + const res = await handleManagementAPI(req, new URL(req.url), config()); + expect(res?.status).toBe(202); + expect(scheduled).toBe(1); + }); }); import { ManagementRequest as Request } from "./helpers/management-auth"; diff --git a/tests/systemd-install-cleanup-hardening.test.ts b/tests/systemd-install-cleanup-hardening.test.ts new file mode 100644 index 0000000000..a3c5d288e2 --- /dev/null +++ b/tests/systemd-install-cleanup-hardening.test.ts @@ -0,0 +1,13 @@ +import { describe, expect, test } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +const source = readFileSync(join(import.meta.dir, "../src/service.ts"), "utf8"); + +describe("systemd install cleanup status hardening", () => { + test("only treats literal not-found as confirmed unit absence", () => { + expect(source).toContain('if (!loadState) throw new Error("systemd service status could not be verified.");'); + expect(source).toContain('return loadState === "not-found" ? null : loadState;'); + expect(source).not.toContain('return !loadState || loadState === "not-found" ? null : loadState;'); + }); +}); diff --git a/tests/tray-proxy-deadline.test.ts b/tests/tray-proxy-deadline.test.ts new file mode 100644 index 0000000000..f62f60e648 --- /dev/null +++ b/tests/tray-proxy-deadline.test.ts @@ -0,0 +1,34 @@ +import { describe, expect, test } from "bun:test"; +import { discoverStableProxyForRestart } from "../src/cli/tray-proxy"; + +describe("restart discovery deadline", () => { + test("fails closed when the deadline expires after the first absence observation", async () => { + let calls = 0; + let expiryChecks = 0; + const result = await discoverStableProxyForRestart({ + findLive: async () => { calls += 1; return null; }, + waitBetweenChecks: async () => { throw new Error("must not wait after expiry"); }, + expired: () => { expiryChecks += 1; return true; }, + }); + expect(result.status).toBe("uncertain"); + expect(result.status === "uncertain" ? result.error.message : "") + .toBe("restart_discovery_deadline_expired"); + expect(calls).toBe(1); + expect(expiryChecks).toBe(1); + }); + + test("fails closed when the deadline expires after the second absence observation", async () => { + let calls = 0; + let expiryChecks = 0; + const result = await discoverStableProxyForRestart({ + findLive: async () => { calls += 1; return null; }, + waitBetweenChecks: async () => {}, + expired: () => { expiryChecks += 1; return expiryChecks === 2; }, + }); + expect(result.status).toBe("uncertain"); + expect(result.status === "uncertain" ? result.error.message : "") + .toBe("restart_discovery_deadline_expired"); + expect(calls).toBe(2); + expect(expiryChecks).toBe(2); + }); +}); diff --git a/tests/tray-proxy.test.ts b/tests/tray-proxy.test.ts index 1ecf0130d7..e5f0d45167 100644 --- a/tests/tray-proxy.test.ts +++ b/tests/tray-proxy.test.ts @@ -1,5 +1,13 @@ import { describe, expect, test } from "bun:test"; -import { runTrayProxyRestart, runTrayProxyStart, type TrayProxyStartIo } from "../src/cli/tray-proxy"; +import { + discoverStableProxyForRestart, + isProxyReplacement, + runProxyRestart, + runTrayProxyStart, + type ProxyRestartIo, + type ProxyRestartLive, + type TrayProxyStartIo, +} from "../src/cli/tray-proxy"; function startIo(overrides: Partial = {}) { const calls: string[] = []; @@ -23,6 +31,17 @@ describe("tray proxy coordinator", () => { expect(calls).toEqual(["info:Proxy already running on port 20200."]); }); + test("restart fallback refuses a target that reappears during the final start check", async () => { + const { io, calls } = startIo({ + findLive: async () => ({ port: 20200 }), + existingIsSuccess: false, + }); + expect(await runTrayProxyStart(io)).toBe(false); + expect(calls.some(call => call.startsWith("error:Proxy appeared"))).toBe(true); + expect(calls).not.toContain("direct"); + expect(calls).not.toContain("service"); + }); + test("refuses an installed but unviable service instead of bypassing it", async () => { const { io, calls } = startIo({ diagnoseService: () => ({ installed: true, startable: false, summary: "stale" }), @@ -71,18 +90,201 @@ describe("tray proxy coordinator", () => { expect(direct.calls).not.toContain("service"); }); - test("restart never starts after a failed stop", async () => { + test("restart degrades to the normal start path only when no proxy is live", async () => { + const calls: string[] = []; + const io: ProxyRestartIo = { + findLive: async () => ({ status: "absent" }), + startWhenStopped: async () => { calls.push("start"); return true; }, + requestInPlaceRestart: async () => { calls.push("request"); return { accepted: true }; }, + waitForReplacement: async () => { calls.push("wait"); return null; }, + }; + expect(await runProxyRestart(io)).toEqual({ ok: true, mode: "started" }); + expect(calls).toEqual(["start"]); + + calls.length = 0; + io.startWhenStopped = async () => { calls.push("start"); return false; }; + expect(await runProxyRestart(io)).toEqual({ ok: false, phase: "start" }); + expect(calls).toEqual(["start"]); + + calls.length = 0; + io.startWhenStopped = async () => { calls.push("skip"); return "skipped"; }; + expect(await runProxyRestart(io)).toEqual({ ok: true, mode: "skipped" }); + expect(calls).toEqual(["skip"]); + + calls.length = 0; + const error = new Error("spawn failed"); + io.startWhenStopped = async () => { calls.push("start"); throw error; }; + expect(await runProxyRestart(io)).toEqual({ ok: false, phase: "start", error }); + expect(calls).toEqual(["start"]); + }); + + test("a live proxy owns one in-place restart and must publish a replacement identity", async () => { const calls: string[] = []; - expect(await runTrayProxyRestart({ - stop: async () => { calls.push("stop"); return false; }, - start: async () => { calls.push("start"); return true; }, - })).toBe(false); - expect(calls).toEqual(["stop"]); - - expect(await runTrayProxyRestart({ - stop: async () => true, - start: async () => { calls.push("start-after-stop"); return true; }, - })).toBe(true); - expect(calls).toContain("start-after-stop"); + const previous: ProxyRestartLive = { pid: 10, port: 10100, source: "runtime" }; + const replacement: ProxyRestartLive = { pid: 20, port: 10100, source: "runtime" }; + const result = await runProxyRestart({ + findLive: async () => ({ status: "live", live: previous }), + startWhenStopped: async () => { calls.push("fallback-start"); return true; }, + requestInPlaceRestart: async observed => { + calls.push(`request:${observed.pid}`); + return { accepted: true }; + }, + waitForReplacement: async observed => { + calls.push(`wait:${observed.pid}`); + return replacement; + }, + }); + expect(result).toEqual({ ok: true, mode: "restarted", live: replacement }); + expect(calls).toEqual(["request:10", "wait:10"]); + }); + + test("request uncertainty observes for a replacement and never falls back to stop/start", async () => { + const calls: string[] = []; + const error = new Error("response connection closed"); + const result = await runProxyRestart({ + findLive: async () => ({ + status: "live", + live: { pid: 10, port: 10100, source: "runtime" }, + }), + startWhenStopped: async () => { calls.push("fallback-start"); return true; }, + requestInPlaceRestart: async () => { + calls.push("request"); + return { accepted: false, uncertain: true, error }; + }, + waitForReplacement: async () => { calls.push("wait"); return null; }, + }); + expect(result).toEqual({ ok: false, phase: "request", error }); + expect(calls).toEqual(["request", "wait"]); + }); + + test("a replacement proves success even when the request response was lost", async () => { + const error = new Error("response connection closed"); + const replacement: ProxyRestartLive = { pid: 20, port: 10100, source: "runtime" }; + const result = await runProxyRestart({ + findLive: async () => ({ + status: "live", + live: { pid: 10, port: 10100, source: "runtime" }, + }), + startWhenStopped: async () => true, + requestInPlaceRestart: async () => ({ accepted: false, uncertain: true, error }), + waitForReplacement: async () => replacement, + }); + expect(result).toEqual({ ok: true, mode: "restarted", live: replacement }); + }); + + test("a definite request rejection does not wait or start another proxy", async () => { + const calls: string[] = []; + const error = new Error("target changed"); + const result = await runProxyRestart({ + findLive: async () => ({ + status: "live", + live: { pid: 10, port: 10100, source: "runtime" }, + }), + startWhenStopped: async () => { calls.push("fallback-start"); return true; }, + requestInPlaceRestart: async () => { + calls.push("request"); + return { accepted: false, uncertain: false, error }; + }, + waitForReplacement: async () => { calls.push("wait"); return null; }, + }); + expect(result).toEqual({ ok: false, phase: "request", error }); + expect(calls).toEqual(["request"]); + }); + + test("an accepted restart that never publishes a replacement fails closed", async () => { + const calls: string[] = []; + const result = await runProxyRestart({ + findLive: async () => ({ + status: "live", + live: { pid: 10, port: 10100, source: "runtime" }, + }), + startWhenStopped: async () => { calls.push("fallback-start"); return true; }, + requestInPlaceRestart: async () => { calls.push("request"); return { accepted: true }; }, + waitForReplacement: async () => { calls.push("wait"); return null; }, + }); + expect(result).toEqual({ ok: false, phase: "replacement" }); + expect(calls).toEqual(["request", "wait"]); + }); + + test("an unverified live target fails closed before the restart request", async () => { + const calls: string[] = []; + const result = await runProxyRestart({ + findLive: async () => ({ + status: "live", + live: { pid: null, port: 10100, source: "config" }, + }), + startWhenStopped: async () => { calls.push("fallback-start"); return true; }, + requestInPlaceRestart: async () => { calls.push("request"); return { accepted: true }; }, + waitForReplacement: async () => { calls.push("wait"); return null; }, + }); + expect(result).toEqual({ ok: false, phase: "identity" }); + expect(calls).toEqual([]); + }); + + test("replacement identity requires a new runtime PID on the same port", () => { + const previous: ProxyRestartLive = { pid: 10, port: 10100, source: "runtime" }; + expect(isProxyReplacement(previous, null)).toBe(false); + expect(isProxyReplacement(previous, { pid: null, port: 10100, source: "runtime" })).toBe(false); + expect(isProxyReplacement(previous, { pid: 10, port: 10100, source: "runtime" })).toBe(false); + expect(isProxyReplacement(previous, { pid: 20, port: 20200, source: "runtime" })).toBe(false); + expect(isProxyReplacement(previous, { pid: 20, port: 10100, source: "config" })).toBe(false); + expect(isProxyReplacement(previous, { pid: 20, port: 10100, source: "runtime" })).toBe(true); + }); + + test("stable absence requires two empty observations and rejects a reappearing target", async () => { + let calls = 0; + const absent = await discoverStableProxyForRestart({ + findLive: async () => { calls += 1; return null; }, + waitBetweenChecks: async () => {}, + }); + expect(absent).toEqual({ status: "absent" }); + expect(calls).toBe(2); + + calls = 0; + const appeared = await discoverStableProxyForRestart({ + findLive: async () => { + calls += 1; + return calls === 1 ? null : { pid: 20, port: 10100, source: "runtime" }; + }, + waitBetweenChecks: async () => {}, + }); + expect(appeared.status).toBe("uncertain"); + expect(calls).toBe(2); + }); + + test("uncertain discovery never starts, requests, or reports a false restart", async () => { + const calls: string[] = []; + const error = new Error("probe timed out"); + const result = await runProxyRestart({ + findLive: async () => ({ status: "uncertain", error }), + startWhenStopped: async () => { calls.push("start"); return true; }, + requestInPlaceRestart: async () => { calls.push("request"); return { accepted: true }; }, + waitForReplacement: async () => { calls.push("wait"); return null; }, + }); + expect(result).toEqual({ ok: false, phase: "request", error }); + expect(calls).toEqual([]); + }); + + test("thrown discovery and replacement errors keep their fail-closed phase", async () => { + const discoveryError = new Error("discovery failed"); + const discovery = await runProxyRestart({ + findLive: async () => { throw discoveryError; }, + startWhenStopped: async () => true, + requestInPlaceRestart: async () => ({ accepted: true }), + waitForReplacement: async () => null, + }); + expect(discovery).toEqual({ ok: false, phase: "request", error: discoveryError }); + + const replacementError = new Error("replacement failed"); + const replacement = await runProxyRestart({ + findLive: async () => ({ + status: "live", + live: { pid: 10, port: 10100, source: "runtime" }, + }), + startWhenStopped: async () => true, + requestInPlaceRestart: async () => ({ accepted: true }), + waitForReplacement: async () => { throw replacementError; }, + }); + expect(replacement).toEqual({ ok: false, phase: "replacement", error: replacementError }); }); }); diff --git a/tests/windows-tray-restart-hardening.test.ts b/tests/windows-tray-restart-hardening.test.ts new file mode 100644 index 0000000000..8e137d9bce --- /dev/null +++ b/tests/windows-tray-restart-hardening.test.ts @@ -0,0 +1,34 @@ +import { describe, expect, test } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +const source = readFileSync(join(import.meta.dir, "../src/tray/windows-tray.ps1"), "utf8"); + +describe("Windows tray restart process hardening", () => { + test("fails a pending action when tracked process state cannot be inspected", () => { + expect(source).toContain("pending process result inspection failed"); + expect(source).toMatch(/catch\s*\{[\s\S]*?pending process result inspection failed[\s\S]*?\$commandFailed\s*=\s*\$true[\s\S]*?\}/); + }); + + test("tracked command failure takes precedence over observed target state", () => { + const failureIndex = source.indexOf("if ($commandFailed) { Complete-PendingAction $false }"); + const reachedIndex = source.indexOf("elseif ($reached) { Complete-PendingAction $true }"); + + expect(failureIndex).toBeGreaterThan(-1); + expect(reachedIndex).toBeGreaterThan(failureIndex); +}); + + test("does not silently swallow pending-process disposal failures during live operation", () => { + const matches = source.match(/pending process dispose failed/g) ?? []; + expect(matches.length).toBeGreaterThanOrEqual(2); + const emptyCatch = ["catch", "{", "}"].join(" "); + expect(source).not.toContain(`try { $script:pendingProcess.Dispose() } ${emptyCatch}`); + }); + + test("tracks Start Proxy and Stop Proxy exit codes like Restart Proxy", () => { + expect(source).toContain('$startProcess = Start-OcxCommand @("__tray-start") -TrackExit'); + expect(source).toContain('$script:pendingProcess = $startProcess'); + expect(source).toContain('$stopProcess = Start-OcxCommand @("stop") -TrackExit'); + expect(source).toContain('$script:pendingProcess = $stopProcess'); + }); +}); diff --git a/tests/windows-tray.test.ts b/tests/windows-tray.test.ts index 9731065f64..70cc0ed1b8 100644 --- a/tests/windows-tray.test.ts +++ b/tests/windows-tray.test.ts @@ -35,6 +35,7 @@ import { setPlatformForTests, } from "../src/lib/windows-secret-acl"; import { handleManagementAPI } from "../src/server/management-api"; +import { MEMORY_DRAIN_RESTART_MS, REPLACEMENT_READY_TIMEOUT_MS } from "../src/server/management/system-restart"; import type { OcxConfig } from "../src/types"; import { INTERNAL_DEADLINE_MS, SPAWN_BUDGET_MS } from "./helpers/test-budget"; @@ -298,6 +299,7 @@ describe("Windows tray packaging and command safety", () => { test("PowerShell controller uses mutex/event shutdown and bans command evaluation", () => { const typescript = readFileSync(join(import.meta.dir, "..", "src", "tray", "windows.ts"), "utf8"); const source = readFileSync(join(import.meta.dir, "..", "src", "tray", "windows-tray.ps1"), "utf8"); + const cli = readFileSync(join(import.meta.dir, "..", "src", "cli", "index.ts"), "utf8"); expect(typescript).not.toContain("\u0000"); expect(typescript).toContain("OCX_TRAY_ENTRY_B64"); expect(typescript).toContain("$startInfo.UseShellExecute = $true"); @@ -307,6 +309,29 @@ describe("Windows tray packaging and command safety", () => { expect(source).toContain("GetPathRoot"); expect(source).toContain("$heartbeat.hostPid = $HostPid"); expect(source).toContain('Start-OcxCommand @("__tray-restart")'); + expect(source).toContain("-TrackExit"); + expect(source).toContain("$script:pendingProcess.HasExited"); + expect(source).toContain('if ($null -ne $script:pendingAction)'); + expect(source).toContain('$startItem.Enabled = $false'); + expect(source).toContain('ignored because $($script:pendingAction) is still pending'); + const startBudget = source.match(/Set-PendingAction "Start Proxy" (\d+)/); + expect(startBudget).not.toBeNull(); + expect(Number(startBudget![1])).toBeGreaterThanOrEqual(75); + const restartBudget = source.match(/Set-PendingAction "Restart Proxy" (\d+)/); + expect(restartBudget).not.toBeNull(); + expect(Number(restartBudget![1]) * 1000).toBeGreaterThanOrEqual( + MEMORY_DRAIN_RESTART_MS + REPLACEMENT_READY_TIMEOUT_MS + 30_000, + ); + expect(cli).toContain("requestBoundSystemRestart(previous, deadlineAt)"); + expect(cli).toContain("Date.now() + PROXY_RESTART_OBSERVE_MS"); + expect(cli).toContain("discoverStableProxyForRestart"); + expect(cli).toContain("isProxyReplacement(previous, live)"); + expect(cli).toContain("process.exitCode = result.ok ? 0 : 1"); + expect(cli).toContain("waitForProxy(40_000)"); + expect(cli).toContain("await handleProxyRestart(() => handleTrayProxyStart(false))"); + expect(cli).toContain("function detachedStartEnvironment()"); + expect(cli).toContain("delete env.OCX_SERVICE"); + expect(cli).not.toContain("OCX_KEEP_ROUTING"); expect(source).toContain('Load-TrayIcon "opencodex-tray-online.ico"'); expect(source).toContain('Load-TrayIcon "opencodex-tray-warning.ico"'); expect(source).toContain('Load-TrayIcon "opencodex-tray-offline.ico"'); diff --git a/tests/winsw-stop-hardening.test.ts b/tests/winsw-stop-hardening.test.ts new file mode 100644 index 0000000000..e3e351eff9 --- /dev/null +++ b/tests/winsw-stop-hardening.test.ts @@ -0,0 +1,19 @@ +import { describe, expect, test } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +describe("WinSW lifecycle stop hardening", () => { + test("re-verifies native service state after stopwait before returning", () => { + const source = readFileSync(join(import.meta.dir, "../src/lib/winsw.ts"), "utf8"); + const start = source.indexOf("export function stopWinswService"); + const end = source.indexOf("export function uninstallWinswService", start); + const stop = source.slice(start, end); + + expect(stop).toContain('runWinsw(["stopwait"])'); + expect(stop).toContain("const status = statusWinswRaw();"); + expect(stop).toContain('status === "stopped" || status === "nonexistent"'); + expect(stop).toContain('status === "unknown"'); + expect(stop).toContain("Native service stop could not be verified."); + expect(stop).toContain("Native service is still running after stop."); + }); +});