Skip to content

Fix slow, fragile OpenClaw upgrade rollback in installer - #107

Merged
yikkuro merged 1 commit into
mainfrom
yikkuro/fix-installer-rollback
Jul 24, 2026
Merged

Fix slow, fragile OpenClaw upgrade rollback in installer#107
yikkuro merged 1 commit into
mainfrom
yikkuro/fix-installer-rollback

Conversation

@yikkuro

@yikkuro yikkuro commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Fixes #108

Problem

Installing MicroClaw on a machine with a previously-interrupted OpenClaw upgrade could hang at ~18–20% for 20+ minutes and then fail, leaving a permanent dead-end that blocked every subsequent install.

Root causes, diagnosed with py-spy on a frozen installer:

  1. _fsync_file opened files with "rb+". os.fsync (→ FlushFileBuffers on Windows) requires write access, but npm packages ship many read-only files. Opening them for writing failed with PermissionError / WinError 5 (access denied) and aborted the rollback.
  2. The fsync walk was serial. It flushed all ~36,000 files of the restored package one at a time (~90 files/sec on a BitLocker + Defender machine → ~22 minutes), with no progress shown, so the installer looked frozen.
  3. A failed rollback was a permanent brick. mark_rollback_failed() left the manifest in rollback-failed (which stays in RECOVERABLE_PHASES) and retained the upgrade lock, so the next install re-entered recovery and failed again with an OpenClaw upgrade is already in progress.

Fixes

1. fsync read-only files + parallelize

  • _fsync_file now catches PermissionError on Windows, temporarily clears the read-only attribute, fsyncs, and restores the original mode in finally. Every file is still durably flushed — no loss of crash-safety.
  • _fsync_payload_tree fsyncs via a ThreadPoolExecutor (fsync is a blocking C call that releases the GIL) instead of a serial crawl, cutting the multi-minute walk dramatically. Used by both backup and restore.

2. Self-heal a failed rollback

  • New OpenClawUpgradeTransaction.discard() drops the on-disk manifest, lock file, and backup tree, and releases the retained lock.
  • _rollback_openclaw_transaction now discards (instead of leaving rollback-failed) when rollback can't complete. Because restores stage a copy and only swap the live tree with a final atomic rename, a failure during staging/fsync leaves the live installation intact; later steps reinstall OpenClaw. Future installs are no longer permanently blocked by UpgradeInProgressError.
  • discard()'s tree removal tolerates read-only files (npm again) so cleanup itself can't fail with WinError 5.

3. Show what the installer is doing

  • Added a progress_detail sub-status line under the progress bar in the installer UI.
  • Fed by the backup/restore file operations, e.g. Restoring OpenClaw files (12,400/36,129 files), so long file ops never look frozen.

Testing

  • python -m unittest discover -s tests107 tests pass.
  • ruff check / ruff format --check → clean.
  • New tests:
    • _fsync_file flushes a read-only file and restores its attribute.
    • _fsync_payload_tree reports progress for every file.
    • A rollback-failed transaction is discard()ed so a fresh install can acquire the lock and proceed.
    • Updated the failed-health-check test to assert the new discard/self-heal behavior.

Related installer-robustness issues: #105 (MAX_PATH during backup), #93 (npm registry block, handled in #106).

The installer could hang for 20+ minutes and then fail while recovering
an interrupted OpenClaw upgrade, leaving a permanent 'rollback-failed'
state that blocked every subsequent install.

Three fixes:

1. fsync read-only files + parallelize. _fsync_file opened files 'rb+',
   which fails with WinError 5 on the read-only files npm ships, aborting
   rollback. It now clears the read-only attribute, fsyncs, and restores
   it, so every file is still durably flushed. _fsync_payload_tree now
   fsyncs via a thread pool instead of a serial ~90 files/sec crawl.

2. Self-heal a failed rollback. Add OpenClawUpgradeTransaction.discard()
   and call it (instead of leaving rollback-failed) so a rollback that
   cannot complete drops its manifest, lock, and backup and releases the
   retained lock. The live install is left intact (staged copy + atomic
   rename) and later steps reinstall OpenClaw, so future installs are no
   longer permanently blocked by UpgradeInProgressError.

3. Show what the installer is doing. Add a progress_detail sub-status
   line under the progress bar, fed by backup/restore file operations
   (e.g. 'Restoring OpenClaw files (12,400/36,129 files)'), so long file
   ops no longer look frozen.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@yikkuro
yikkuro merged commit 389f04a into main Jul 24, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P0] Installer hangs 20+ min then bricks on interrupted OpenClaw upgrade recovery (read-only fsync WinError 5)

1 participant