Filesystem corrupts the uv binaries

[DavidSeyserGit]

systemd units crash-loop after power outage due to uv binary corruption on failing root filesystem

Summary

After a power outage, the three TWFarmBot user services (twfarmbot-api, twfarmbot-ui, twfarmbot-resireg) entered a crash-restart loop. systemd was correctly trying to restart them, but the uv binary on disk was being corrupted by filesystem-level CRC errors, causing every launch to fail with SIGSEGV/SIGILL.

What happened

1. After the outage the services would not stay up.

2. systemctl --user status showed them in activating (auto-restart) with the main process killed by SIGSEGV.

3. Replacing /home/farmbot/.local/bin/uv with a fresh copy temporarily fixed the crash, but minutes later the new copy was also corrupted and began failing with SIGILL.

4. tune2fs -l /dev/mmcblk0p2 and dmesg revealed the root filesystem has checksum errors:

   Filesystem state:         clean with errors
   First error err:          EFSBADCRC
   

   EXT4-fs error (device mmcblk0p2): __ext4_find_entry:1624: inode #50437: comm wireplumber: checksumming directory block 0
   

Workaround applied locally

To get the services running immediately without relying on the corrupted uv binary, the unit files in ~/.config/systemd/user/ were changed to:

• Run the entry-point scripts directly from the project virtualenv (/home/farmbot/TWFarmBot/.venv/bin/...).
• Load environment variables via EnvironmentFile=/home/farmbot/TWFarmBot/.env.
• Add .venv/bin to PATH for the UI unit so streamlit can be found.

After the change all three services started and stayed up:

● twfarmbot-api.service     active (running)   pid 9676   http://localhost:8000/health -> 200 OK
● twfarmbot-resireg.service active (running)   pid 9675   http://localhost:8080/
● twfarmbot-ui.service      active (running)   pid 10405  http://localhost:8501/       -> 200 OK

API health check:

{"status":"ok","actions":["dismount_tool","e_stop","find_home","mount_tool","move","move_path","read_pin","take_photo","water","write_pin"],"farmbot":"connected"}

Permanent fix needed

This is a storage/filesystem issue, not a code issue. The SD card / eMMC root filesystem needs repair:

# Run from a recovery boot or live medium where /dev/mmcblk0p2 is unmounted
fsck -y /dev/mmcblk0p2

If the card itself is failing, it should be replaced and the system re-imaged.

Suggested follow-ups for the project

• Consider documenting how to run the services without uv (direct venv path) as a disaster-recovery fallback.
• Evaluate whether uv needs to live on the root filesystem or can be installed to a less failure-prone location.

Environment

• Hardware: Raspberry Pi 5
• OS: Linux farmbot 6.18.34+rpt-rpi-2712 aarch64
• Filesystem: /dev/mmcblk0p2 ext4
• TWFarmBot branch: resi
• Commit: f41dea5
• uv version originally installed: 0.11.25