Skip to content

Multiple NVMe disks have I/O errors #124

@ethnt

Description

@ethnt
$ zpool status blockbuster -v
  pool: blockbuster
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub in progress since Tue Oct 21 12:40:49 2025
        6.76T / 88.5T scanned at 1.42G/s, 4.48T / 88.5T issued at 964M/s
        0B repaired, 5.06% done, 1 days 01:24:12 to go
config:

        NAME                            STATE     READ WRITE CKSUM
        blockbuster                     ONLINE       0     0     0
          wwn-0x5000c500e5cb7f7d-part1  ONLINE       0     0     0
          wwn-0x5000c500e5c823f4-part1  ONLINE       0     0     0
          wwn-0x5000c500e5cb3b2e-part1  ONLINE       0     0     0
          wwn-0x5000c500e5cd75c2-part1  ONLINE       0     0     0
          wwn-0x5000c500e5ccbf09-part1  ONLINE       0     0     0
        logs
          nvme-eui.002538b431a2306e     ONLINE       0     0     0
        cache
          nvme-eui.002538b631a0e093     FAULTED      0    66     0  too many errors

errors: Permanent errors have been detected in the following files:

        blockbuster/media:<0x0>
        blockbuster/tmp:<0x0>
$ dmesg -T
[Thu Oct  2 10:15:16 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1120805595, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:16 2025] I/O error, dev nvme3n1, sector 1120805595 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:16 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=573851416064 size=131072 flags=1048768
[Thu Oct  2 10:15:16 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1120805851, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:16 2025] I/O error, dev nvme3n1, sector 1120805851 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:16 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=573851547136 size=131072 flags=1048768
[Thu Oct  2 10:15:16 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1120808155, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:16 2025] I/O error, dev nvme3n1, sector 1120808155 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:16 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=573852726784 size=131072 flags=1048768
[Thu Oct  2 10:15:18 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1121398799, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:18 2025] I/O error, dev nvme3n1, sector 1121398799 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:18 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574155136512 size=131072 flags=1048768
[Thu Oct  2 10:15:18 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1121468713, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:18 2025] I/O error, dev nvme3n1, sector 1121468713 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:18 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574190932480 size=131072 flags=1048768
[Thu Oct  2 10:15:18 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1121468969, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:18 2025] I/O error, dev nvme3n1, sector 1121468969 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:18 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574191063552 size=131072 flags=1048768
[Thu Oct  2 10:15:18 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1121471273, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:18 2025] I/O error, dev nvme3n1, sector 1121471273 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:18 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574192243200 size=131072 flags=1048768
[Thu Oct  2 10:15:19 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1121693481, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:19 2025] I/O error, dev nvme3n1, sector 1121693481 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:19 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574306013696 size=131072 flags=1048768
[Thu Oct  2 10:15:19 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1121836867, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:19 2025] I/O error, dev nvme3n1, sector 1121836867 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:19 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574379427328 size=131072 flags=1048768
[Thu Oct  2 10:15:19 2025] nvme3n1: I/O Cmd(0x1) @ LBA 1121837123, 256 blocks, I/O Error (sct 0x0 / sc 0x4)
[Thu Oct  2 10:15:19 2025] I/O error, dev nvme3n1, sector 1121837123 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[Thu Oct  2 10:15:19 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574379558400 size=131072 flags=1048768
[Thu Oct  2 10:15:19 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574380738048 size=131072 flags=1048768
[Thu Oct  2 10:15:20 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574417180160 size=131072 flags=1048768
[Thu Oct  2 10:15:20 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574417311232 size=131072 flags=1048768
[Thu Oct  2 10:15:20 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574418490880 size=131072 flags=1048768
[Thu Oct  2 10:15:20 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574454810624 size=131072 flags=1048768
[Thu Oct  2 10:15:20 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574454941696 size=131072 flags=1048768
[Thu Oct  2 10:15:20 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574456121344 size=131072 flags=1048768
[Thu Oct  2 10:15:20 2025] zio pool=blockbuster vdev=/dev/disk/by-id/nvme-eui.002538b631a0e093-part1 error=5 type=2 offset=574477879296 size=131072 flags=1048768

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghardwareHardware-related issues

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions