Persistent disk mount issue with Nitro instances

[Ivaylogi98]

Summary

On AWS Nitro-based instances, attaching a persistent disk and deploying fails with:

mount /dev/nvme2n1p1 /var/vcap/store -> wrong fs type, bad superblock

BOSH then detaches the disk and rolls back. The VM is left with only root and ephemeral
disks.

Environment

• AWS Nitro-based instance
• Agent config at time of failure:

  "disks": {
    "system": "/dev/xvda",
    "ephemeral": "/dev/sdb",
    "persistent": {}
  }
  

Observed behaviour

• Persistent disk is created and attached successfully by the CPI
• Agent resolves the device path and attempts to mount it
• Mount fails: wrong fs type, bad superblock
• BOSH detaches the disk and rolls back

Analysis

persistent is empty in the agent settings at mount time, meaning the agent resolves
the device path dynamically via IDDevicePathResolver. On Nitro instances, NVMe PCIe
enumeration order is non-deterministic. If the symlink in /dev/disk/by-id/ resolves to
the wrong NVMe device — one that is unformatted or has the wrong filesystem — the mount
fails with bad superblock.

The disk ID passed from the CPI may not match the symlink name, or there may be multiple
matching symlinks causing ambiguity.

Related

• Fix nvme instance storage discovery #407 — fixes the same class of problem for NVMe instance
  storage discovery
• Fix NVMe raw_instance_storage device enumeration for all instance families bosh-aws-cpi-release#196 — companion CPI-side change

Co-authored with Claude Code