No luck with mirror setup

[silverspr]

I would add my accolades and thank you for the outstanding work you’ve done creating this script. I know how challenging such an undertaking is given every user out there in the wild has a different spin, experience which adds to the challenge. You never know what those users are going to do! All this to say I think I’ve followed the instructions as you intended.

I’ve had great success installing to a single drive. For a single drive the process seems rock solid, I’ve tested a number of times using a VM and a 24.04.3 server live CD. Kudos to you for making this available with the server version. As an aside, I did have a problem with XFCE4. At the login in screen rec’d a "could not start session". I ended up installing without a desktop and adding the ubuntu-desktop-minimal later.

I’’m not having as much success with the mirror on 2X nvme0n, I’ve tried 3X (I have the inputs memorized!!), and each time no boot or boot menus. I’ve run the script with the -d option and will include. On reviewing the setup log, from what I can tell it looks like everything copies correctly… but there is no boot/efi directory, only the /boot dir survives the reboot. As a result there is no ZBM or Refind menus.

I’ve looked at the setup log … can’t figure out where its going wrong? I’m sure you’ll see it right away!! This is on a VM, set up with 2 nvme0n. I’m enclosing the setup log and a few snapshots of the setup and finds before the reboot after install. Ultimately since I have a working single disk, I can likely add another drive to make it a mirror but am curious why can’t get it to work out of the box.

I too am on an Asrock server board; this was mentioned by another user of your script in an earlier discussion, not sure if any relevance.

showing setup

boot dir

a shot of the boot manager

ZFS-setup2.log

thanks

[silverspr]

I tried running the script again, I've attached the a screenshot of the system after reboot using lsblk. no mountpoints, unable to see what if what was installed but the disks look like they set up properly?

[Halfwalker]

Thank you for the kudos, it's very much appreciated.

Your test was in a VM with two nvme drives ? That sounds like the drives were passed into the VM directly - you're not using a .vmdk or .vdi disk file right ? I haven't tested that scenario, only the virtual disk images. I wonder if there's a weird disk naming change happening or something ?

It's sounding like the mirror for /boot/efi is not being created correctly. So, so couple of places to look (and I'll do some investigation too) would be :

https://github.com/Halfwalker/ZFS-root/blob/master/ZFS-root.sh#L994-L1009
This is where the mdadm raid is setup for /boot/efi. I would add a set -x right before to ensure the log captures the mdadm commands and results, and add a read -r PAUSING at line 1010, so the script pauses. Then in another terminal (alt-f2) make sure that the mdadm raid was set up correctly. Look at /dev/md/BOOT_EFI. Check mdstat with cat /proc/mdstat.

Also check the /mnt/builder/etc/fstab file to make sure that the entry was created correctly and looks sane.

https://github.com/Halfwalker/ZFS-root/blob/master/ZFS-root.sh#L1318
This is where the mdadm raid /boot/efi gets mounted. Put another read -r PAUSING right after that so you can verify it's getting mounted correctly. This would also validate the fstab entry.

Right after that is https://github.com/Halfwalker/ZFS-root/blob/master/ZFS-root.sh#L1320 where refind gets installed. Another read there, then check /mnt/builder/boot/efi to be sure it was installed correctly.

I have a Virtualbox VM set up with 3 virtual drives - will run an install on that shortly.

[Halfwalker]

Confirmed - the 3-disk install I just created fails to boot

Although I think it might have been because of a heredoc mistake. I had cat > blah < EOF and rather than EOF it was EIF ...

Testing again

[silverspr]

hoping I followed your instructions correctly...I used either "ls" or "cat" to validate file....

Showing VM storage setup...using vdi disk file. Should I be setting up differently?

/proc/mdstat + /dev/md/BOOT_EFI + /mnt/builder/etc/fstab

/boot/efi

/mnt/builder/boot/efi

ZFS-setup4.log

[Halfwalker]

Yup - you got it. I tried another setup, and it looks like refind isn't installing correctly.

/boot/efi/EFI/refind$ sudo refind-install --yes
ShimSource is none
Installing rEFInd on Linux....
ESP was found at /boot/efi using vfat
Found rEFInd installation in /boot/efi/EFI/refind; upgrading it.
Copied rEFInd binary files

Notice: Backed up existing icons directory as icons-backup.
Existing refind.conf file found; copying sample file as refind.conf-sample
to avoid overwriting your customizations.

Could not identify ESP in AddBootEntry()!
Existing //boot/refind_linux.conf found; not overwriting.

Note that line about not identifying the ESP.

Poked around in /usr/sbin/refind-install - line 1268 is the AddBootEntry function. The problem seems to stem from the first part of the function - it can't match /dev/md127 to any of the items in the output of lsblk -r

  :
++ grep '/boot/efi ' /etc/mtab
++ cut -d ' ' -f 1
++ tail -n1
+ InstallPart=/dev/md127
++ grep disk
++ lsblk -r
++ cut -f 1 -d ' '
+ for Name in `lsblk -r | grep disk | cut -f 1 -d " "`
+ [[ /dev/md127 == *\s\d\a* ]]
+ for Name in `lsblk -r | grep disk | cut -f 1 -d " "`
+ [[ /dev/md127 == *\s\d\b* ]]
+ for Name in `lsblk -r | grep disk | cut -f 1 -d " "`
+ [[ /dev/md127 == *\s\d\c* ]]
+ for Name in `lsblk -r | grep disk | cut -f 1 -d " "`
+ [[ /dev/md127 == *\z\d\0* ]]
+ [[ -z '' ]]
+ echo 'Could not identify ESP in AddBootEntry()!'
Could not identify ESP in AddBootEntry()!

What weird is that refind-install hasn't changed. And it was working perfectly fine as of a month or so ago, last time I did a multi-disk install.

So right now I'm not sure what's going on, or the best way to fix it. The output of lsblk -r | grep disk is this

$ lsblk -r | grep disk
sda 8:0 0 10G 0 disk
sdb 8:16 0 10G 0 disk
sdc 8:32 0 10G 0 disk
zd0 230:0 0 300M 0 disk [SWAP]

and sure enough, md127 isn't there (since it isn't a disk per se).

[Halfwalker]

I have to play with it, but I suspect we should be able to add refind to the ESP for each individual disk. But that's a pain, blah.

[Halfwalker]

OK, core problem is that refind-install can't set raid member partitions as boot entries in the UEFI nvram. It's only looking for which disk has /boot/efi mounted. It would need additional logic to parse the raid setup, find the member partitions and add them specifically.

So, never mind ... For a raid boot disk system, we manually run efibootmgr ourselves to add all the partitions and just be done with it.

Just tested in Virtualbox and it works fine. Clean boot and all.

One weird thing though, the initial boot shows an entry for rEFInd in the rEFInd menu.

If you select it you just get right back to the rEFInd menu, without that entry. Inception anyone ?

[silverspr]

Sorry for the delay, dropped my laptop...in for repairs. Awesome work on your part, thanks. Was able to set up the mirrored drive: been testing for the last week or so.

When testing mirror for redundancy, able to boot if the second drive "fails" but if the first drive "fails" the boot also fails with cannot import pool. In other words, cannot boot from the second drive i.e. mirror_1. Googled the issue, says have to repair the bootloader or/and reconstruct the array. suggested to reinstall grub: not sure how to proceed with refind/zbm?

screenshot shows mountpoint for raid1 on first mirror, not the second. Is that the issue?

any suggestions?
Thanks again.

[silverspr]

Hi...yes I'm testing on a VM. Remove by selecting the first drive > remove attachment > OK > Start.

ZBM launches and thats where I get "cannot import pool".

In the recovery shell no pools..nada..nothing

Here I've added the removed drive .. now back to raid1. Note I was able add md127 to HHD2, but thats as far as I got.

[silverspr]

I hesitate to bring this up... but
I broke my existing zfs-on-root mirror, had to reinstall. Reinstall is missing zrepl and zrepl base rescue. No snapshots on boot. I tried the install 3X ... hopefully its me and not you :)

[Halfwalker]

Party hard and fail fast, right ? I should have virtualbox back up and running so I can test in the next day or so

[Halfwalker]

I wonder if "pool" in Unable to import pool is a generic term for the pool in question, or the name of the pool. In which case there's a mismatch - your pool is named rpool per the zbm.prefer setting

[Halfwalker]

OK, Virtualbox 7.2 up and running. I had an existing VM with a raid install using 3x disks from a few weeks ago. I tested releasing both 1 and 2 disks in various permutations, and it was able to boot cleanly. Reattached the disk(s) and it still booted cleanly.

The most I had to do was a manual mdadm --manage /dev/md127 --add /dev/sdb1 to re-add the re-attached disk - one here as an example. Something may have changed somewhere though, so I will be doing a clean 2-disk install with the current script as it stands in the repo. Stay tuned

[silverspr]

My guess .. thinking its more a generic term...

…

On Fri, Sep 12, 2025 at 5:28 PM Halfwalker ***@***.***> wrote: I wonder if "pool" in Unable to import pool is a generic term for the pool in question, or the *name* of the pool. In which case there's a mismatch - your pool is named rpool per the zbm.prefer setting — Reply to this email directly, view it on GitHub <#9 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFUM4FSUFWGTIHJKTWJOXYD3SNQLTAVCNFSM6AAAAACEUGS2Z6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMZYHA4DCOA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[silverspr]

now that you are up and running thought I'd try reinstalling. Still not able to boot from 2nd HDD (nvme0n2).
Zrepl is back though.

Wasn't clear how you were able to run mdadm --manage --add /dev/nvme01p1 .... I'm getting device or resource busy error. tried umount but no difference, didn't try using a livecd. Long story short, not able to boot from nvme02p1.

[Halfwalker]

Very weird - Can't replicate the issue. I just built a new VM with 2 virtual disks og 10G each. Here it is after first boot, looking fine

I shut it down, then release one disk from the VM and reboot. zpool and mdstat look as expected, missing a disk

Shutdown again, and re-attach that disk, reboot. Now the pool resilvers, and mdstat shows the disk missing. Add it back in manually with sudo mdadm --manage /dev/md127 --add /dev/sdb1 and it's all clean again

Then I did the same process with the other disk. Released it, reboot, see the missing disk. Shut down, re-attach disk, reboot. Same process, see the pool resilver and mdstat missing the disk. Have to manually add it again.

[silverspr]

Hi...I so appreciate all the time and effort you've put into trying to resolve my problem. Thank you Thank you.
I must be doing something different/wrong .. I will continue to work with the script. thanks again.

will try another install.

Alrightty then....fresh install. I'm using NVMe controller in the VM: still same issue, not able to boot from second HDD if I simply remove the first drive. On a pure fluke, I removed both drives from the controller only the controller is left, saved (OK). Then added only the second device nvme0n2, saved (OK). Lo and behold it booted. I've been able to duplicate this functionality. Is this an oddity with the VM and NVMe? or is it working as intended? Here's hoping the metal on metal works.

Again thanks for all your help...I'm moving with loading onto a live server....

[Halfwalker]

OK, that's weird. Which version of Virtualbox ? How exactly are you setting up the VM - I'll see if I can replicate. Maybe weirdness with the nvme virtualbox controller ?

[silverspr]

You've done so much already....don't spend a lot of time on it :)
only have 2 vm's this one and a clonezilla
oracle VBox version 7.1.12r169651

[Halfwalker]

Ah ha. I was able to replicate with the NVME controller rather than the Sata controller in Virtualbox. If the 1st nvme virtual disk is released, then the 2nd ones does NOT show up when zfsbootmenu comes up. If the 2nd disk is release (so there's still one on port 0) then it comes up as normal.

This appears to be something to do with how Virtualbox handles nvme drives I think. If it doesn't see one on port 0, then I guess it just stops scanning ? Be interesting to see how bare metal handles it.

Workaround - move the existing 2nd disk from port 1 to port 0 and reboot. It all comes up as expected.

This is after I added the 1st device back in on port 1

Using the Sata controller you can remove either disk and it will all work as expected. Nvme is handled differently, dammit.

[silverspr]

sweet...thank you yet again. yahhh...it wasn't me!!
your answer should be upvoted as the solution....

[silverspr]

closed