You don’t really install FreeNAS as you do with some other OS. Instead you have a device (often an USB key) that boots FreeNAS which itself manages your data storage. Indeed why wasting some precious SATA port for a boot device when all you need is a little USB key? Especially when thits boot device is less than 2GB in size.

The problem with USB keys is that they wear out over time. So it inevitably comes a time when it must be replaced. This is what happened to me a few days ago.

The freenas-boot pool switched to a DEGRADED state because checksum error happened too frequently. Fortunately a zpool scrub freenas-boot detected no error, and it seems that zfs was still able to correct those. So instead of reflashing a new USB key and restoring my config backup, I could from FreeNAS itself create a new bootable USB key and replace replace the faulty device in the freenas-boot pool.

Let’s login on FreeNAS and plug in your new USB key. We first find and verify what is the device name for the USB key, in my case it’s da2:

$ gpart part show da2
=>       1  60088319  da2  MBR  (29G)
         1        31       - free -  (16K)
        32  60088288    1  fat32lba  (29G)

You see that MBR table and a FAT32 partition, in my case it can only be the new USB key. We will replace that with GPT partitions. This will contain a boot partition (MBR boot or EFI boot), and a ZFS partition. As root:

# Clean that MBR and create GPT partition table
gpart destroy -F da2
gpart create -s GPT da2

# If you use MBR boot
gpart add -s 64K -t freebsd-boot da2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2

# If you use EFI boot
gpart add -s 128M -t efi da2
gpart bootcode -p /boot/boot1.efifat -i 1 da2

# Create the ZFS partition
gpart add -t freebsd-zfs da2

Now it’s time to replace the device:

$ zpool replace -f freenas-boot da1p2 da2p2
cannot replace da1p2 with da2p2: device is too small
# OOPS! :(

If you got the message device is too small, that is because you’ve been trying to replace a device with another one that is smaller than the original (in term of ZFS partition size I suppose). ZFS requires that you replace a device with another with a size greater or equal to the old one. This can be a problem especially if you have a ZFS partition along with others in the partition table like here, or if the vendor announces a certain disk size but only within a certain approximation. You could result in a disk announced to be the exact same size ending up to be very slighty shorter than your original disk. And as a result you won’t be able to use it to replace any disk in your ZFS pool/vdev. That’s why some recommend to create ZFS filesystems several percents less than the available disk size.

But since we messed up, instead we will create a pool on the other USB key and use ZFS send/recv to migrate the files.

# Create the pool on the new USB key.
# Note that we have to call it freenas-boot2
# because freenas-boot already exists.
zpool create freenas-boot2 da2p2 

# Create a recursive snapshot of the original pool
# that we will use to restore the backup. 
zfs snapshot -r freenas-boot@restore

# Send the snapshot from the old to the new USB key.
zfs send -R freenas-boot@restore | zfs recv -F freenas-boot2

We also need to save the original setting for bootfs because FreeNAS has its own way of managing the boot pool. We will set this value later on the new pool.

$ zpool get bootfs
NAME          PROPERTY  VALUE                        SOURCE
freenas-boot  bootfs    freenas-boot/ROOT/11.3-U4.1  local

We still have to change that name from freenas-boot2 to freenas-boot. However we cannot do so on the running instance of FreeNAS because it already has a pool named freenas-boot. So, we will import it on another ZFS capable machine and fix its name.

(but if anyone knows how to do this live, I’m interested)

# On some other ZFS capable host (FreeBSD for instance ;) )
# import the pool and change its name back to freenas-boot.
# The -f is required because it comes from another host.
zpool import -f freenas-boot2 freenas-boot

# Configure default boot filesystem.
# Remember the value of bootfs we found before on the original pool.
zpool set bootfs=freenas-boot/ROOT/11.3-U4.1 freenas-boot

# Export the pool again.
zpool export freenas-boot

Finally you can clear the old snapshot:

zfs destroy -R freebsd-boot@restore