Encrypted ZFS Root on Ubuntu Server 20.04 (with USB Unlock)
It’s all nice and dandy to setup unencrypted ZFS on server or setting it up with boot encryption. However, what if we want to use USB to unlock the encrypted drive? And no, it’s not as crazy as it seems. Scenarios are actually plentiful.
One scenario is when you have your servers encrypted (as realistically everybody should) but you don’t necessarily want (or can) enter the password. If you can plug in USB with a key and make ZFS use that key, you suddenly have password-less boot without connecting to the server. After boot is done, you can unplug the USB and store it somewhere safe. And this can be done by literally anybody you trust - it doesn’t have to be you.
My favorite scenario is using it with self-erasing USB drive. You place the encryption key on the small drive and it will be there for every boot. You can use your server as you normally would. However, if power is lost, your key will disappear and content of server will not be accessible anymore. When would such crazy scenario happen you ask? Well, if anybody is stealing your server they have to unplug it first. And yes, your server is gone but at least your data is not.
I admit I never had that scenario happen to me - fortunately all my servers are still accounted for. But I did RMA disk drives. And worrying about erasing the data when you cannot access it anymore is a bit too late.
Whatever might be your case, let me guide you through setting up natively-encrypted ZFS with a key on the USB drive.
Once you enter the shell of the installation media, the very first step is setting up a few variables - location of disk and USB drive, followed by pool and host name. This way we can use them going forward and avoid accidental mistakes. Make sure to replace these values with the ones appropriate for your system.
DISK=/dev/disk/by-id/^^ata-xxx^^
USB=/dev/disk/by-id/^^usb-xxx^^
POOL=^^Ubuntu^^
HOST=^^server^^Next let’s sort out question of the encryption key. Assumption is that the key will be on the first partition of the FAT formatted USB drive and we’ll mount it at /tmpusb. While you could create the key material directly, I personally prefer the passphrase as it makes life easier in the case of recovery. If you already have the passphrase on the drive just skip the last command as it will overwrite it.
mkdir /tmpusb
mount -t vfat -o rw "$USB-part1" /tmpusb
echo -n "^^password^^" > /tmpusb/boot.pwdGeneral idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are installing on SSD blkdiscard will trim all the data. You can safely ignore any errors on disks that don’t support it.
blkdiscard $DISK 2>/dev/null
sgdisk --zap-all $DISK
sgdisk -n1:1M:+127M -t1:EF00 -c1:EFI $DISK
sgdisk -n2:0:+512M -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:0 -t3:8309 -c3:Ubuntu $DISK
sgdisk --print $DISKTo kick off the fun of the installation we need debootstrap and zfsutils-linux package.
apt update
apt install --yes debootstrap zfsutils-linuxNow we’re ready to create system ZFS pool.
zpool create -o ashift=12 -o autotrim=on \
-O compression=lz4 -O normalization=formD \
-O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
-O encryption=aes-256-gcm -O keyformat=passphrase -O keylocation=file:///tmpusb/boot.pwd \
-O canmount=off -O mountpoint=none -R /mnt/install $POOL $DISK-part3
zfs create -o canmount=noauto -o mountpoint=/ $POOL/System
zfs mount $POOL/SystemAssuming UEFI boot, two additional partitions are needed - one for EFI and one for booting. I don’t have ZFS pool for boot partition but a plain old ext4 as I find potential fixup works better that way.
yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/
mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efiBootstrapping Ubuntu on the newly created pool is next. This will take a while.
debootstrap focal /mnt/install/
zfs set devices=off $POOLOur newly copied system is lacking a few files and we should make sure they exist before proceeding.
echo $HOST > /mnt/install/etc/hostname
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/Finally we’re ready to “chroot” into our new system.
mount --rbind /dev /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys /mnt/install/sys
chroot /mnt/install /usr/bin/env DISK=$DISK USB=$USB POOL=$POOL bash --loginLet’s not forget to setup locale and time zone.
locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales
dpkg-reconfigure tzdataTo mount EFI and boot partitions, we need to do some fstab setup too:
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
/boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
/boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstabNow we’re ready to onboard the latest Linux image.
apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-genericFollowed by the boot environment packages.
apt install --yes zfs-initramfs plymouth grub-efi-amd64-signed shim-signedNow it’s time to setup boot scripts to ensure USB drive is mounted before ZFS needs it. I found that initramfs’ init-premount directory is the ideal spot.
cat << EOF > /usr/share/initramfs-tools/scripts/init-premount/tmpusb
#!/bin/sh -e
PREREQ="udev"
prereqs() {
echo "\$PREREQ"
}
case \$1 in
prereqs)
prereqs
exit 0
;;
esac
USB="$USB"
POOL="$POOL"
echo "Waiting for \$USB"
for I in \`seq 1 20\`; do
if [ -e "\$USB" ]; then break; fi
echo -n .
sleep 1
done
echo
sleep 2
if [ -e "\$USB" ]; then
mkdir /tmpusb
mount -t vfat -o ro "\$USB-part1" /tmpusb
if [ \$? -eq 0 ]; then
exit 0
else
echo "Error mounting \$USB-part1" >&2
fi
else
echo "Cannot find \$USB" >&2
fi
exit 1
EOF
# chmod 755 /usr/share/initramfs-tools/scripts/init-premount/tmpusb
# cat << EOF > /usr/share/initramfs-tools/scripts/init-bottom/tmpusb
#!/bin/sh -e
PREREQ="udev"
prereqs() {
echo "\$PREREQ"
}
case \$1 in
prereqs)
prereqs
exit 0
;;
esac
if [ -e "/tmpusb" ]; then
umount /tmpusb
rmdir /tmpusb
fi
EOF
chmod 755 /usr/share/initramfs-tools/scripts/init-bottom/tmpusbThe first script will wait for USB drive if needed and mount it at /tmpusb for ZFS to find. Second script is there just for a bit of cleanup.
If USB drive is not mounted, this will cause boot to fail. If we want ZFS to ask for the passphrase instead (despite having file as the keylocation) a further customization is needed. But note these commands might need adjustment and they definitely need to be repeated each time ZFS package is updated. I might go into the details in some future post but suffice to say this is really not future-proof solution but it’s the minimum set of changes that I could make sed work with.
sed -i 's/load-key/load-key -L prompt/' /usr/share/initramfs-tools/scripts/zfs
sed -i '0,/load-key/ {s/-L prompt//}' /usr/share/initramfs-tools/scripts/zfs
sed -i '/KEYSTATUS=/i \\t\t\t$ZFS load-key "${ENCRYPTIONROOT}"' /usr/share/initramfs-tools/scripts/zfs
sed -i '/KEYSTATUS=/i \\t\t\tKEYLOCATION=prompt' /usr/share/initramfs-tools/scripts/zfsIn lieu of warning, suffice it to say these changes to zfs script are suitable only for this scenario and don’t really work for anything else.
Now we get grub started and update our boot environment. Due to Ubuntu 19.10 having some kernel version kerfuffle, we need to manually create initramfs image. This is also a good moment to check if our script is in.
KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL
lsinitramfs /boot/initrd.img-$KERNEL | grep tmpusbGrub update is what makes EFI tick.
update-grub 2>/dev/null
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
--recheck --no-floppyShort package upgrade will not hurt.
apt dist-upgrade --yesWe can omit creation of the swap dataset but I personally find its good to have it just in case.
zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
-o sync=always -o primarycache=metadata -o secondarycache=none $POOL/Swap
mkswap -f /dev/zvol/$POOL/Swap
echo "/dev/zvol/$POOL/Swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resumeThis is a good time to install other packages (e.g.,openssh-server) and do any setup you might need (e.g.,firewall). If nothing else, then setup root password so you have a way to log in (I personally prefer to create another user and leave root passwordless).
passwdAs installation is finally done, we can exit our chroot environment.
exitAnd cleanup mount points.
umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -aAfter the reboot you should be able to enjoy your installation.
reboot