Manually Installing Encrypted ZFS on Ubuntu 24.04
For my daily driver I’m fortunate enough to have mirrored ZFS setup, my secondary machine has only a single SSD slot. While that makes mirror setup unproductive, I still want to use ZFS. Yes, it cannot automatically correct errors but it can at least help me know about them. And that’s before considering datasets, quotas, and beautiful snapshots.
For this setup I will use LUKS instead of the native ZFS encryption. Computer is fast enough that I don’t notice difference during daily work and total encryption is worth it for me.
As before, if you are beginner with ZFS, you might want to use Ubuntu’s built-in ZFS installatiion method. It will result in similar enough setup but without all these manual steps.
Why do I use this setup? Well, I like to setup my own partitions and I definitely love setting up my own datasets. In addition, this is the only way to make Ubuntu do a very minimal install. Lastly, I like to use hibernation with my computers and setting this during manual installation is often easier than sorting issues later.
With preamble out of the way, let’s go over all the steps to make it happen.
The first step is to boot into the USB installation and use “Try Ubuntu” option. Once on the a desktop, we want to open a terminal, and, since all further commands are going to need root access, we can start with that.
sudo -iNext step should be setting up a few variables - disk, hostname, and username. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.
DISK=/dev/disk/by-id/<whatever>
HOST=radagast
USERNAME=josipI want to partition disk into 4 partitions. The first two partitions are unencrypted and in charge of booting (boot + EFI). While I love encryption, I almost never encrypt the boot partitions in order to make my life easier as you cannot seamlessly integrate the boot partition password prompt with the later password prompt. Thus encrypted boot would require you to type the password twice (or thrice if you decide to use native ZFS encryption on top of that). Third partition is largest and will contain all user data. The last partition is actually swap that we need for hibernation support. Since we’re using SSD, its position doesn’t matter (on the hard drive you definitely want it way ahead) so I put it these days at the end to match my mirrored ZFS setup.
All these requirements come in the following few partitioning commands:
DISK_LASTSECTOR=$(( `blockdev --getsz $DISK` / 2048 * 2048 - 2048 - 1 ))
DISK_SWAPSECTOR=$(( DISK_LASTSECTOR - 64 * 1024 * 1024 * 1024 / 512 ))
blkdiscard -f $DISK 2>/dev/null
sgdisk --zap-all $DISK
sgdisk -n1:1M:+255M -t1:EF00 -c1:EFI $DISK
sgdisk -n2:0:+1792M -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:$DISK_SWAPSECTOR -t3:8309 -c3:LUKS $DISK
sgdisk -n4:0:$DISK_LASTSECTOR -t4:8200 -c4:Swap $DISK
sgdisk --print $DISKThe next step is to setup all LUKS partitions. If you paid attention, that means we need to repeat formatting a total of 2 times. Unless you want to deal with multiple password prompts, make sure to use the same password for each:
cryptsetup luksFormat -q --type luks2 \
--sector-size 4096 \
--cipher aes-xts-plain64 --key-size 256 \
--pbkdf argon2i $DISK-part3
cryptsetup luksFormat -q --type luks2 \
--sector-size 4096 \
--cipher aes-xts-plain64 --key-size 256 \
--pbkdf argon2i $DISK-part4Since creating encrypted partitions doesn’t mount them, we do need this as a separate step. I like to name my LUKS devices based on partition names so we can recognize them more easily:
cryptsetup luksOpen \
--persistent --allow-discards \
--perf-no_write_workqueue --perf-no_read_workqueue \
$DISK-part3 ${DISK##*/}-part3
cryptsetup luksOpen \
--persistent --allow-discards \
--perf-no_write_workqueue --perf-no_read_workqueue \
$DISK-part4 ${DISK##*/}-part4Finally, we can set up our ZFS pool with an optional step of setting quota to roughly 85% of disk capacity. Since we’re using LUKS, there’s no need to setup any ZFS keys. Name of the pool will match name of the host and it will contain several datasets to start with. Most of my stuff goes to either Data dataset for general use or to VirtualBox dataset for virtual machines. Consider this just a suggestion and a good starting point, adjust as needed:
zpool create -o ashift=12 -o autotrim=on \
-O compression=lz4 -O normalization=formD \
-O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
-O quota=1600G \
-O canmount=off -O mountpoint=none -R /mnt/install \
${HOST^} /dev/mapper/${DISK##*/}-part3
zfs create -o canmount=noauto -o mountpoint=/ \
-o reservation=100G \
${HOST^}/System
zfs mount ${HOST^}/System
zfs create -o canmount=noauto -o mountpoint=/home \
${HOST^}/Home
zfs mount ${HOST^}/Home
zfs set canmount=on ${HOST^}/Home
zfs create -o canmount=noauto -o mountpoint=/Data \
-o 28433:snapshot=360 \
${HOST^}/Data
zfs set canmount=on ${HOST^}/Data
zfs create -o canmount=noauto -o mountpoint=/VirtualBox \
-o recordsize=32K \
${HOST^}/VirtualBox
zfs set canmount=on ${HOST^}/VirtualBox
zfs set devices=off ${HOST^}With ZFS done, we might as well setup boot, EFI, and swap partitions too:
yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/
mkfs.msdos -F 32 -n EFI -i 4d65646f $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi
mkswap /dev/mapper/${DISK##*/}-part4At this time, I also often disable IPv6 as I’ve noticed that on some misconfigured IPv6 networks it takes ages to download packages. This step is both temporary (i.e., IPv6 is disabled only during installation) and fully optional:
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
sysctl -w net.ipv6.conf.lo.disable_ipv6=1To start the fun we need to debootstrap our OS. As of this step, you must be connected to the Internet:
apt update
apt dist-upgrade --yes
apt install --yes debootstrap
debootstrap noble /mnt/install/We can use our live system to update a few files on our new installation:
echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
rm /mnt/install/etc/apt/sources.list
cp /etc/apt/sources.list.d/ubuntu.sources /mnt/install/etc/apt/sources.list.d/ubuntu.sources
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/At last, we’re ready to chroot into our new system.
mount --rbind /dev /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys /mnt/install/sys
chroot /mnt/install /usr/bin/env \
DISK=$DISK HOST=$HOST USERNAME=$USERNAME USERID=$USERID \
bash --loginWith our newly installed system running, let’s not forget to set up locale and time zone:
locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales
dpkg-reconfigure tzdataNow we’re ready to onboard the latest Linux kernel. I find that hwe is a nice compromise between using generic and oem but you can use whichever you or your hardware prefers:
apt update
apt install --yes --no-install-recommends linux-generic-hwe-24.04 linux-headers-generic-hwe-24.04Now we set up crypttab so our encrypted partitions are decrypted on boot:
echo "${DISK##*/}-part3 $DISK-part3 none \
luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
echo "${DISK##*/}-part4 $DISK-part4 none \
luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
cat /etc/crypttabTo mount all those partitions, we also need some fstab entries too. ZFS entries are not strictly needed. I just like to add them in order to hide our LUKS encrypted ZFS from the file manager:
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
/boot ext4 noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
/boot/efi vfat noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab
echo "/dev/mapper/${DISK##*/}-part3 \
none auto nofail,nosuid,nodev,noauto 0 0" >> /etc/fstab
echo "/dev/mapper/${DISK##*/}-part4 \
swap swap nofail 0 0" >> /etc/fstab
cat /etc/fstabOn systems with a lot of RAM, I like to adjust memory settings a bit. This is inconsequential in the grand scheme of things, but I like to do it anyway. Think of it as wearing “lucky” socks:
echo "vm.swappiness=10" >> /etc/sysctl.conf
echo "vm.min_free_kbytes=1048576" >> /etc/sysctl.confNow we can create the boot environment:
apt install --yes zfs-initramfs cryptsetup keyutils \
grub-efi-amd64-signed shim-signed
update-initramfs -c -k allAnd then, we can get grub going. Do note we also set up booting from swap (needed for hibernation) here too. If you’re using secure boot, bootloaded-id HAS to be Ubuntu:
apt install --yes grub-efi-amd64-signed shim-signed
sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
RESUME=UUID=$(blkid -s UUID -o value /dev/mapper/${DISK##*/}-part4)\"/" \
/etc/default/grub
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
--bootloader-id=Ubuntu --recheck --no-floppyI don’t like snap so I preemptively banish it from ever being installed:
apt remove --yes snapd 2>/dev/null
echo 'Package: snapd' > /etc/apt/preferences.d/snapd
echo 'Pin: release *' >> /etc/apt/preferences.d/snapd
echo 'Pin-Priority: -1' >> /etc/apt/preferences.d/snapd
apt updateAnd now, finally, we can install our minimal desktop environment:
apt install --yes ubuntu-desktop-minimal manSince Firefox is a snapd package (banished), we can install it manually:
add-apt-repository --yes ppa:mozillateam/ppa
cat << 'EOF' | sed 's/^ //' | tee /etc/apt/preferences.d/mozillateamppa
Package: firefox*
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 501
EOF
apt update && apt install --yes firefoxChrome aficionados, can install it too:
pushd /tmp
wget --inet4-only https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt install ./google-chrome-stable_current_amd64.deb
popdFor hibernation, I like to change sleep settings so that hibernation kicks in after 13 minutes of sleep:
sed -i 's/.*AllowSuspend=.*/AllowSuspend=yes/' \
/etc/systemd/sleep.conf
sed -i 's/.*AllowHibernation=.*/AllowHibernation=yes/' \
/etc/systemd/sleep.conf
sed -i 's/.*AllowSuspendThenHibernate=.*/AllowSuspendThenHibernate=yes/' \
/etc/systemd/sleep.conf
sed -i 's/.*HibernateDelaySec=.*/HibernateDelaySec=13min/' \
/etc/systemd/sleep.confFor that we also need to do a minor lid switch configuration adjustment:
apt install -y pm-utils
sed -i 's/.*HandlePowerKey=.*/HandlePowerKey=hibernate/' \
/etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitch=.*/HandleLidSwitch=suspend-then-hibernate/' \
/etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitchExternalPower=.*/HandleLidSwitchExternalPower=ignore/' \
/etc/systemd/logind.confLastly, we need to have a user too.
adduser --disabled-password --gecos '' $USERNAME
usermod -a -G adm,cdrom,dialout,dip,lpadmin,plugdev,sudo,tty $USERNAME
echo "$USERNAME ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USERNAME
passwd $USERNAMEIt took a while, but we can finally exit our debootstrap environment:
exitLet’s clean all mounted partitions and get ZFS ready for next boot:
sync
umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -aAfter reboot, we should be done and our new system should boot with a password prompt.
rebootOur newly installed system should boot now.
[2024-08-22: Automatically determining the last sector in DISK_LASTSECTOR variable] [2024-08-28: Removed manual user ID assignment]