Manually Installing Encrypted ZFS on Ubuntu 24.04

Josip Medved Linux

For my daily driver I'm fortunate enough to have mirrored ZFS setup,
my secondary machine has only a single SSD slot. While that makes mirror setup
unproductive, I still want to use ZFS. Yes, it cannot automatically correct
errors but it can at least help me know about them. And that's before
considering datasets, quotas, and beautiful snapshots.

For this setup I will use LUKS instead of the native ZFS encryption.
Computer is fast enough that I don't notice difference during daily work and
total encryption is worth it for me.

As before, if you are beginner with ZFS, you might want to use Ubuntu's built-in
ZFS installatiion method. It will result in similar enough setup but without all
these manual steps.

Why do I use this setup? Well, I like to setup my own partitions and I
definitely love setting up my own datasets. In addition, this is the only way to
make Ubuntu do a very minimal install. Lastly, I like to use hibernation with my
computers and setting this during manual installation is often easier than
sorting issues later.

With preamble out of the way, let's go over all the steps to make it happen.

The first step is to boot into the USB installation and use "Try Ubuntu" option.
Once on the a desktop, we want to open a terminal, and, since all further
commands are going to need root access, we can start with that.

sudo -i

Next step should be setting up a few variables - disk, hostname, and username.
This way we can use them going forward and avoid accidental mistakes. Just make
sure to replace these values with ones appropriate for your system.

DISK=/dev/disk/by-id/<whatever>
HOST=radagast
USERNAME=josip

I want to partition disk into 4 partitions. The first two partitions are
unencrypted and in charge of booting (boot + EFI). While I love encryption, I
almost never encrypt the boot partitions in order to make my life easier as you
cannot seamlessly integrate the boot partition password prompt with the later
password prompt. Thus encrypted boot would require you to type the password
twice (or thrice if you decide to use native ZFS encryption on top of that).
Third partition is largest and will contain all user data. The last partition is
actually swap that we need for hibernation support. Since we're using SSD, its
position doesn't matter (on the hard drive you definitely want it way ahead) so
I put it these days at the end to match my mirrored ZFS setup.

All these requirements come in the following few partitioning commands:

DISK_LASTSECTOR=$(( `blockdev --getsz $DISK` / 2048 * 2048 - 2048 - 1 ))
DISK_SWAPSECTOR=$(( DISK_LASTSECTOR - 64 * 1024 * 1024 * 1024 / 512 ))

blkdiscard -f $DISK 2>/dev/null
sgdisk --zap-all                                $DISK
sgdisk -n1:1M:+255M           -t1:EF00 -c1:EFI  $DISK
sgdisk -n2:0:+1792M           -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:$DISK_SWAPSECTOR -t3:8309 -c3:LUKS $DISK
sgdisk -n4:0:$DISK_LASTSECTOR -t4:8200 -c4:Swap $DISK
sgdisk --print                                  $DISK

The next step is to setup all LUKS partitions. If you paid attention, that means
we need to repeat formatting a total of 2 times. Unless you want to deal with
multiple password prompts, make sure to use the same password for each:

cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i $DISK-part3

cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i $DISK-part4

Since creating encrypted partitions doesn't mount them, we do need this as a
separate step. I like to name my LUKS devices based on partition names so we can
recognize them more easily:

cryptsetup luksOpen \
    --persistent --allow-discards \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    $DISK-part3 ${DISK##*/}-part3

cryptsetup luksOpen \
    --persistent --allow-discards \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    $DISK-part4 ${DISK##*/}-part4

Finally, we can set up our ZFS pool with an optional step of setting quota to
roughly 85% of disk capacity. Since we're using LUKS, there's no need to setup
any ZFS keys. Name of the pool will match name of the host and it will contain
several datasets to start with. Most of my stuff goes to either Data dataset
for general use or to VirtualBox dataset for virtual machines. Consider this
just a suggestion and a good starting point, adjust as needed:

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O quota=1600G \
    -O canmount=off -O mountpoint=none -R /mnt/install \
    ${HOST^} /dev/mapper/${DISK##*/}-part3

zfs create -o canmount=noauto -o mountpoint=/ \
    -o reservation=100G \
    ${HOST^}/System
zfs mount ${HOST^}/System

zfs create -o canmount=noauto -o mountpoint=/home \
           ${HOST^}/Home
zfs mount ${HOST^}/Home
zfs set canmount=on ${HOST^}/Home

zfs create -o canmount=noauto -o mountpoint=/Data \
           -o 28433:snapshot=360 \
           ${HOST^}/Data
zfs set canmount=on ${HOST^}/Data

zfs create -o canmount=noauto -o mountpoint=/VirtualBox \
           -o recordsize=32K \
           ${HOST^}/VirtualBox
zfs set canmount=on ${HOST^}/VirtualBox

zfs set devices=off ${HOST^}

With ZFS done, we might as well setup boot, EFI, and swap partitions too:

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI -i 4d65646f $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

mkswap /dev/mapper/${DISK##*/}-part4

At this time, I also often disable IPv6 as I've noticed that on some
misconfigured IPv6 networks it takes ages to download packages. This step is
both temporary (i.e., IPv6 is disabled only during installation) and fully
optional:

sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
sysctl -w net.ipv6.conf.lo.disable_ipv6=1

To start the fun we need to debootstrap our OS. As of this step, you must be
connected to the Internet:

apt update
apt dist-upgrade --yes
apt install --yes debootstrap
debootstrap noble /mnt/install/

We can use our live system to update a few files on our new installation:

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
rm /mnt/install/etc/apt/sources.list
cp /etc/apt/sources.list.d/ubuntu.sources /mnt/install/etc/apt/sources.list.d/ubuntu.sources
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

At last, we're ready to chroot into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install /usr/bin/env \
    DISK=$DISK HOST=$HOST USERNAME=$USERNAME USERID=$USERID \
    bash --login

With our newly installed system running, let's not forget to set up locale and
time zone:

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we're ready to onboard the latest Linux kernel. I find that hwe is a nice
compromise between using generic and oem but you can use whichever you or
your hardware prefers:

apt update
apt install --yes --no-install-recommends linux-generic-hwe-24.04 linux-headers-generic-hwe-24.04

Now we set up crypttab so our encrypted partitions are decrypted on boot:

echo "${DISK##*/}-part3 $DISK-part3 none \
      luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab

echo "${DISK##*/}-part4 $DISK-part4 none \
      luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab

cat /etc/crypttab

To mount all those partitions, we also need some fstab entries too. ZFS
entries are not strictly needed. I just like to add them in order to hide our
LUKS encrypted ZFS from the file manager:

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab

echo "/dev/mapper/${DISK##*/}-part3 \
    none auto nofail,nosuid,nodev,noauto 0 0" >> /etc/fstab

echo "/dev/mapper/${DISK##*/}-part4 \
    swap swap nofail 0 0" >> /etc/fstab

cat /etc/fstab

On systems with a lot of RAM, I like to adjust memory settings a bit. This is
inconsequential in the grand scheme of things, but I like to do it anyway. Think
of it as wearing "lucky" socks:

echo "vm.swappiness=10" >> /etc/sysctl.conf
echo "vm.min_free_kbytes=1048576" >> /etc/sysctl.conf

Now we can create the boot environment:

apt install --yes zfs-initramfs cryptsetup keyutils \
                  grub-efi-amd64-signed shim-signed

update-initramfs -c -k all

And then, we can get grub going. Do note we also set up booting from swap
(needed for hibernation) here too. If you're using secure boot, bootloaded-id HAS
to be Ubuntu:

apt install --yes grub-efi-amd64-signed shim-signed
sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
    RESUME=UUID=$(blkid -s UUID -o value /dev/mapper/${DISK##*/}-part4)\"/" \
    /etc/default/grub

update-grub

grub-install --target=x86_64-efi --efi-directory=/boot/efi \
             --bootloader-id=Ubuntu --recheck --no-floppy

I don't like snap so I preemptively banish it from ever being installed:

apt remove --yes snapd 2>/dev/null
echo 'Package: snapd'    > /etc/apt/preferences.d/snapd
echo 'Pin: release *'   >> /etc/apt/preferences.d/snapd
echo 'Pin-Priority: -1' >> /etc/apt/preferences.d/snapd
apt update

And now, finally, we can install our minimal desktop environment:

apt install --yes ubuntu-desktop-minimal man

Since Firefox is a snapd package (banished), we can install it manually:

add-apt-repository --yes ppa:mozillateam/ppa
cat << 'EOF' | sed 's/^    //' | tee /etc/apt/preferences.d/mozillateamppa
    Package: firefox*
    Pin: release o=LP-PPA-mozillateam
    Pin-Priority: 501
EOF
apt update && apt install --yes firefox

Chrome aficionados, can install it too:

pushd /tmp
wget --inet4-only https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt install ./google-chrome-stable_current_amd64.deb
popd

For hibernation, I like to change sleep settings so that hibernation kicks in
after 13 minutes of sleep:

sed -i 's/.*AllowSuspend=.*/AllowSuspend=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*AllowHibernation=.*/AllowHibernation=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*AllowSuspendThenHibernate=.*/AllowSuspendThenHibernate=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*HibernateDelaySec=.*/HibernateDelaySec=13min/' \
    /etc/systemd/sleep.conf

For that we also need to do a minor lid switch configuration adjustment:

apt install -y pm-utils

sed -i 's/.*HandlePowerKey=.*/HandlePowerKey=hibernate/' \
    /etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitch=.*/HandleLidSwitch=suspend-then-hibernate/' \
    /etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitchExternalPower=.*/HandleLidSwitchExternalPower=ignore/' \
    /etc/systemd/logind.conf

Lastly, we need to have a user too.

adduser --disabled-password --gecos '' $USERNAME
usermod -a -G adm,cdrom,dialout,dip,lpadmin,plugdev,sudo,tty $USERNAME
echo "$USERNAME ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USERNAME
passwd $USERNAME

It took a while, but we can finally exit our debootstrap environment:

exit

Let's clean all mounted partitions and get ZFS ready for next boot:

sync
umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

After reboot, we should be done and our new system should boot with a password
prompt.

reboot

Our newly installed system should boot now.

[2024-08-22: Automatically determining the last sector in DISK_LASTSECTOR variable]
[2024-08-28: Removed manual user ID assignment]

Leave a Reply

Your email address will not be published. Required fields are marked *