Encrypted ZFS on Ubuntu 22.10

Ubuntu 22.10 in release notes brought one unpleasant news: “The option to install using zfs as a file system and encryption has been disabled due to a bug”. The official recommendation is to “simply” install Ubuntu 22.04 and upgrade from there. However, if you are willing to go over a lot of manual steps, that’s not necessary.

Those familiar with my previous installation guides will note two things. First, steps below will use LUKS encryption instead of the native ZFS option. I’ve been going back and forth on this one as I do like native ZFS encryption but in setups with hibernation enabled (and this one will be one of those), having both swap with LUKS and native ZFS would require user to type password twice. Not a great user experience.

Second, my personal preferences will “leak through”. This guide will clearly show dislike of snap system and use of way too big swap partition to facilitate hibernation under even the worst-case scenario. Your preferences might vary and thus you might want to adjust guide as necessary. The important part is disk and boot partition setup, everything else is just extra fluff.

Without further ado, let’s proceed with the install.

After booting into Ubuntu desktop installation (via “Try Ubuntu” option) we want to open a terminal. Since all further commands are going to need root credentials, we can start with that.

sudo -i

The very first step should be setting up a few variables - disk, hostname, and username. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system. I like to use upper-case for ZFS pool as that’s what will appear as password prompt. It just looks nicer and ZFS doesn’t care either way.

DISK=/dev/disk/by-id/<diskid>
HOST=<hostname>
USER=<username>

For my setup I want 4 partitions. First two partitions will be unencrypted and in charge of booting. While I love encryption, I decided not to encrypt boot partition in order to make my life easier as you cannot integrate boot partition password prompt with the later data password prompt thus requiring you to type password twice. Both swap and ZFS partitions are fully encrypted.

Also, my swap size is way too excessive since I have 64 GB of RAM and I wanted to allow for hibernation under the worst of circumstances (i.e., when RAM is full). Hibernation usually works with much smaller partitions but I wanted to be sure and my disk was big enough to accommodate.

Lastly, while blkdiscard does nice job of removing old data from the disk, I would always recommend also using dd if=/dev/urandom of=$DISK bs=1M status=progress if your disk was not encrypted before.

blkdiscard -f $DISK
sgdisk --zap-all                      $DISK
sgdisk -n1:1M:+127M -t1:EF00 -c1:EFI  $DISK
sgdisk -n2:0:+896M  -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:+65G   -t3:8200 -c3:Swap $DISK
sgdisk -n4:0:0      -t4:8309 -c4:ZFS  $DISK
sgdisk --print                        $DISK

Once partitions are created, we want to setup our LUKS encryption.

cryptsetup luksFormat -q --type luks2 \
    --cipher aes-xts-plain64 --key-size 512 \
    --pbkdf argon2i $DISK-part4
cryptsetup luksFormat -q --type luks2 \
    --cipher aes-xts-plain64 --key-size 512 \
    --pbkdf argon2i $DISK-part3

Since creating encrypted partition doesn’t mount them, we do need this as a separate step. Notice I use host name as the name of the main data partition.

cryptsetup luksOpen $DISK-part4 $HOST
cryptsetup luksOpen $DISK-part3 swap

Finally, we can setup our ZFS pool with an optional step of setting quota to roughly 80% of disk capacity. Adjust exact value as needed.

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O canmount=off -O mountpoint=none -R /mnt/install ${HOST^} /dev/mapper/$HOST
zfs set quota=1.5T ${HOST^}

I used to be fan of using just a main dataset for everything but these days I use more conventional “separate root dataset” approach.

zfs create -o canmount=noauto -o mountpoint=/ ${HOST^}/Root
zfs mount ${HOST^}/Root

And a separate home partition will not be forgotted.

zfs create -o canmount=noauto -o mountpoint=/home ${HOST^}/Home
zfs mount ${HOST^}/Home
zfs set canmount=on ${HOST^}/Home

With all datasets in place, we can finish setting the main dataset properties.

zfs set devices=off ${HOST^}

Now it’s time to format swap.

mkswap /dev/mapper/swap

And then boot partition.

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot

And finally EFI partition.

mkfs.msdos -F 32 -n EFI -i 4d65646f $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

At this time, I also like to disable IPv6 as I’ve noticed that on some misconfigured IPv6 networks it takes ages to download packages. This step is both temporary (i.e., IPv6 is disabled only during installation) and fully optional.

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1

To start the fun we need debootstrap package. Starting this step, you must be connected to the Internet.

apt update && apt install --yes debootstrap

Bootstrapping Ubuntu on the newly created pool comes next. This will take a while.

debootstrap $(basename `ls -d /cdrom/dists/*/ | grep -v stable | head -1`) /mnt/install/

We can use our live system to update a few files on our new installation.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials. Don’t worry if this returns errors - that just means you are not using wireless.

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Atlast, we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK USER=$USER \
    bash --login

With our newly installed system running, let’s not forget to setup locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales
dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by the boot environment packages.

apt install --yes zfs-initramfs cryptsetup keyutils grub-efi-amd64-signed shim-signed

Now it’s time to setup crypttab so our encrypted partitions are decrypted on boot.

echo "$HOST UUID=$(blkid -s UUID -o value $DISK-part4) none \
    luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
echo "swap UUID=$(blkid -s UUID -o value $DISK-part3) none \
    swap,luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
cat /etc/crypttab

To mount all those partitions, we need also some fstab entries. The last entry is not strictly needed. I just like to add it in order to hide our LUKS encrypted ZFS from file manager.

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "UUID=$(blkid -s UUID -o value /dev/mapper/swap) \
    none swap defaults 0 0" >> /etc/fstab
echo "/dev/disk/by-uuid/$(blkid -s UUID -o value /dev/mapper/$HOST) \
    none auto nosuid,nodev,nofail 0 0" >> /etc/fstab
cat /etc/fstab

If hibernation is desired, a few settings need to be added.

sed -i 's/.*AllowSuspend=.*/AllowSuspend=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*AllowHibernation=.*/AllowHibernation=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*AllowSuspendThenHibernate=.*/AllowSuspendThenHibernate=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*HibernateDelaySec=.*/HibernateDelaySec=13min/' \
    /etc/systemd/sleep.conf

Since we’re doing laptop setup, we can also setup lid switch. I like to set it as suspend-then-hibernate.

sed -i 's/.*HandleLidSwitch=.*/HandleLidSwitch=suspend-then-hibernate/' \
    /etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitchExternalPower=.*/HandleLidSwitchExternalPower=suspend-then-hibernate/' \
    /etc/systemd/logind.conf

Lastly, I adjust swappiness a bit.

echo "vm.swappiness=10" >> /etc/sysctl.conf

And then, we can get grub going. Do note we also setup booting from swap (hibernation support) here too.

sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
    nvme.noacpi=1 \
    RESUME=UUID=$(blkid -s UUID -o value /dev/mapper/swap)\"/" \
    /etc/default/grub
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy
update-initramfs -u -k all

Next step is to create boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -c -k $KERNEL

Finally, we can install our desktop environment.

apt install --yes ubuntu-desktop-minimal

Once installation is done, I like to remove snap and banish it from ever being installed.

apt remove --yes snapd
echo 'Package: snapd'    > /etc/apt/preferences.d/snapd
echo 'Pin: release *'   >> /etc/apt/preferences.d/snapd
echo 'Pin-Priority: -1' >> /etc/apt/preferences.d/snapd

Since Firefox is only available as snapd package, we can install it manually.

add-apt-repository --yes ppa:mozillateam/ppa
cat << 'EOF' | sed 's/^    //' | tee /etc/apt/preferences.d/mozillateamppa
    Package: firefox*
    Pin: release o=LP-PPA-mozillateam
    Pin-Priority: 501
EOF
apt update && apt install --yes firefox

While at it, I might as well get Chrome too.

cd /tmp
wget --inet4-only https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt install ./google-chrome-stable_current_amd64.deb

For Framework Laptop I use here, we need one more adjustment due to Dell audio needing a special care. Note that owners of Gen12 boards need a few more adjustments.

echo "options snd-hda-intel model=dell-headset-multi" >> /etc/modprobe.d/alsa-base.conf

Of course, we need to have a user too.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dialout,dip,lpadmin,plugdev,sudo,tty $USER
echo "$USER ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USER
passwd $USER

I like to add some extra packages and do one final upgrade before calling it done.

add-apt-repository --yes universe
apt update && apt dist-upgrade --yes

It took a while but we can finally exit our debootstrap environment.

exit

Let’s clean all mounted partitions and get ZFS ready for next boot.

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

With reboot, we should be done and our new system should boot with a password prompt.

reboot

Once we log into it, we need to adjust boot image and test hibernation. If you see your desktop after waking it up, all is good.

sudo update-initramfs -u -k all
sudo systemctl hibernate

If you get Failed to hibernate system via logind: Sleep verb "hibernate" not supported go into BIOS and disable secure boot (Enforce Secure Boot option). Unfortunately, secure boot and hibernation still don’t work together but there is some work in progress to make it happen in future. At this time, you need to select one or the other.

PS: If you already installed system with secure boot and hibernation is not working, run update-initramfs -c -k all and try again.

PPS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 22.04, 21.10, and 20.04.

PPPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, 19.04, and 18.10.

Reading 74HC165 with Reduced Pin Count

For a project of mine, I had to read an 8-bit DIP switch. Easy solution was to use 74HC165 but there I had a problem. Fully interfacing with it requires 4 lines: Shift/Load, Clock, Clock Inhibit, and Q7. However, I only had 3 lines available.

Obvious one was getting rid of Clock Inhibit as that one can be tied to low permanently. This leaves only 3 lines but with a slightly unclear handling as official datasheet uses Clock Inhibit in all examples.

After a few tests with chips from different manufacturers, I found the following code works with all.

bitbang_cp_low();  __delay_us(1);  // bring clock down
bitbang_pl_low();  __delay_us(1);  // perform Load
bitbang_pl_high(); __delay_us(1);  // load done

uint8_t value = !bitbang_so_get(); // get the first bit as that one is already latched
for (uint8_t i = 0; i < 7; i++) {  // do the next 7 bits
    bitbang_cp_high(); __delay_us(1);  // clock goes high
    value <<= 1;                       // shift value 1 bit to have empty bit for the next load
    if (!bitbang_so_get()) { value |= 0x01; }  // read serial output
    bitbang_cp_low(); __delay_us(1);  // clock goes low for the next round
}

return value;

One can argue that 1µs delay is a bit too much but I actually found a few chips obtained from eBay that positively needed it to function properly. One could argue that those chips should be excluded as all reliable manufacturers specify it at 250ns or less. However, quadrupling that value in my use case (reading DIP switch on startup) wasn’t that important.

Regardless, adjust overall code as needed.

Comic Code and the Panose Category

Most people overlooking my coding are puzzled. Hey, isn’t that Comic Sans? And no, I don’t code in Comic Sans. What kind of animal you take me for?!

I use Comic Code. It’s a monospace variant of Comic Sans. A completelly different beast. :)

Honestly, after moving through many monospaced fonts over the years, this is the one font I found most comfortable to use. And yes, it’s not free. However, if you get it directly from Toshi Omagan, you’ll probably get a decent discount; I know I did. And no, there is no free alternative that supports Unicode properly.

Once I got the font via email, I started using it in VS Code, Visual Studio, Notepad++, Putty… Wait! It doesn’t work in Putty? Yep. My font of choice was not in the list. Uff. I had to select support for variable pitch fonts. Well, no biggie. And then I tried to change it for my Windows Console only to see there was no way to set it at all.

In search for a solution, and after learning way more than I needed (darn Wikipedia is black hole), I zeroed onto Panose classification and its Proportion category. Yep, Windows uses this category to decide which fonts to show in its classic dialog box when application asks for a fixed-spacing.

Fortunately for me, there is an excellent tool called Panosifier that allows changing any Panose setting without going far into the binary. I just ran it with --proportion 9 argument and, voila, my Comic Code was now visible even in Windows Console.

If you too decide to use this beautiful font, don’t worry. Its author has adjusted Panose since so font you receive should have it already set.

CRC-16 Nibble Lookup Table

I already wrote about using smaller-than-256-byte lookup table. What we ended up with was a smaller 32 byte lookup setup for CRC-8. However, if you’re sending a lot of data, CRC-8 won’t do. Can one make a nibble lookup table for CRC-16 too?

Of course one can. It’s actually exactly the same - just expand all variables to 16 bytes. In order to calculate lookup table, one can just use this code:

for (uint8_t i = 0; i < 16; i++) {
    uint16_t crc = (uint16_t)i;
    for (uint8_t j = 1; j <= 16; j++) {
        if (crc & 0x8000) {
            crc = (uint16_t)((crc << 1) ^ poly);
        } else {
            crc <<= 1;
        }
    }
    Crc16LutL[i] = crc;
}
for (uint8_t i = 0; i < 16; i++) {
    uint16_t crc = (uint16_t)i << 4;
    for (uint8_t j = 1; j <= 16; j++) {
        if (crc & 0x8000) {
            crc = (uint16_t)((crc << 1) ^ poly);
        } else {
            crc <<= 1;
        }
    }
    Crc16LutH[i] = crc;
}

I will be using polynomial 0xA2EB (0xD175 in Koopman’s notation) since that one seems to be the best at this time. Mind you, in year from now, one might find another polynomial with better characteristics yet.

// Polynomial 0xA2EB
const uint16_t Crc16LutL[] = { 0x0000, 0xA2EB, 0xE73D, 0x45D6, 0x6C91, 0xCE7A, 0x8BAC, 0x2947, 0xD922, 0x7BC9, 0x3E1F, 0x9CF4, 0xB5B3, 0x1758, 0x528E, 0xF065, };
const uint16_t Crc16LutH[] = { 0x0000, 0x10AF, 0x215E, 0x31F1, 0x42BC, 0x5213, 0x63E2, 0x734D, 0x8578, 0x95D7, 0xA426, 0xB489, 0xC7C4, 0xD76B, 0xE69A, 0xF635, };

uint16_t crc16(uint8_t* data, uint8_t length) {
    uint16_t crc = 0;
    while (length--) {
        uint8_t combo = (crc >> 8) ^ *data++;
        crc = (crc << 8) ^ Crc16LutL[combo & 0x0F] ^ Crc16LutH[combo >> 4];
    }
    return crc;
}

These extra 64 bytes (two lookup tables, 32 bytes each) allows us to use shift-less CRC code. And yes, there will one shift operation in there but XC8 compiler should actually optimize that one away in most cases.

Since we do have a bit more complicated data handling, implemented on a microcontroller using XC8 compiler, this code needs 141 words of program memory (was 80 for CRC-8) and uses 16 bytes of RAM (was 4 for CRC-8). All in all, pretty handleable by almost any PIC microcontroller.

Interestingly, here you cannot use trick of removing const in order to move usage more toward memory. While RAM usage for such case indeed increases to 76 bytes, program memory usage doesn’t go down at all.

As always, the example code is available for download.

Mystery of a High ADC Reading

Microchip makes reading ADC easy enough. Select a channel, start measurement, read a registed - doesn’t get much easier than that. However, in one project of mine I got a bit stumped. While most of my readings were spot on, one was stubornly too high.

As I went over PIC16F15386 documentation a bit I saw a following note: “It is recommended that … the user selects Vss channel before connecting to the channel with the lower voltage.” Whether due to a high channel count of some other pecularity of this PIC, capacitance was strong with this one. One of the rare times when reading instructions actually solved an issue.

Well, solved might have been too optimistic of a statement. While my low voltage ADC channel now read a correct value, my higher voltage inputs read slightly too low. Yes, I am aware I sound like a daddy bear looking at his bed but I too wanted my readings to be just right and for the darn Goldilocks to leave my home.

What I found working for me is doing the following: switch channel to GND, (dummy) read ADC, switch to the correct channel, do the first (proper) reading, do the second reading, average the two readings. In code it would be something like this:

ADCON0bits.CHS = 0b011011;    // select Vss channel
ADCON0bits.GO_nDONE = 1;      // start an A/D conversion cycle
while (ADCON0bits.GO_nDONE);  // wait for conversion to complete

ADCON0bits.CHS = channel;     // select channel

ADCON0bits.GO_nDONE = 1;      // start an A/D conversion cycle
while (ADCON0bits.GO_nDONE);  // wait for conversion to complete
uint16_t value1 = ADRES;      // read value

ADCON0bits.GO_nDONE = 1;      // start an A/D conversion cycle
while (ADCON0bits.GO_nDONE);  // wait for conversion to complete
uint16_t value2 = ADRES;      // read value

ADCON0bits.CHS = 0b011011;    // select Vss channel

return (value1 + value2 + 1) / 2;

Now, the obvious issue here is that three readings are done when only one is needed. Since we want to do averaging, there is nothing that can be done about reading it twice. However, if you are not pooling it all the time, you can pretty much skip the first (dummy) reading as switching to Vss channel at the end of routine does the trick.

While 16-bit operations are not the most pleasant thing 8-bit PIC can do, it’s actually not too bad as we’re talking about only 3 additions and one right shift (division by 2 gets optimized). Not good, but not terrible.

Even better, this works flawlessly with ADCs that have no issues with reading being too high or too low. This means I don’t need to worry if I use routine with a different PIC microcontroller.

All in all, it’s a cheap way to get a correct reading.