Installing UEFI ZFS Root on Ubuntu 21.10

Before reading further you should know that Ubuntu has a ZFS setup option since 19.10. You should use it instead of the manual installation procedure unless you need something special. In my case that special something is the native ZFS encryption and custom partitioning I find more suitable for laptop.

After booting into Ubuntu desktop installation we want to get a root prompt. All further commands are going to need root credentials anyhow.

sudo -i

The very first step should be setting up a few variables - disk, pool, host name, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system. I like to use upper-case for ZFS pool as that’s what will appear as password prompt. It just looks nicer and ZFS doesn’t care either way.

DISK=/dev/disk/by-id/^^ata_disk^^
POOL=^^Ubuntu^^
HOST=^^desktop^^
USER=^^user^^

General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are not planning to have multiple kernels, decreasing boot partition size might be a good idea (512 MB is ok).

blkdiscard -f $DISK

sgdisk --zap-all                     $DISK

sgdisk -n1:1M:+63M -t1:EF00 -c1:EFI  $DISK
sgdisk -n2:0:+960M -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:0     -t3:BF00 -c3:Root $DISK

sgdisk --print                       $DISK

Finally we’re ready to create system ZFS pool. Note that you need to encrypt it at the moment it’s created.

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
    -O canmount=off -O mountpoint=none $POOL $DISK-part3

On top of this encrypted pool we could create our root set (as I did in previous guides) or just use default dataset itself. I found that actually works better for me. In either case, our new root file system will end up at /mnt/install.

zfs set canmount=noauto $POOL
zfs set mountpoint=/ $POOL

mkdir /mnt/install
mount -t zfs -o zfsutil $POOL /mnt/install

Assuming UEFI boot, two additional partitions are needed. One for EFI and one for booting. Unlike what you get with the official guide, here I don’t have ZFS pool for boot partition but a plain old ext4. I find potential fixup works better that way and there is a better boot compatibility. If you are thinking about mirroring, making it bigger and ZFS might be a good idea. For a single disk, ext4 will do.

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

To start the fun we need debootstrap package.

apt install --yes debootstrap

Bootstrapping Ubuntu on the newly created pool is next. This will take a while.

debootstrap $(basename `ls -d /cdrom/dists/*/ | grep -v stable | head -1`) /mnt/install/

We can use our live system to update a few files on our new installation.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials. Don’t worry if this returns errors - that just means you are not using wireless.

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK POOL=$POOL USER=$USER \
    bash --login

Let’s not forget to setup locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes zfs-initramfs grub-efi-amd64-signed shim-signed tasksel

To mount boot and EFI partition, we need to do some fstab setup.

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we get grub started and update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

tasksel install ubuntu-desktop-minimal

Short package upgrade will not hurt.

apt dist-upgrade --yes

We can omit creation of the swap dataset but I personally find a small one handy.

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
    -o sync=always -o primarycache=metadata -o secondarycache=none $POOL/Swap
mkswap -f /dev/zvol/$POOL/Swap
echo "/dev/zvol/$POOL/Swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

If one is so inclined, /home directory can get a separate dataset too but I usually skip it these days. I just proceed to create the user, assign a few extra groups to it, and make sure its home has correct owner.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
passwd $USER

As install is ready, we can exit our chroot environment.

exit

And cleanup our mount points.

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
umount /mnt/install
zpool export -a

After the reboot you should be able to enjoy your installation.

reboot

PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 22.04 and 20.04

PPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, 19.04, and 18.10.

ZFS Pool on SSD

I am a creature of habit. Long time ago I found ZFS setup that works for me and didn’t change much since. But sometime I wonder if those settings still hold with SSDs in game. Most notably, are 4K blocks still the best?

Since I already “had” to update my desktop to Ubuntu 21.10, I used that opportunity to clear my disks and have it installed from scratch. And it would be a shame not to run some tests first on my XPG SX6000 Pro - SSD I use for pure data storage. After trimming this DRAM-less SSD, I tested the pool across multiple recordsize values and at ashift values of 12 (4K block) and 13 (8K block).

My goal was finding a good default settings for both bulk storage and virtual machines. Unfortunately, those are quite opposite requirements. Bulk storage benefits greatly from good sequential access while virtual machines love random IO more. Fortunately, with ZFS, one can accomplish both using two datasets with different recordsize values. But ashift value has to be the same.

Due to erase block sizes getting larger and larger, I expected performance to be better with 8K “sectors” (ashift=13) than what I usually used (ashift=12). But I was surprised.

First of all, results were all over the place but it seems that ashift=12 is still a valid starting point. It might be due to my SSD having smaller than expected erase page but I doubt it. My thoughts go more toward SSDs being optimized for the 4K load. And the specific SSD I used to test with is DRAM-less thus allowing any such optimizations to be even more visible.

Optimizations are probably also the reason for 128K performing so well in the random IO scenarios. Yes, for sequential access you would expect it, but for random access it makes no sense how fast it is. No matter what’s happening, it’s definitely making recordsize=128K still the best general choice. Regardless, for VMs, I created a sub-dataset with much smaller 4K records (and compression off) just to lower write-amplification a bit.

The full test results are in Google Sheets. For testing I used fio’s fio-rand-RW.fio and fio-seq-RW.fio profiles.

Zip in Git Bash

While creating build system that works across the platforms, one can find issues in the most basic things. And that’s even when shell is the same. For example, while Bash on Linux and Windows works out the same, a lot of supporting tools differ - a lot. And there’s no better example than creating a zip archive.

Under Linux you can count on zip command being available. Even if one doesn’t have it, it’s easy to install without messing with their desktop. On Windows story gets more complicated. Git Bash for example doesn’t have it even compiled and there’s no really good way to add it. Yes, you can use any application but different one is installed on every system. To create more “fun”, supporting multiple applications also means dealing with their command-line arguments. And yes, 7-Zip has completely different syntax as compared to WinRAR.

However, when it comes to making zip archive, there’s actually a solution that works for both Windows (via Git Bash) and Linux. Surprisingly, the answer is perl.

If one is careful to use Perl’s older IO::Compress::Zip library, creating an archive becomes a simple task:

perl -e '
  use strict;
  use warnings;
  use autodie;
  use IO::Compress::Zip qw(:all);
  zip [
    "src/mimetype",
    <"src/META-INF/*.*">,
    <"src/OEBPS/*.*">,
    <"src/OEBPS/chapters/*.*">
  ] => "bin/book.epub",
       FilterName => sub { s[^src/][] },
       Zip64 => 0,
  or die "Zip failed: $ZipError\n";
'

Yeah, might not be ideal when it comes to beauty but it definitely works across platforms.

Prepping Image for ESXi

I like using vboxmanage for disk conversions. When dealing with major formats it often can do everything I need. For example, if I wanted to convert raw disk image to .vmdk, it’s easy:

vboxmanage convertdd in.raw --format VMDK out.vmdk

However, sometime this simple tool is too simple. For example, using that image with ESXi, any modern version will just give you “Not a supported disk format (sparse VMDK version too old)”.

But it’s not like vboxmanage is the only game in town. For example one can use qemu-img.

qemu-img convert -f raw -O vmdk in.raw out.vmdk

Different tool, same error.

For ESXi to work, we need to tweak options a bit.

qemu-img convert -f raw -O vmdk \\ -o adapter\_type=lsilogic,subformat=streamOptimized,compat6 \\ in.raw out.vmdk

And this one does the trick.

Case of the Incorrect USB VID

While manually executing script to reflash Sierra EM7455 (since automated steps didn’t work), I accidentally skipped a command - one to change USB VID. For those not familiar with USB, there are two numbers - VID and PID - combination of which allows for system to recognize your device. If you change one and not the other, your system will not recognize device anymore.

In my case, I have created a device 413c:9070. It was a monster with VID belonging to Dell and PID of a generic Sierra device. Combination that’s definitely not recognized. And I couldn’t correct it because that would require serial connection to the device. Serial connection I didn’t have due to the device not being recognized. In Windows you’re officially screwed. But Ubuntu comes to rescue.

In Linux it’s trivial to convince system your “hybrid” device is a valid one. Just load option module and tell your serial driver there’s a new kid in town. In my case, this looked something like this:

sudo modprobe option
echo 413c 9071 | sudo tee /sys/bus/usb-serial/drivers/option1/new_id

With this I got my device’s connection back at ttyUSB2 and could use minicom to manually change VID:

sudo minicom -b 115200 -D /dev/ttyUSB2
 AT!USBVID=1199

Lenovo P70 Fingerprint Reader Under Ubuntu

Fingerprint support under Linux is spotty at best. For many laptops one will never be able to use it. Fortunately, somebody went through the effort to make drivers for VFS7500 Touch Fingerprint Sensor used on P70.

Unfortunately instructions to install it on Ubuntu 20.04 are not really complete so I here is what worked for me. Newer Ubuntu steps should be reasonably similar if not same but I haven’t checked them from scratch.

First I installed a few packages. Since drivers use combination of snap and python, we obviously need them. GMP library is needed to as it’s dependency of pyfast library.

sudo apt-get install --yes snapd python3-pip libgmp-dev
sudo pip install pyusb pycrypto pyfast

Then we can follow original install procedure:

sudo snap install validity-sensors-tools
sudo snap connect validity-sensors-tools:raw-usb
sudo snap connect validity-sensors-tools:hardware-observe

With all in place we can finally initialize reader:

sudo validity-sensors-tools.initializer

Lastly, install libfprint TOD driver.

sudo add-apt-repository -u ppa:3v1n0/libfprint-vfs0090
sudo apt-get install --yes libfprint-2-tod-vfs0090

Now you should see Fingerprint Login in system’s Users applet. Once you enroll your one or more finger, you should be able to use it for login.

Manually Installing Ubuntu 21.04 on Surface Go

Now, one can install Ubuntu perfectly well onto Surface Go without any shenanigans. Just follow a guide on how to boot install USB and you’re golden. But I like my installations to be a bit special. :)

After booting into Ubuntu desktop installation one needs a root prompt. All further commands are going to need root credentials anyhow.

$ sudo -i

The very first step should be setting up a few variables - disk, host, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.

DISK=/dev/disk/by-id/^^ata_disk^^
HOST=^^desktop^^
USER=^^user^^

Disk setup is really minimal. Please note you can actually reduce size of boot partition but that might get you in trouble if you start playing with low latency kernel. Some extra space will help here.

blkdiscard $DISK

sgdisk --zap-all                       $DISK
sgdisk -n1:1M:+47M -t1:EF00 -c1:EFI    $DISK
sgdisk -n2:0:+720M -t2:8300 -c2:Boot   $DISK
sgdisk -n3:0:0     -t3:8309 -c3:Ubuntu $DISK

sgdisk --print                         $DISK

I usually encrypt just the root partition as having boot partition unencrypted does offer advantages and having standard kernels exposed is not much of a security issue.

cryptsetup luksFormat -q --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf pbkdf2 --hash sha256 $DISK-part3

Since crypt device name is displayed on every startup, for Surface Go I like to use host name here.

cryptsetup luksOpen $DISK-part3 ${HOST^}

Now we can prepare all needed partitions.

yes | mkfs.ext4 /dev/mapper/${HOST^}
mkdir /mnt/install
mount /dev/mapper/${HOST^} /mnt/install/

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

To start the fun we need debootstrap package.

apt update ; apt install --yes debootstrap

And then we can get basic OS on the disk. This will take a while.

debootstrap $(basename `ls -d /cdrom/dists/*/ | head -1`) /mnt/install/

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials:

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK HOST=$HOST USER=$USER \
    bash --login

Let’s not forget to setup locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes initramfs-tools cryptsetup keyutils grub-efi-amd64-signed shim-signed tasksel

Since we’re dealing with encrypted data, we should auto mount it via crypttab. If there are multiple encrypted drives or partitions, keyscript really comes in handy to open them all with the same password. As it doesn’t have negative consequences, I just add it even for a single disk setup.

echo "${HOST^} UUID=$(blkid -s UUID -o value $DISK-part3)  none \
    luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
cat /etc/crypttab

To mount boot and EFI partition, we need to do some fstab setup too:

echo "UUID=$(blkid -s UUID -o value /dev/mapper/${HOST^}) \
    / ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
    mem_sleep_default=deep\"/" /etc/default/grub
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

tasksel install ubuntu-desktop-minimal

Short package upgrade will not hurt.

add-apt-repository universe
apt update ; apt dist-upgrade --yes

The only remaining task before restart is to create the user, assign a few extra groups to it, and make sure its home has correct owner.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
echo "$USER ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USER
passwd $USER

Before finishing it up, I like to install Surface Go WiFi and backlight tracer packages. This will allow for usage of wireless once we boot into installed system and for remembering light level between plugged/unplugged states.

wget -O /tmp/surface-go-wifi_amd64.deb \
    https://www.medo64.com/download/surface-go-wifi_0.0.5_amd64.deb
apt install --yes /tmp/surface-go-wifi_amd64.deb

wget -O /tmp/backlight-tracer_amd64.deb \
    https://www.medo64.com/download/backlight-tracer_0.1.1_all.deb
apt install --yes /tmp/backlight-tracer_amd64.deb

As install is ready, we can exit our chroot environment.

exit

And unmount our disk:

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}

After the reboot you should be able to enjoy your installation.

reboot

Once booted I like to setup suspend to react on power button and and to disable automatic brightness changes.

gsettings set org.gnome.settings-daemon.plugins.power button-power 'suspend'
gsettings set org.gnome.settings-daemon.plugins.power power-button-action 'suspend'
gsettings set org.gnome.settings-daemon.plugins.power ambient-enabled 'false'
gsettings set org.gnome.mutter experimental-features "['x11-randr-fractional-scaling']"

My preferred scale factor is 150% (instead of default 200%) but you’ll need to change that in settings manually.

HexDump's Illegal Seek

After I upgraded to Ubuntu 21.04, my TmpUsb script suddenly started reporting the following hexdump: stdin: Illegal seek.

Line causing issue was the one determining partition serial number:

dd if=/dev/sda bs=512 skip=1 count=1 | hexdump -s39 -n4 -e '4/1 "%02X"'

It seems that hexdump got a bit too stricter with its input parameters and now disallows skipping bytes in fifo stream. I haven’t investigated much but my guess is that skipping 39 bytes probably messes with its internal buffer. In any case, dd has no such issues so the same code can be done without skipping in hexdump.

dd if=/dev/sda bs=1 skip=551 count=4 | hexdump -n4 -e '4/1 "%02X"'`

The best part is that this is compatible with older versions too.

Dazed and Confused, but Trying to Continue

From time to time, I would see the following slightly poetic statement on my console.

Uhhuh. NMI received for unknown reason 31 on CPU 3.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue

The first Internet search brought me sadness and dismay - my hardware was failing. It took going a bit deeper to find that AMD servers have that issue quite often even in the absence of real hardware failure.

Solution? Disable darn C-States. It’s a server after all.

PS: And no, even if you want to keep C-States, reason 31 is nothing to worry about - it’s been happening on my system for 2 years before this and I had no issues with it. It’s just annoyance and nothing more.

Recording Both Microphone and Speaker Under Ubuntu 21.04

When one records audio under Linux, issue that quite a few applications have is recording both microphone input and headphone output. And that’s true for SimpleScreenRecorder, otherwise really good recording application.

However, Linux always has a way around those restrictions and those can be actually found on SimpleScreenRecorder pages if you look deep enough.

pactl load-module module-null-sink \
  sink_name=duplex_out sink_properties=device.description="\"Duplex\ Output\""
pactl load-module module-null-sink \
  sink_name=app_out sink_properties=device.description="\"Application\ Output\""
pactl load-module module-loopback source=app_out.monitor
pactl load-module module-loopback source=app_out.monitor sink=duplex_out
pactl load-module module-loopback sink=duplex_out
pactl set-default-sink app_out

Trick is to essentially create two new output devices (i.e. sinks). One of them (app_out) will just be a target toward which applications should direct their output. Magic happens with the second output (duplex_out) which combines application output and what comes from microphone.

Now when you record audio, you can just point application to Duplex Output and record both sides.

PS: To make these changes permanent, they can be entered into /etc/pulse/default.pa. Of course, quoting rules are bit different so adjust accordingly if you have a space in your description.

…
# Recording setup
load-module module-null-sink sink_name=duplex_out sink_properties=device.description="Duplex\ Output"
load-module module-null-sink sink_name=app_out sink_properties=device.description="Application\ Output"
load-module module-loopback source=app_out.monitor
load-module module-loopback source=app_out.monitor sink=duplex_out
load-module module-loopback sink=duplex_out
set-default-sink app_out