Testing Native ZFS Encryption Speed (Ubuntu 22.04)

[2022-10-30: There is a newer version of this post]

With the new Ubuntu LTS release, it came time to repeat my ZFS encryption testing. Is ZFS speed better, worse, or the same?

I won’t go into the test procedure much since I explained it back when I did it the first time. Outside of really minor differences in the exact disk size, procedure didn’t change. What did change is that I am not doing it on virtual machine anymore.

These tests I did on Framework laptop with i5-1135G7 processor and 32GB of RAM. It’s a bit more consistent setup than the virtual machine I used before. Due to this change, numbers are not really comparable to ones from previous tests but that should be fine - our main interest is in the relative numbers.

First of all, we can see that CCM encryption is not worth a dime if you have any AES-capable processor. Difference between CCM and any other encryption I tested is huge with CCM being 5-6 times slower. Only once I turned off the AES support in BIOS does its inclusion make even a minimal sense as this actually improves its performance. And no, it doesn’t suck less - it’s just that all other encryption methods suck more.

Assuming our machine has a processor made in the last 5 or so years, the native ZFS GCM encryption becomes the clear winner. Yes, 128-bit variant is a bit faster than 256-bit one (as expected) but difference is small enough that it probably wont matter. What will matter is that any GCM wins over LUKS. Yes, reads are slightly faster using standard XTS LUKS but writes are clearly favoring the native ZFS encryption.

Unless you really need the ultimate cryptographic opacity a LUKS encryption brings, a native ZFS encryption using GCM is still a way to go. And yes, even though GCM modes are performant, we still lose about 10-15% in writes and about 30% on reads when compared to no encryption at all. Mind you, as with all synthetic tests giving you the worst figures, the real performance loss is much lower.

Make what you want of it, but I’ll keep encrypting my drives. They’re plenty fast.

PS: You can take a peek at the raw data if you’re so inclined.

Ubuntu 22.04 on Surface Go

With the new LTS Ubuntu just round the corner, it came time to refresh my Surface Go operating system. For this guide I will assume you have the Windows currently running so you can write Ubuntu 22.04 image using Rufus utility. Make sure to use GPT partition scheme targeting UEFI systems.

First we need to partition disk and install Linux for which we have to boot from Ubuntu USB drive. To do this go to Recovery Options and select Restart now. From the boot menu then select Use a device and finally use Linpus lite. If you are using Ubuntu, there is no need to disable secure boot or meddle with USB boot order as 22.04 fully supports secure boot.

While you could proceed from here with normal Ubuntu install, I like a bit more involved process that includes a bit of command line. Since we need root prompt, we should open Terminal and get those root credentials going.

sudo -i

The very next step should be setting up a few variables - host, user name, and disk(s). This way we can use them going forward and avoid accidental mistakes.

HOST=^^desktop^^
USER=^^user^^
DISK=/dev/mmcblk0

Disk setup is really minimal. Notice that both boot and EFI partition will need to be on internal disk as BIOS doesn’t know how to boot from micro-SD card.

blkdiscard -f $DISK
sgdisk --zap-all                       $DISK
sgdisk -n1:1M:+63M -t1:EF00 -c1:EFI    $DISK
sgdisk -n2:0:+640M -t2:8300 -c2:Boot   $DISK
sgdisk -n3:0:0     -t3:8309 -c3:Ubuntu $DISK
sgdisk --print                         $DISK

I usually encrypt just the root partition as having boot partition unencrypted does offer advantages and having standard kernels exposed is not much of a security issue.

cryptsetup luksFormat -q --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf pbkdf2 --hash sha256 ${DISK}p3

Since crypt device name is displayed on every startup, for Surface Go I like to use host name here.

cryptsetup luksOpen ${DISK}p3 ${HOST^}

At last we can prepare all needed partitions.

yes | mkfs.ext4 /dev/mapper/${HOST^}
mkdir /mnt/install
mount /dev/mapper/${HOST^} /mnt/install/

yes | mkfs.ext4 ${DISK}p2
mkdir /mnt/install/boot
mount ${DISK}p2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI -i 4d65646f ${DISK}p1
mkdir /mnt/install/boot/efi
mount ${DISK}p1 /mnt/install/boot/efi

To start the fun we need debootstrap package. Do make sure you have Wireless network connected at this time as otherwise operation will not succeed.

apt update ; apt install --yes debootstrap

And then we can get basic OS on the disk. This will take a while.

debootstrap $(basename `ls -d /cdrom/dists/*/ | head -1`) /mnt/install/

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials:

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/
``

Finally we're ready to "chroot" into our new system.

```sh
mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env HOST=$HOST USER=$USER DISK=$DISK \
    bash --login

For new system we need to setup the locale and the time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes initramfs-tools cryptsetup keyutils grub-efi-amd64-signed shim-signed

Since we’re dealing with encrypted data, we should auto mount it via crypttab. If there are multiple encrypted drives or partitions, keyscript really comes in handy to open them all with the same password. As it doesn’t have negative consequences, I just add it even for a single disk setup.

echo "${HOST^} UUID=$(blkid -s UUID -o value ${DISK}p3)  none \
    luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
cat /etc/crypttab

To mount boot and EFI partition, we need to do some fstab setup too:

echo "UUID=$(blkid -s UUID -o value /dev/mapper/${HOST^}) \
    / ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value ${DISK}p2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value ${DISK}p1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
    mem_sleep_default=deep\"/" /etc/default/grub
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

apt install --yes ubuntu-desktop-minimal firefox

Short package upgrade will not hurt.

add-apt-repository universe
apt update ; apt dist-upgrade --yes

The only remaining task before restart is to create the user, assign a few extra groups to it, and make sure its home has correct owner.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
echo "$USER ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USER
passwd $USER

As install is ready, we can exit our chroot environment.

exit

And unmount our disk:

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}

After the reboot you should be able to enjoy your Ubuntu installation.

reboot

If all went fine, congratulations, you have your Ubuntu up and running.

My ZFS Settings

In multiple posts so far I’ve created a ZFS pool using pretty much the same parameters. But I never bothered to explain why I chose them. Until now…

From my latest ZFS-related post, I have the following pool creation command:

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
    -O canmount=off -O mountpoint=none $POOL $DISK

ashift=12

This setting controls the block size of your pool and should match whatever your (spinning) disk uses. Realistically, you’ll probably use 4K sectors thus 12 is a good starting value. Why the heck 12? Well, this is expressed as 2ⁿ and 2¹² is 4 KB. I like to force it because often ZFS might wrongly auto-detect value 9 (512 bytes) which shouldn’t be really used these days. This is not really ZFS’ fault but consequence of some disks being darn liars to preserve compatibility.

Even if you do have 512-byte disks today, any replacement down the road will be at least 4K. Since the only way to change this option is to recreate the pool one should think ahead and go with 4K immediately.

When it comes to SSD setups there might be some benefit in going even higher since SSD usually use 8K or even larger erase blocks. However, since SSDs are much more forgiving when it comes to the random access, most of time it’s simply not worth it because large block sizes will cause other issues (e.g., slack space).

autotrim=on

Support for trim is really important for SSD and completely irrelevant when it comes to the spinning rust. Since my NAS uses good-old hard drives, this setting really doesn’t apply. But I also use ZFS on my laptop and there it makes a huge difference. So I include it always just not to forget it by accident when it matters.

compression=lz4

While zstd seems to be a compression darling, I still prefer lz4 for my local datasets because it’s much easier on the CPU. There’s also an option to turn off compression completely, but I honestly cannot determine any speed improvement in a general case. Using compression is like receiving free space, so why not?

normalization=formD

As ZFS uses Unicode (UTF-8 more specifically), it has an interesting problem that two filenames might look the same but they might have two different expressions. Most known example might be Å which can be expressed either as Å or as combination of A and a separate ring mark. From the point of user, both these are the same. But they have a different binary expression (U+00C5 vs U0041 U+030A).

Setting normalization explicitly just ensures each file name is stored in its canonical Unicode representation and thus things that look the same are going to be the same. I personally like formD on a philosophical level but any normalization will do the same. Just don’t stick with default value of none.

acltype=posixacl

This option allows you to store extra access attributes not covered by a “standard” user/group/world affair. The most common need for these attributes is with SELinux. However, even if you’re not using SELinux, you should enable it as it doesn’t really impact anything if not used. And you might consider using SELinux in the future.

xattr=sa

This option will tell ZFS to store extra access attributes (see above) with the metadata. This is a huge performance boost if you use them. If you don’t use them it has no effect so you might as well future-proof your setup.

dnodesize=auto

Assuming you already save all these extra attributes, it’s obvious they cannot really fit nicely in one metadata node. Unless it’s a big one. Once set, this option (assuming feature@large_dnode=enabled) will allow larger than normal metadata at the cost of some compatibility. Assuming you have ZFS 0.8.4 or above, you really have nothing to worry about.

atime=off

Posix standard specifies that one should always update access time whenever file or directory is accessed. You went into your home directory - update. You opened a file without changing anything - update. These darn updates really stack up and there is really no general use case where you would need to know when the file was read. This flag will turn off these updates.

encryption=aes-256-gcm

I like my datasets encrypted. Ideally one would use full disk encryption but using ZFS native encryption is a close second with unique benefits at a cost of minor data leaks (essentially only ZFS dataset names). And GCM encryption is usually the fastest here.

keyformat=passphrase

Call me old-fashioned but I prefer a passphrase to a binary key. Reason is that I can enter passphrase more easily in a pinch.

keylocation=prompt

For my laptop I keep prompt as a key source so I can easily type it. For servers, I use file:// syntax here since I keep my passphrase on a TmpUsb USB drive. This allows me to reboot server without entering key every time but in the case it’s ever stolen my data is inaccessible.

canmount=off, mountpoint=none

As a rule, I try not to have top-level dataset mountable. I just use it to set defaults and data goes only in sub-datasets.

And that’s all the explanation I’m ready to offer.

Installing UEFI ZFS Root on Ubuntu 21.10

Before reading further you should know that Ubuntu has a ZFS setup option since 19.10. You should use it instead of the manual installation procedure unless you need something special. In my case that special something is the native ZFS encryption and custom partitioning I find more suitable for laptop.

After booting into Ubuntu desktop installation we want to get a root prompt. All further commands are going to need root credentials anyhow.

sudo -i

The very first step should be setting up a few variables - disk, pool, host name, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system. I like to use upper-case for ZFS pool as that’s what will appear as password prompt. It just looks nicer and ZFS doesn’t care either way.

DISK=/dev/disk/by-id/^^ata_disk^^
POOL=^^Ubuntu^^
HOST=^^desktop^^
USER=^^user^^

General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are not planning to have multiple kernels, decreasing boot partition size might be a good idea (512 MB is ok).

blkdiscard -f $DISK

sgdisk --zap-all                     $DISK

sgdisk -n1:1M:+63M -t1:EF00 -c1:EFI  $DISK
sgdisk -n2:0:+960M -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:0     -t3:BF00 -c3:Root $DISK

sgdisk --print                       $DISK

Finally we’re ready to create system ZFS pool. Note that you need to encrypt it at the moment it’s created.

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
    -O canmount=off -O mountpoint=none $POOL $DISK-part3

On top of this encrypted pool we could create our root set (as I did in previous guides) or just use default dataset itself. I found that actually works better for me. In either case, our new root file system will end up at /mnt/install.

zfs set canmount=noauto $POOL
zfs set mountpoint=/ $POOL

mkdir /mnt/install
mount -t zfs -o zfsutil $POOL /mnt/install

Assuming UEFI boot, two additional partitions are needed. One for EFI and one for booting. Unlike what you get with the official guide, here I don’t have ZFS pool for boot partition but a plain old ext4. I find potential fixup works better that way and there is a better boot compatibility. If you are thinking about mirroring, making it bigger and ZFS might be a good idea. For a single disk, ext4 will do.

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

To start the fun we need debootstrap package.

apt install --yes debootstrap

Bootstrapping Ubuntu on the newly created pool is next. This will take a while.

debootstrap $(basename `ls -d /cdrom/dists/*/ | grep -v stable | head -1`) /mnt/install/

We can use our live system to update a few files on our new installation.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials. Don’t worry if this returns errors - that just means you are not using wireless.

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK POOL=$POOL USER=$USER \
    bash --login

Let’s not forget to setup locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes zfs-initramfs grub-efi-amd64-signed shim-signed tasksel

To mount boot and EFI partition, we need to do some fstab setup.

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we get grub started and update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

tasksel install ubuntu-desktop-minimal

Short package upgrade will not hurt.

apt dist-upgrade --yes

We can omit creation of the swap dataset but I personally find a small one handy.

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
    -o sync=always -o primarycache=metadata -o secondarycache=none $POOL/Swap
mkswap -f /dev/zvol/$POOL/Swap
echo "/dev/zvol/$POOL/Swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

If one is so inclined, /home directory can get a separate dataset too but I usually skip it these days. I just proceed to create the user, assign a few extra groups to it, and make sure its home has correct owner.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
passwd $USER

As install is ready, we can exit our chroot environment.

exit

And cleanup our mount points.

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
umount /mnt/install
zpool export -a

After the reboot you should be able to enjoy your installation.

reboot

PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 22.04 and 20.04

PPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, 19.04, and 18.10.

ZFS Pool on SSD

I am a creature of habit. Long time ago I found ZFS setup that works for me and didn’t change much since. But sometime I wonder if those settings still hold with SSDs in game. Most notably, are 4K blocks still the best?

Since I already “had” to update my desktop to Ubuntu 21.10, I used that opportunity to clear my disks and have it installed from scratch. And it would be a shame not to run some tests first on my XPG SX6000 Pro - SSD I use for pure data storage. After trimming this DRAM-less SSD, I tested the pool across multiple recordsize values and at ashift values of 12 (4K block) and 13 (8K block).

My goal was finding a good default settings for both bulk storage and virtual machines. Unfortunately, those are quite opposite requirements. Bulk storage benefits greatly from good sequential access while virtual machines love random IO more. Fortunately, with ZFS, one can accomplish both using two datasets with different recordsize values. But ashift value has to be the same.

Due to erase block sizes getting larger and larger, I expected performance to be better with 8K “sectors” (ashift=13) than what I usually used (ashift=12). But I was surprised.

First of all, results were all over the place but it seems that ashift=12 is still a valid starting point. It might be due to my SSD having smaller than expected erase page but I doubt it. My thoughts go more toward SSDs being optimized for the 4K load. And the specific SSD I used to test with is DRAM-less thus allowing any such optimizations to be even more visible.

Optimizations are probably also the reason for 128K performing so well in the random IO scenarios. Yes, for sequential access you would expect it, but for random access it makes no sense how fast it is. No matter what’s happening, it’s definitely making recordsize=128K still the best general choice. Regardless, for VMs, I created a sub-dataset with much smaller 4K records (and compression off) just to lower write-amplification a bit.

The full test results are in Google Sheets. For testing I used fio’s fio-rand-RW.fio and fio-seq-RW.fio profiles.

Zip in Git Bash

While creating build system that works across the platforms, one can find issues in the most basic things. And that’s even when shell is the same. For example, while Bash on Linux and Windows works out the same, a lot of supporting tools differ - a lot. And there’s no better example than creating a zip archive.

Under Linux you can count on zip command being available. Even if one doesn’t have it, it’s easy to install without messing with their desktop. On Windows story gets more complicated. Git Bash for example doesn’t have it even compiled and there’s no really good way to add it. Yes, you can use any application but different one is installed on every system. To create more “fun”, supporting multiple applications also means dealing with their command-line arguments. And yes, 7-Zip has completely different syntax as compared to WinRAR.

However, when it comes to making zip archive, there’s actually a solution that works for both Windows (via Git Bash) and Linux. Surprisingly, the answer is perl.

If one is careful to use Perl’s older IO::Compress::Zip library, creating an archive becomes a simple task:

perl -e '
  use strict;
  use warnings;
  use autodie;
  use IO::Compress::Zip qw(:all);
  zip [
    "src/mimetype",
    <"src/META-INF/*.*">,
    <"src/OEBPS/*.*">,
    <"src/OEBPS/chapters/*.*">
  ] => "bin/book.epub",
       FilterName => sub { s[^src/][] },
       Zip64 => 0,
  or die "Zip failed: $ZipError\n";
'

Yeah, might not be ideal when it comes to beauty but it definitely works across platforms.

Prepping Image for ESXi

I like using vboxmanage for disk conversions. When dealing with major formats it often can do everything I need. For example, if I wanted to convert raw disk image to .vmdk, it’s easy:

vboxmanage convertdd in.raw --format VMDK out.vmdk

However, sometime this simple tool is too simple. For example, using that image with ESXi, any modern version will just give you “Not a supported disk format (sparse VMDK version too old)”.

But it’s not like vboxmanage is the only game in town. For example one can use qemu-img.

qemu-img convert -f raw -O vmdk in.raw out.vmdk

Different tool, same error.

For ESXi to work, we need to tweak options a bit.

qemu-img convert -f raw -O vmdk \\ -o adapter\_type=lsilogic,subformat=streamOptimized,compat6 \\ in.raw out.vmdk

And this one does the trick.

Case of the Incorrect USB VID

While manually executing script to reflash Sierra EM7455 (since automated steps didn’t work), I accidentally skipped a command - one to change USB VID. For those not familiar with USB, there are two numbers - VID and PID - combination of which allows for system to recognize your device. If you change one and not the other, your system will not recognize device anymore.

In my case, I have created a device 413c:9070. It was a monster with VID belonging to Dell and PID of a generic Sierra device. Combination that’s definitely not recognized. And I couldn’t correct it because that would require serial connection to the device. Serial connection I didn’t have due to the device not being recognized. In Windows you’re officially screwed. But Ubuntu comes to rescue.

In Linux it’s trivial to convince system your “hybrid” device is a valid one. Just load option module and tell your serial driver there’s a new kid in town. In my case, this looked something like this:

sudo modprobe option
echo 413c 9071 | sudo tee /sys/bus/usb-serial/drivers/option1/new_id

With this I got my device’s connection back at ttyUSB2 and could use minicom to manually change VID:

sudo minicom -b 115200 -D /dev/ttyUSB2
 AT!USBVID=1199

Lenovo P70 Fingerprint Reader Under Ubuntu

Fingerprint support under Linux is spotty at best. For many laptops one will never be able to use it. Fortunately, somebody went through the effort to make drivers for VFS7500 Touch Fingerprint Sensor used on P70.

Unfortunately instructions to install it on Ubuntu 20.04 are not really complete so I here is what worked for me. Newer Ubuntu steps should be reasonably similar if not same but I haven’t checked them from scratch.

First I installed a few packages. Since drivers use combination of snap and python, we obviously need them. GMP library is needed to as it’s dependency of pyfast library.

sudo apt-get install --yes snapd python3-pip libgmp-dev
sudo pip install pyusb pycrypto pyfast

Then we can follow original install procedure:

sudo snap install validity-sensors-tools
sudo snap connect validity-sensors-tools:raw-usb
sudo snap connect validity-sensors-tools:hardware-observe

With all in place we can finally initialize reader:

sudo validity-sensors-tools.initializer

Lastly, install libfprint TOD driver.

sudo add-apt-repository -u ppa:3v1n0/libfprint-vfs0090
sudo apt-get install --yes libfprint-2-tod-vfs0090

Now you should see Fingerprint Login in system’s Users applet. Once you enroll your one or more finger, you should be able to use it for login.

Manually Installing Ubuntu 21.04 on Surface Go

Now, one can install Ubuntu perfectly well onto Surface Go without any shenanigans. Just follow a guide on how to boot install USB and you’re golden. But I like my installations to be a bit special. :)

After booting into Ubuntu desktop installation one needs a root prompt. All further commands are going to need root credentials anyhow.

$ sudo -i

The very first step should be setting up a few variables - disk, host, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.

DISK=/dev/disk/by-id/^^ata_disk^^
HOST=^^desktop^^
USER=^^user^^

Disk setup is really minimal. Please note you can actually reduce size of boot partition but that might get you in trouble if you start playing with low latency kernel. Some extra space will help here.

blkdiscard $DISK

sgdisk --zap-all                       $DISK
sgdisk -n1:1M:+47M -t1:EF00 -c1:EFI    $DISK
sgdisk -n2:0:+720M -t2:8300 -c2:Boot   $DISK
sgdisk -n3:0:0     -t3:8309 -c3:Ubuntu $DISK

sgdisk --print                         $DISK

I usually encrypt just the root partition as having boot partition unencrypted does offer advantages and having standard kernels exposed is not much of a security issue.

cryptsetup luksFormat -q --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf pbkdf2 --hash sha256 $DISK-part3

Since crypt device name is displayed on every startup, for Surface Go I like to use host name here.

cryptsetup luksOpen $DISK-part3 ${HOST^}

Now we can prepare all needed partitions.

yes | mkfs.ext4 /dev/mapper/${HOST^}
mkdir /mnt/install
mount /dev/mapper/${HOST^} /mnt/install/

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

To start the fun we need debootstrap package.

apt update ; apt install --yes debootstrap

And then we can get basic OS on the disk. This will take a while.

debootstrap $(basename `ls -d /cdrom/dists/*/ | head -1`) /mnt/install/

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials:

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK HOST=$HOST USER=$USER \
    bash --login

Let’s not forget to setup locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes initramfs-tools cryptsetup keyutils grub-efi-amd64-signed shim-signed tasksel

Since we’re dealing with encrypted data, we should auto mount it via crypttab. If there are multiple encrypted drives or partitions, keyscript really comes in handy to open them all with the same password. As it doesn’t have negative consequences, I just add it even for a single disk setup.

echo "${HOST^} UUID=$(blkid -s UUID -o value $DISK-part3)  none \
    luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
cat /etc/crypttab

To mount boot and EFI partition, we need to do some fstab setup too:

echo "UUID=$(blkid -s UUID -o value /dev/mapper/${HOST^}) \
    / ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
    mem_sleep_default=deep\"/" /etc/default/grub
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

tasksel install ubuntu-desktop-minimal

Short package upgrade will not hurt.

add-apt-repository universe
apt update ; apt dist-upgrade --yes

The only remaining task before restart is to create the user, assign a few extra groups to it, and make sure its home has correct owner.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
echo "$USER ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USER
passwd $USER

Before finishing it up, I like to install Surface Go WiFi and backlight tracer packages. This will allow for usage of wireless once we boot into installed system and for remembering light level between plugged/unplugged states.

wget -O /tmp/surface-go-wifi_amd64.deb \
    https://www.medo64.com/download/surface-go-wifi_0.0.5_amd64.deb
apt install --yes /tmp/surface-go-wifi_amd64.deb

wget -O /tmp/backlight-tracer_amd64.deb \
    https://www.medo64.com/download/backlight-tracer_0.1.1_all.deb
apt install --yes /tmp/backlight-tracer_amd64.deb

As install is ready, we can exit our chroot environment.

exit

And unmount our disk:

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}

After the reboot you should be able to enjoy your installation.

reboot

Once booted I like to setup suspend to react on power button and and to disable automatic brightness changes.

gsettings set org.gnome.settings-daemon.plugins.power button-power 'suspend'
gsettings set org.gnome.settings-daemon.plugins.power power-button-action 'suspend'
gsettings set org.gnome.settings-daemon.plugins.power ambient-enabled 'false'
gsettings set org.gnome.mutter experimental-features "['x11-randr-fractional-scaling']"

My preferred scale factor is 150% (instead of default 200%) but you’ll need to change that in settings manually.