Installing UEFI ZFS Root on Ubuntu 20.04 (with Native Encryption)

There is a newer version of this guide for Ubuntu 21.10.

Technically, I already have a guide for encrypted ZFS setup on Ubuntu 20.04. However, that guide used Geli and, as correctly one reader noted in comments (thanks Alex!), there was no reason not to use ZFS’ native encryption. So, here is adjusted variant of my setup.

First of all, Ubuntu 20.04 has a ZFS setup option as of 19.10. You should use it instead of the manual installation procedure unless you need something special. Namely, manual installation allows for encryption, in addition to the custom pool layout and naming. You should also check the great Root on ZFS installation guide that’s part of ZFS-on-Linux project for a full picture. I find its final ZFS layout a bit too complicated for my taste but there is a lot of interesting tidbits on that page. Here is my somewhat simplified version of the same, intended for a singe disk installation.

After booting into Ubuntu desktop installation we want to get a root prompt. All further commands are going to need root credentials anyhow.

sudo -i

The very first step should be setting up a few variables - disk, pool, host name, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.

DISK=/dev/disk/by-id/^^ata_disk^^
POOL=^^ubuntu^^
HOST=^^desktop^^
USER=^^user^^

General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are planning to have multiple kernels, increasing boot partition size might be a good idea.

blkdiscard $DISK

sgdisk --zap-all                        $DISK

sgdisk -n1:1M:+127M -t1:EF00 -c1:EFI    $DISK
sgdisk -n2:0:+512M  -t2:8300 -c2:Boot   $DISK
sgdisk -n3:0:0      -t3:8309 -c3:Ubuntu $DISK

sgdisk --print                          $DISK

Finally we’re ready to create system ZFS pool. Note that you need to encrypt it at the moment it’s created.

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
    -O canmount=off -O mountpoint=none -R /mnt/install $POOL $DISK-part3

On top of this encrypted pool, we can create our root dataset.

zfs create -o canmount=noauto -o mountpoint=/ $POOL/root
zfs mount $POOL/root

Assuming UEFI boot, two additional partitions are needed. One for EFI and one for booting. Unlike what you get with the official guide, here I don’t have ZFS pool for boot partition but a plain old ext4. I find potential fixup works better that way and there is a better boot compatibility. If you are thinking about mirroring, making it bigger and ZFS might be a good idea. For a single disk, ext4 will do.

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

To start the fun we need debootstrap package.

apt install --yes debootstrap

Bootstrapping Ubuntu on the newly created pool is next. This will take a while.

debootstrap focal /mnt/install/

zfs set devices=off $POOL

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials. Don’t worry if this returns errors - that just means you are not using wireless.

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK POOL=$POOL USER=$USER \
    bash --login

Let’s not forget to setup locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes zfs-initramfs grub-efi-amd64-signed shim-signed tasksel

To mount boot and EFI partition, we need to do some fstab setup.

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we get grub started and update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

tasksel install ubuntu-desktop-minimal

Short package upgrade will not hurt.

apt dist-upgrade --yes

We can omit creation of the swap dataset but I personally find a small one handy.

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
    -o sync=always -o primarycache=metadata -o secondarycache=none $POOL/swap
mkswap -f /dev/zvol/$POOL/swap
echo "/dev/zvol/$POOL/swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

If one is so inclined, /home directory can get a separate dataset too.

rmdir /home
zfs create -o mountpoint=/home $POOL/home

The only remaining task before restart is to create the user, assign a few extra groups to it, and make sure its home has correct owner.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
passwd $USER

As install is ready, we can exit our chroot environment.

exit

And cleanup our mount points.

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

After the reboot you should be able to enjoy your installation.

reboot

PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 22.04 and 20.04

PPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, 19.04, and 18.10.

[2020-06-27: Added blkdiscard and autotrim.]

Fixing Git Author Name and Email

After moving Mercurial repository to Git you might want to update user names and emails.

The first step would be to see the all the names:

git log --format='%an <%ae>' | git log --format='%cn <%ce>' | sort | uniq

With this information in hand, we can adjust names with filter-branch:

git filter-branch --force --commit-filter '
    OLD_NAME="^^unknown^^"
    NEW_NAME="^^My name^^"
    NEW_EMAIL="^^myemail@example.com^^"
    if [ "$GIT_AUTHOR_NAME" = "$OLD_NAME" ]; then
        GIT_AUTHOR_NAME="$NEW_NAME"
        GIT_AUTHOR_EMAIL="$NEW_EMAIL"
    fi
    if [ "$GIT_COMMITTER_NAME" = "$OLD_NAME" ]; then
        GIT_COMMITTER_NAME="$NEW_NAME"
        GIT_COMMITTER_EMAIL="$NEW_EMAIL"
    fi
    git commit-tree "$@";
' --tag-name-filter cat -- --branches --tags --all

git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin

Noisy Microphone on Lenovo P70 Under Linux

I use Lenovo P70 as my Ubuntu workstation and most of the time I am reasonably ok with it. However, join a call and my microphone is either too low or there is way too much background noise. And that’s whether I use internal microphone or headset.

I tried playing with the slider but that’s annoying beyond belief. Sweet spot seems to be in bottom fifth but there is a really fine line between making background noise stop and my voice inaudible. Since there is no numerical value next to slider, eyeing out the perfect level is close to impossible.

As it often happens in Linux, solution comes courtesy of command line.

amixer set Capture 80%

I found that 80% level is where background noise becomes drastically less noticeable while still allowing my voice to come through.

PS: Interestingly, 100% level from command line puts slider at 1/5th of the sale. That means that (over)amplification can get up to 500%. A bit much…

Easy Type-C, Finally

It has been a while since USB Type-C became popular but I still avoided it for my hobby designs. It wasn’t due to its performance, oh no - performance was much better than what I needed. You see, I still use USB 2.0 for literally all my electronics projects. While not fast in the grand scheme of USB, it’s plenty fast for serial communication and that’s what I end up designing my gadgets around 90% of the time. My issues with the connector were due to soldering. Hand-soldering, to be precise.

Most of the type-C receptacle designs are of surface-mount kind. On its own that’s not an issue - I’ve hand-soldered my share of finicky narrow pitch ICs. Issue is the location of 24 pins that are needed. While 12 pins are almost reachable by a thin soldering iron, the other 12 pins are usually under the device. Not an issue if you are making them professionally or even in an oven but impossibility for soldering by hand.

And no, you cannot solder just one side of connector and be compliant. It doesn’t matter if you’re only interested in USB 2.0 as I was - you still need to connect both sides as you need to connect 5.1K resistor for device to be properly recognized. And yes, even if you fail to do that it will work with the type-A to type-C cable. It will sometimes work even with the type-C to type-C. However, things that happen only sometimes have a nasty habit of happening at the most unfortunate time.

Connectors that used a combination of through-hole design and surface-mount pads looked promising on the paper. The unreachable pads underneath were converted to pins and soldering problem got solved. Unfortunately, a lot of such connectors had a shield over the rear pads. Now rear pads became a problem. I did find a few with more open space that were easier to hand-solder but at a pricey $3.

But finally, I found the type-C connector that suits my use case perfectly - USB4085-GF-A. It’s an USB 2.0 connector that allows connection to 16 out of 24 type-C pins. The sacrificed pins mean that you won’t ever be able to use it with the high-speed devices (thus 2.0 moniker). However, the pins that remain include not only all the standard USB 2.0 power and data connections but also CC1 and CC2. These two pins are really important if you want your device to be part of the type-C universe and pull 1.5 A of current. The only thing needed in addition to the old type-A designs is two 5.1K pull-down resistors on each of those pins and you’re golden.

Mind you, the connector is not perfect. At $1.50 it’s a bit pricey but not excessively so. If you are using the standard 1.6 mm PCB, you will need to take a great care to position it flush in order for short pins to even reach to the other side. Hole spacing means you will need to use 6/6 PCB manufacturing which, while often supported, is still a bit less common than bog-standard 8/8. But these are all minor issues and hand-soldering is a breeze.

And despite all this goodness, I only used it for a few test contraptions and not in any of my proper projects. This is due to its biggest drawback - there is only one manufacturer and, outside of my initial purchase, they are pretty much on backorder the whole time. Even at the time of this writing, the quantity is 0 on DigiKey.

Once this connector becomes more available, I am looking forward to using it.

PS: For those wanting to have type-C plug directly on the board, there is a Molex 105444-0001. Just don’t forget to order 0.80 mm PCB.

PPS: For the 90° type-C connector, KUSBX-SL-CS1N14-B looks promising as it also allows connection to both CC1 and CC2 - a necessity for the proper type-C device. I haven’t tried it myself though.

Using TLS 1.3 from .NET 4.0 Application

Due to ubiquitous support for .NET 4.0 on all Windows platforms (including Windows 10) out-of-box, I still keep most of my freeware apps on it. Yes, I lose some new fancy features but not dealing with .NET Framework download makes it really convenient. But there is one thing that proved to be a problem - TLS 1.3.

When I changed my web server to use only TLS 1.2 and above, built-in upgrade suddenly stopped working. Digging a bit showed reason was that .NET 4.0 supported only TLS 1.0 by default and my web server where upgrades were located required TLS 1.2 at the minimum.

For the latest .NET versions, updating to a higher TLS is easy enough:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls13
                                     | SecurityProtocolType.Tls12
                                     | SecurityProtocolType.Tls11
                                     | SecurityProtocolType.Tls;

But Tls11, Tls12, and Tls13 enums are not part of .NET 4.0. However, because .NET is so closely integrated with Windows, sometime it’s worth asking it directly - by specifying enum’s numerical value:

ServicePointManager.SecurityProtocol = (SecurityProtocolType)12288
                                     | (SecurityProtocolType)3072
                                     | (SecurityProtocolType)768
                                     | SecurityProtocolType.Tls;

If you run this code before making the first HTTP request, suddenly you are not limited to the SSL and the ancient TLS anymore.

As this code still requires a bit of error checking, I finally ended up with the function below:

try { //try TLS 1.3
    ServicePointManager.SecurityProtocol = (SecurityProtocolType)12288
                                         | (SecurityProtocolType)3072
                                         | (SecurityProtocolType)768
                                         | SecurityProtocolType.Tls;
} catch (NotSupportedException) {
    try { //try TLS 1.2
        ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072
                                             | (SecurityProtocolType)768
                                             | SecurityProtocolType.Tls;
    } catch (NotSupportedException) {
        try { //try TLS 1.1
            ServicePointManager.SecurityProtocol = (SecurityProtocolType)768
                                                 | SecurityProtocolType.Tls;
        } catch (NotSupportedException) { //TLS 1.0
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
        }
    }
}

This code ensures the highest TLS is supported even from the poor old .NET 4.0.