Testing Native ZFS Encryption Speed (Ubuntu 22.04)

[2022-10-30: There is a newer version of this post]

Illustration

With the new Ubuntu LTS release, it came time to repeat my ZFS encryption testing. Is ZFS speed better, worse, or the same?

I won’t go into the test procedure much since I explained it back when I did it the first time. Outside of really minor differences in the exact disk size, procedure didn’t change. What did change is that I am not doing it on virtual machine anymore.

These tests I did on Framework laptop with i5-1135G7 processor and 32GB of RAM. It’s a bit more consistent setup than the virtual machine I used before. Due to this change, numbers are not really comparable to ones from previous tests but that should be fine - our main interest is in the relative numbers.

First of all, we can see that CCM encryption is not worth a dime if you have any AES-capable processor. Difference between CCM and any other encryption I tested is huge with CCM being 5-6 times slower. Only once I turned off the AES support in BIOS does its inclusion make even a minimal sense as this actually improves its performance. And no, it doesn’t suck less - it’s just that all other encryption methods suck more.

Assuming our machine has a processor made in the last 5 or so years, the native ZFS GCM encryption becomes the clear winner. Yes, 128-bit variant is a bit faster than 256-bit one (as expected) but difference is small enough that it probably wont matter. What will matter is that any GCM wins over LUKS. Yes, reads are slightly faster using standard XTS LUKS but writes are clearly favoring the native ZFS encryption.

Unless you really need the ultimate cryptographic opacity a LUKS encryption brings, a native ZFS encryption using GCM is still a way to go. And yes, even though GCM modes are performant, we still lose about 10-15% in writes and about 30% on reads when compared to no encryption at all. Mind you, as with all synthetic tests giving you the worst figures, the real performance loss is much lower.

Make what you want of it, but I’ll keep encrypting my drives. They’re plenty fast.


PS: You can take a peek at the raw data if you’re so inclined.

Wireguard on Mikrotik RouterOS 7 (and an Ubuntu Client Setup)

With an upgrade to Mikrotuk RouterOS 7.2, my OpenVPN setup started showing signs of distress in the form of a connection loss every hour or so. Instead of downgrading to the previously good version, I decided to abandon OpenVPN altogether. I figured it was about time to get Wireguard going.

As with OpenVPN setup, I will show all steps assuming you’re comfortable with both RouterOS and Ubuntu command line. And yes, an Ubuntu setup will work pretty much for any other linux with just a few minot changes.

First we need to create a Wireguard interface on the Mikrotik router. Here make a note of the “SERVER-PUBLIC” key. You will need it later.

/interface wireguard
add listen-port=51820 name=wireguard1
print
 Flags: X - disabled; R - running``
 0  R name="wireguard1" listen-port=51820 private-key="^^SERVER-PRIVATE^^" public-key="^^SERVER-PUBLIC^^"``

With the interface created, we need to add IP address for it. In my case, I choose 192.168.2.1 in a completely separate 192.168.2.0/24 subnet for this purpose.

/ip address
add address=^^192.168.2.1/24^^ network=^^192.168.2.0^^ interface=wireguard1

Finally, assuming you have a firewall sorted out, we need to add two rules - one for Wireguard itself and another one to allow communication with other nodes connected to the same router. I will add both of them at the very beginning but you should adjust their location to fit with your setup.

/ip firewall filter
add chain=input protocol=udp dst-port=51820 action=accept place-before=0
add chain=forward in-interface=wireguard1 action=accept place-before=1

To allow Wireguard clients access to Internet, we also need to do some masquerade (assuming ether1 is your Internet interface).

/ip firewall nat
chain=srcnat src-address=^^192.168.2.0/24^^ out-interface=ether1 action=masquerade

Now we need to get onto Ubuntu client and set wireguard there. The first step is, of course, to install some packages.

sudo apt update
sudo apt install --yes wireguard

Before anything else, we need a private and public key created. I like to get them both into variables instead of the files. Yes, it’s not as secure but for a single-user computer it’s good enough. Do make note of client’s public key as we’ll need it soon.

WG_PRIVATE_KEY=`wg genkey`
WG_PUBLIC_KEY=`echo $WG_PRIVATE_KEY | wg pubkey`
echo $WG_PUBLIC_KEY
 CLIENT-PUBLIC

Now we need to create a Wireguard configuration file. Make sure to replace “SERVER-PUBLIC” with whatever public key you generated on server (not client!) and for endpoint make sure you give IP (or DNS name) of your router.

cat << EOF | sudo tee /etc/wireguard/wg0.conf
[Interface]
PrivateKey = $WG_PRIVATE_KEY
Address = ^^192.168.2.20/24^^

[Peer]
PublicKey = ^^SERVER-PUBLIC^^
Endpoint = ^^192.168.0.1^^:51820
AllowedIPs = 0.0.0.0/0
EOF

Once we have the config file ready, we need to get back to RouterOS and add our client as a peer using its public key. Note that this “CLIENT-PUBLIC” is a public key we got in Ubuntu just a few moments ago.

/interface wireguard peers
add interface=wireguard1 allowed-address=^^192.168.2.20/24^^ public-key="^^CLIENT-PUBLIC^^"

If everything went fine, you should have VPN properly configured. The easiest way of checking it is to simply bring interface up and check the route. It should show us using Wireguard interface (and IP) with pings flowing freely.

sudo wg-quick up wg0

ip route get 1.1.1.1
 1.1.1.1 dev wg0 table 51820 src 192.168.2.20 uid 1000

ping 1.1.1.1

If we want this connection to be up every time we boot the system, we can enable it as a service

sudo systemctl enable wg-quick@wg0.service

And that’s all folks.

Ubuntu 22.04 on Surface Go

With the new LTS Ubuntu just round the corner, it came time to refresh my Surface Go operating system. For this guide I will assume you have the Windows currently running so you can write Ubuntu 22.04 image using Rufus utility. Make sure to use GPT partition scheme targeting UEFI systems.

Illustration

First we need to partition disk and install Linux for which we have to boot from Ubuntu USB drive. To do this go to Recovery Options and select Restart now. From the boot menu then select Use a device and finally use Linpus lite. If you are using Ubuntu, there is no need to disable secure boot or meddle with USB boot order as 22.04 fully supports secure boot.

While you could proceed from here with normal Ubuntu install, I like a bit more involved process that includes a bit of command line. Since we need root prompt, we should open Terminal and get those root credentials going.

sudo -i

The very next step should be setting up a few variables - host, user name, and disk(s). This way we can use them going forward and avoid accidental mistakes.

HOST=^^desktop^^
USER=^^user^^
DISK=/dev/mmcblk0

Disk setup is really minimal. Notice that both boot and EFI partition will need to be on internal disk as BIOS doesn’t know how to boot from micro-SD card.

blkdiscard -f $DISK
sgdisk --zap-all                       $DISK
sgdisk -n1:1M:+63M -t1:EF00 -c1:EFI    $DISK
sgdisk -n2:0:+640M -t2:8300 -c2:Boot   $DISK
sgdisk -n3:0:0     -t3:8309 -c3:Ubuntu $DISK
sgdisk --print                         $DISK

I usually encrypt just the root partition as having boot partition unencrypted does offer advantages and having standard kernels exposed is not much of a security issue.

cryptsetup luksFormat -q --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf pbkdf2 --hash sha256 ${DISK}p3

Since crypt device name is displayed on every startup, for Surface Go I like to use host name here.

cryptsetup luksOpen ${DISK}p3 ${HOST^}

At last we can prepare all needed partitions.

yes | mkfs.ext4 /dev/mapper/${HOST^}
mkdir /mnt/install
mount /dev/mapper/${HOST^} /mnt/install/

yes | mkfs.ext4 ${DISK}p2
mkdir /mnt/install/boot
mount ${DISK}p2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI -i 4d65646f ${DISK}p1
mkdir /mnt/install/boot/efi
mount ${DISK}p1 /mnt/install/boot/efi

To start the fun we need debootstrap package. Do make sure you have Wireless network connected at this time as otherwise operation will not succeed.

apt update ; apt install --yes debootstrap

And then we can get basic OS on the disk. This will take a while.

debootstrap $(basename `ls -d /cdrom/dists/*/ | head -1`) /mnt/install/

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials:

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/
``

Finally we're ready to "chroot" into our new system.

```sh
mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env HOST=$HOST USER=$USER DISK=$DISK \
    bash --login

For new system we need to setup the locale and the time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes initramfs-tools cryptsetup keyutils grub-efi-amd64-signed shim-signed

Since we’re dealing with encrypted data, we should auto mount it via crypttab. If there are multiple encrypted drives or partitions, keyscript really comes in handy to open them all with the same password. As it doesn’t have negative consequences, I just add it even for a single disk setup.

echo "${HOST^} UUID=$(blkid -s UUID -o value ${DISK}p3)  none \
    luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
cat /etc/crypttab

To mount boot and EFI partition, we need to do some fstab setup too:

echo "UUID=$(blkid -s UUID -o value /dev/mapper/${HOST^}) \
    / ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value ${DISK}p2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value ${DISK}p1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
    mem_sleep_default=deep\"/" /etc/default/grub
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

apt install --yes ubuntu-desktop-minimal firefox

Short package upgrade will not hurt.

add-apt-repository universe
apt update ; apt dist-upgrade --yes

The only remaining task before restart is to create the user, assign a few extra groups to it, and make sure its home has correct owner.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
echo "$USER ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USER
passwd $USER

As install is ready, we can exit our chroot environment.

exit

And unmount our disk:

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}

After the reboot you should be able to enjoy your Ubuntu installation.

reboot

If all went fine, congratulations, you have your Ubuntu up and running.

CyberCard

Illustration

As I got another CyberPower UPS for my devices, I again decided to use a custom card to connect it to my computer. However, using my existing PCB was a no-go as chip shortage meant my bellowed Si8621 was unobtanium again. Well, I guess it was time to have it redesigned anyhow…

For those joining the party late, this custom card is intended for the CyberPower expansion slot thus allowing for a more direct query and control interface than what you usually get from the 1U UPS range. Benefit of this approach is that you can script whatever support you want for you operating system without having to deal with CyberPower’s software (or, in case of Linux, lack there of).

Fortunate for me, PCBgogo was interesting in sponsoring some boards so I went onto designing the new version. This included cleaning board a bit, swapping controller into an easier-to-solder SOIC package, and lastly swapping the digital isolator for some optocouplers.

Optocouplers actually fit this project rather well as UART signal is inverted on UPS side and speed is low enough (2,400 baud) that selection process almost doesn’t matter. From previous projects I had some LTV-817S optocouplers that are still both cheap and easily obtainable. Even better, there are a few other optocouplers in the same 4-pin package so there is little risk of not finding a replacement if needed.

With all changes done, I went onto PCBgogo to order some boards and was greeted by an interface really similar to what I saw before with other PCB houses. It seems they all share the same web guy. :)

This board didn’t need much so I stayed with normal FR4 but those playing with high power LEDs might be interested in aluminium or copper substrate.

Color selection was reasonable with “normal” colors only affecting manufacturing time (green was the fastest) while there was an extra charge for matte variants. At least it was so at the time I did the original order. When I look at it now, there seems to be a premium for anything other than green (that has a $5 special price).

Illustration

Lead HASL finish comes standard while all other surfaces are available at extra expense. I might be spoiled brat but I would love to have ENIG. And, while ENIG is one of the options, paying $40+ is way too expensive in my opinion. And yes, for this board it really doesn’t matter. But if you go to SSOP and smaller, ENIG is god-given.

Once order was made, the gerber files were manually inspected by a real person. As it happens, my board actually had a paste mask incorrectly set above one pad and they actually caught it. All got sorted over email with PCBgogo manually applying corrections. Nice extra!

The finished PCB arrived reasonably quickly (especially since I wanted the cheapest shipping) and I have no complaints there. PCBs came vacuum packaged which was probably not necessary for their health but nice anyhow. The edges were properly cleaned up and board was quite leveled despite being only HASL. It seems to me that all PCB manufacturers pretty much solved HASL these days.

Interestingly I received one extra PCB in my package which reminded me of old times when manufacturers would do this just in case one of the boards was faulty (electronics version of baker’s dozen, I guess). With PCBgogo this wasn’t a case and all boards were in perfect working condition.

When looking boards sideways, I was surprised to see (at least) two slightly different FR4 substrates indicating either two manufacturing lines or two different manufacturers. Soldermask color was the same for all boards so I suspect it’s the former but I don’t recall ever noticing a substrate difference with any other board house. Regardless, if the exact substrate is important for your board, I suspect you would go for at least “impedance control” option.

Illustration

In addition to the PCB I also got cover and there lies my major complaint - the dreadful PCB order number. Instead of placing it on the back, it got silkscreened right on the front of the board. Since human was involved in the review, I hoped to a more sensible placement on something that’s obviously a front panel. In their defense, they do offer an option to remove it if you leave a note and this seems to be offered at no extra cost. Just be careful not to forget writing it in.

With all components soldered, I got the board into my new UPS. Worked like a charm.


Please note I have received PCBs for this project for free from PCBgogo. They didn’t request anything specific for me to say and all opinions given here are mine and mine alone.

PS: You can see the latest version of the board on GitHub. The latest version actually gets rid of voltage regulator too so it’s as simple as it gets now.

Cananka for Framework Laptop

Illustration

I find the Framework laptop an interesting concept and the only thing preventing me from buying one was the fact there’s no 15" model. As I use a laptop screen as my primary visual interface most of the time, I simply find 13" a tad bit too small. However, I couldn’t help but notice their developer program for expansion cards. And that peeked my interest. Can I fit a CAN bus interface there?

All resources for expansion cards are part of a reasonably structured GitHub repository. In there you’ll find KiCAD templates for creating your own PCB and a bunch of a mechanical drawings in addition to an enclosure suitable for 3D printing. Those using other PCB software are kinda out of luck as board description is not sufficient to replicate all the curves board has. Fortunately, due to KiCAD using a (relatively speaking) simple human-readable file format, I managed to create a DipTrace PCB template.

Since I already have a CAN bus USB controller, I figured to just make everything a bit smaller while retaining as much features I could. It was immediately clean that I couldn’t support a fully insulated interface as there was no chance to fit all components. Cananka mini was as good as it gets. While most of LEDs had to go due to the lack of space, the programmatically-controlled bus termination thankfully remained.

After much jiggling of components in order to fit them, all was done. A type-C connector was a pain in the butt to hand solder and 0.5 mm microcontroller and UART controller were close to follow. Even better, all passives retained their 0805 size. I did have to go with a full size PCB in order to fit everything onto a single PCB side but I figured 3D printed case would be ok.

Selecting a user-facing connector was a pure trouble and I went over multiple ideas to no avail. My favorite failure was using an edge connector with a Phoenix ZEC connector attached to it but I found it sticking from the side of laptop was a bit too much. At the end I went with a surprisingly well-fitting bodge. A vertical JST PH connector required just a minimal pin bending to fit highly restricted module height perfectly. Not only that but the connector is common enough that finding a premade cable (or making your own) will present no trouble.

Manufacturing board was actually a breeze as I modified PCB slightly as not to cause warnings with the Molex 105444-0001 connector. Any manufacturer capable of producing double-sided 0.8mm PCB at 6/6 should be just fine. And yes, you really need thinner than regular PCB in order to mount a type-C connector.

With PCB in hand and after a LONG time waiting for components (darn car manufacturers got all the good CAN bus stuff), I was ready to test the device. And here is the genius of Framework’s extension card design - it’s still just a type-C so I managed to test everything before ever needing a Framework device. Really easy R&D.

Illustration

However, with all electrical stuff out of way, it came time to see if my expansion card can actually fit. I contacted Framework for a loaner laptop to test it, got it in a few days, tried to push card in, and… the darn thing fitted like a glove. :)

While I would consider the whole adventure a success, I am not completely happy. First of all, a full size PCB was a mistake. I expected it to terminate 1 or 2 millimeters before module’s edge but in reality it’s flush with the frame. That means my connector is actually sticking out a bit. Probably not enough to matter for normal handling but any harder blow to the side (e.g. due to throwing a laptop into the bag) runs risk of just shearing the connector off. Also, while 3D printed case is fine for prototyping, having a better looking finish would go a long way.

Illustration

What’s next then? Well, the first step will be shortening PCB by a few millimeters as not to have connector sticking out from side. This change will be easily done as there is still enough space on a PCB. But that’s not where I’ll stop.

My thoughts go toward making the board even smaller as to fit into the existing Framework USB type-A enclosure. While this will mean I will have to go toward double-sided PCB load, it will look MUCH neater even though there will be some extra space. My thinking is to 3D print a transparent plug that would allow a status LED to shine through. That would avoid the “gaping hole” look while actually serving a purpose.

Timeline for this is a painful topic. While I managed to obtain some PIC18F25K80 chips, I am completely out of MCP2221A. And the soonest anybody seems to be able to get them is end of the year. And let’s not even talk about MCP2561 CAN bus controller. In short, I don’t expect anything new before the next year no matter how quickly I turn the PCBs around.

However, for now, I have the single Framework with a “native” CAN bus interface out there. :)


PS: While I did ask Framework for a loaner, I was told to just keep it once I queried how to send it back. I don’t believe my experience with developing and designing expansion card was affected by this as 95% of this work was completed way before a free Framework landed in my lap. I was never asked by a Framework team to write anything nor they conditioned sending me laptop to a positive coverage.

PPS: Yes, despite now having a 13" Framework in my possession, I still hope to purchase a 15" variant if it becomes available.