Linux, Unix, and whatever they call that world these days

A Key to Mute the Microphone

One thing I love about my work Lenovo is its Microphone mute button. While every application offers mute functionality, having it as a special button is really handy. You can even do scripting around that. So, I wanted the same for my Framework 16.

Since Framework 16 keyboard is QMK based (albeit older version), changing key assignment was mostly figuring out which key is muting microphone. Not to keep you in the suspense - that key is F20. Each press on F20 mutes and unmutes microphone - just how standard audio mute functionality does to outputs.

So, with the knowledge of the key, the only decision left was where to assign that key too. And for that, I found = key on numpad looking the best. My whole current Numpad setup looks like this (both with and without NumLock):

┌────┬────┬────┬────┐     ┌────┬────┬────┬────┐
│Esc │PScr│MicM│Mute│     │Esc │Calc│ =  │ <- │
├────┼────┼────┼────┤     ├────┼────┼────┼────┤
│ Num│Bck-│Bck+│Vol-│     │ Num│ /  │ *  │ -  │
├────┼────┼────┼────┤     ├────┼────┼────┼────┤
│Home│ ↑  │PgUp│    │     │ 7  │ 8  │ 9  │    │
├────┼────┼────┤    │     ├────┼────┼────┤    │
│ ←  │    │ →  │Vol+│     │ 4  │ 5  │ 6  │ +  │
├────┼────┼────┼────┤     ├────┼────┼────┼────┤
│End │ ↓  │PdDn│    │     │ 1  │ 2  │ 3  │    │
├────┴────┼────┤    │     ├────┴────┼────┤    │
│ Insert  │Del │Entr│     │ 0       │ .  │Entr│
└─────────┴────┴────┘     └─────────┴────┴────┘

Thus, that changed my definition of keyboard to:

[_FN] = LAYOUT(
    KC_ESC,  S(KC_PRINT_SCREEN), KC_F20,  KC_MUTE,
    KC_NUM,  KC_BRID, KC_BRIU, KC_VOLD,
    KC_P7,   KC_P8,   KC_P9,
    KC_P4,   KC_P5,   KC_P6,   KC_VOLU,
    KC_P1,   KC_P2,   KC_P3,
    KC_INS,  KC_DEL,  KC_ENT
),

Short recompile later and my numpad now has that extra key for much easier muting. As always, QMK code is freely available.

Capturing Govee Temperature in Docker

In my previous post I discussed reading Govee sensor temperature in a script. And that is perfectly fine. However, this is not ideal for my server environment. What I want is a Docker container.

Since I like Alpine images, the first step was to compile GoveeBTTempLogger. After installing prerequisites, I was greeted with bunch of errors:

/root/GoveeBTTempLogger/goveebttemplogger.cpp: In function 'bool ValidateDirectory(const std::filesystem::__cxx11::path&)':
/root/GoveeBTTempLogger/goveebttemplogger.cpp:924:23: error: aggregate 'ValidateDirectory(const std::filesystem::__cxx11::path&)::stat64 StatBuffer' has incomplete type and cannot be defined
  924 |         struct stat64 StatBuffer;
      |                       ^~~~~~~~~~
/root/GoveeBTTempLogger/goveebttemplogger.cpp:925:59: error: invalid use of incomplete type 'struct ValidateDirectory(const std::filesystem::__cxx11::path&)::stat64'
  925 |         if (0 == stat64(DirectoryName.c_str(), &StatBuffer))
      |                                                           ^
...

As lovers of Alpine know, due to its use of musl, this is not an uncommon occurrence. Fix was easy enough so I created a pull request myself. With this sorted out, it was time for Dockerfile.

Base prerequisites were obvious:

FROM alpine:latest
USER root

RUN apk add dbus bluez bluez-dev libstdc++
RUN rc-update add dbus bluetooth default
...

However, depending on services is not something Alpine does out-of-box. Openrc runlevel requires more direct access to machine. But, since I was not the first person needing it, solution already exists and it’s called softlevels. To enable them, three lines are enough:

RUN apk add openrc
RUN mkdir -p /run/openrc/exclusive
RUN touch /run/openrc/softlevel

This and a couple of wrapper scripts was all that was needed to get it running. But, I was still one step away from making it work in my environment. I needed compose.yaml and this is what I came up with (notice dbus volume):

services:
  govee2telegraf:
    container_name: govee2telegraf
    image: medo64/govee2telegraf:latest
    restart: unless-stopped
    privileged: true
    environment:
      TELEGRAF_HOST: <host>
      TELEGRAF_PORT: <port>
      TELEGRAF_BUCKET: <bucket>
      TELEGRAF_USERNAME: <username>
      TELEGRAF_PASSWORD: <password>
    volumes:
      - /var/run/dbus/:/var/run/dbus/:z

Image is available on DockerHub and everything else is on GitHub.

Capturing Temperature of Govee Bluetooth Sensors

I have an quite a few Govee temperature and humidity sensors. They’re reasonably priced, quite accurate, and they’re bluetooth LE. Yes, that allows them to sip power but at a cost that I cannot reach them when outside of home. Well, unless I get one of Govee hubs and connect them to cloud. But, is there a way to bypass the cloud and push all to my telegraf instance? Well, now there is!

First of all, why Telegraf? Obvious answer is because I have it already setup in my network and connected with my Grafana GUI. Longer answer is because I like the idea of telegraf. You have a centralized database and pushing to it is as easy as sending HTTP request. Everything is quite free-form and any mess you create is sorted out when data is displayed in Grafana.

Next question is, how? Well, I originally planned to roll my own script by abusing bluetoothctl scripting. However, during research I fount out that gentleman named William C Bonner already did pretty much the exact thing I wanted to. His GoveeBTTempLogger already both captures and decodes Govee temperature and humidity data.

And yes, there is no x64 package precompiled but, surprisingly, README.md instructions actually work. That said, I opted to build binaries a bit differently. This allowed me to install binary into /usr/local/bin/.

sudo apt install build-essential cmake git libbluetooth-dev libdbus-1-dev
git clone https://github.com/wcbonner/GoveeBTTempLogger.git
cd GoveeBTTempLogger
cmake -B ./build
sudo cmake --build ./build --target install

Once compiled, we can start application and, hopefully, see all the temperatures.

goveebttemplogger

And, if you just want to see the current values, that’s enough. If you check into README.md a bit more, you can also setup application to output web pages. Unfortunately, there is no telegraf output option. Or thankfully, since this gives me option to roll my own script around this nice tool.

What I ended up with is the following.

TG_HOST=<ip>
TG_PORT=<port>
TG_BUCKET=<bucket>
TG_USERNAME=<user>
TG_PASSWORD=<password>

while IFS= read -r LINE; do
  DATA=`echo "$LINE" | grep '(Temp)' | grep '(Humidity)' | grep '(Battery)'`
  if [ "$DATA" == "" ]; then continue; fi

  DEVICE=`echo $DATA | awk '{print $2}' | tr -d '[]'`
  TEMPERATURE=`echo $DATA | awk '{print $4}' | tr -dc '0-9.'`
  HUMIDITY=`echo $DATA | awk '{print $6}' | tr -dc '0-9.'`
  BATTERY=`echo $DATA | awk '{print $8}' | tr -dc '0-9.'`

  printf "%s %5s°C %4s%% %3s%%\n" $DEVICE $TEMPERATURE $HUMIDITY $BATTERY
  CONTENT="temp,device=$DEVICE temp=${TEMPERATURE},humidity=${HUMIDITY},battery=${BATTERY} `date +%s`"$'\n'
  CONTENT_LEN=$(echo -en ${CONTENT} | wc -c)
  echo -ne "POST /api/v2/write?u=$TG_USERNAME&p=$TG_PASSWORD&bucket=${TG_BUCKET}&precision=s HTTP/1.0\r\nHost: $TG_HOST\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: ${CONTENT_LEN}\r\n\r\n${CONTENT}" | nc -w 15 $TG_HOST $TG_PORT
done < <(/usr/local/bin/goveebttemplogger --passive)

This script goes over goveebttemplogger output and extracts device MAC address and its data. That data is then packed into Telegrafs line format and simply posted into nc as raw HTTP output. Not more difficult than wget or curl.

Wrapping this into a service so it runs in the background is an exercise left to the reader.

Encrypted Kubuntu 24.10 with ZFS and Hibernation

For a while now I have published manual installation steps for pretty much any Ubuntu release since 18.10. Since, with Plasma 6, I switched to KDE and Kubuntu, I am abandoning that series. This guide will be the first one with manual installation steps for Kubuntu.

First, the big question: Why manual? Well, I view two things as absolutely mandatory for my Linux desktop: ZFS and encryption. One that I really like to have is hibernation. Default installation uses ZFS’ native encryption which, while nice and fast, doesn’t encrypt metadata. I am not saying that ZFS encryption is bad - I am just saying I am not necessarily comfortable with it. And swap size is way too small for hibernation to work. Thus, I like to do it all manually.

That said, if GUI installation leaves the system in state you like, there is no reason to follow this guide. Save yourself a bit of time.

As previously noted, this guide is for Kubuntu - an Ubuntu variant using KDE and not Gnome. While I was Gnome user for quite a long time, it was more of a stockholm syndrom rather than love. With KDE Plasma 6, I found something I actually like. As for the guide, Ubuntu and Kubuntu have a huge overlap and thus most of the steps are actually the same up to GUI install. It should be easy enough to look into an old Ubuntu guide and adjust the steps. I will probably still update Ubuntu steps for long-term releases (last one was 24.04)

With prologue done, what am I actually trying to achieve? Well, I want a minimal Kubuntu installation for my Framework 13 laptop that supports ZFS and hibernation. Both data and hibernation files are to be protected by LUKS encryption.

Finally, let’s go over all the steps to make it happen.

The first step is to boot into the USB installation and use “Try Ubuntu” option. Once on the a desktop, we want to open a terminal, and, since all further commands are going to need root access, we can start with that.

sudo -i

Then we add a few packets that come by default with Ubuntu install but are extras on Kubuntu.

apt update
apt install -y gdisk zfsutils-linux

Next step should be setting up a few variables - disk, hostname, and username. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.

DISK=/dev/disk/by-id/<disk>
HOST=<host>
USERNAME=<user>

For partition, I like to have 4 of them. The first two partitions are unencrypted and in charge of booting (boot + EFI). While I love encryption, I almost never encrypt the boot partitions in order to make my life easier as you cannot seamlessly integrate the boot partition password prompt with the later password prompt. Thus encrypted boot would require you to type the password twice (or thrice if you decide to use native ZFS encryption on top of that).

Third partition is swap that we need for hibernation support. I will use 64GB here because my laptop can have up to 96GB. While swap only requires about 40% of RAM to be backed by swap, having a few gigs extra will not hurt.

The last partition is the largest and will contain all user data.

One extra calculation is needed to figure out 4K aligned sector count.

LASTDISKSECTOR=$(( `blockdev --getsz $DISK` / 2048 * 2048 - 1 ))

And then we can create all the partitions needed:

blkdiscard -f $DISK 2>/dev/null
sgdisk --zap-all                               $DISK
sgdisk -n1:1M:+255M          -t1:EF00 -c1:EFI  $DISK
sgdisk -n2:0:+1792M          -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:+64G            -t3:8200 -c3:Swap $DISK
sgdisk -n4:0:$LASTDISKSECTOR -t4:8309 -c4:LUKS $DISK
sgdisk --print                                 $DISK

To ease commands used later, here we can get partition UUIDs. While we can use disk names directly (as I often did before), using partition UUIDs helps if we ever clone disk to another physical drive.

PART1=`blkid -s PARTUUID -o value $DISK-part1`
PART2=`blkid -s PARTUUID -o value $DISK-part2`
PART3=`blkid -s PARTUUID -o value $DISK-part3`
PART4=`blkid -s PARTUUID -o value $DISK-part4`

The next step is to setup all LUKS partitions. If you paid attention, that means we need to repeat formatting a total of 2 times. Unless you want to deal with multiple password prompts, make sure to use the same password for both:

cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i /dev/disk/by-partuuid/$PART4
cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i /dev/disk/by-partuuid/$PART3

Since creating encrypted partitions doesn’t mount them, we do need this as a separate step. I like to name my LUKS devices based on partition names so we can recognize them more easily:

cryptsetup luksOpen \
    --persistent --allow-discards \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    /dev/disk/by-partuuid/$PART4 $PART4
cryptsetup luksOpen \
    --persistent --allow-discards \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    /dev/disk/by-partuuid/$PART3 $PART3

Finally, we can set up our ZFS pool with an optional step of setting quota to roughly 85% of disk capacity. Since we’re using LUKS, there’s no need to setup any ZFS keys. Name of the pool will match name of the host and it will contain several datasets to start with. Most of my stuff goes to either Data dataset for general use or to VirtualBox dataset for virtual machines. Consider this just a suggestion and a good starting point, adjust as needed.

zpool create \
    -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O quota=3200G \
    -O canmount=off -O mountpoint=none -R /mnt/install \
    ${HOST^} /dev/mapper/$PART4
zfs create \
    -o reservation=100G \
    -o 28433:snapshot=72 \
    -o devices=on \
    -o canmount=noauto -o mountpoint=/ \
    ${HOST^}/System
zfs mount ${HOST^}/System
zfs create \
    -o 28433:snapshot=240 \
    -o canmount=noauto -o mountpoint=/home \
    ${HOST^}/Home
zfs mount ${HOST^}/Home
zfs set canmount=on ${HOST^}/Home
zfs create \
    -o 28433:snapshot=360 \
    -o canmount=noauto -o mountpoint=/Data \
    ${HOST^}/Data
zfs set canmount=on ${HOST^}/Data
zfs create \
    -o recordsize=32K \
    -o 28433:snapshot=72 \
    -o canmount=noauto -o mountpoint=/VirtualBox \
    ${HOST^}/VirtualBox
zfs set canmount=on ${HOST^}/VirtualBox
zfs set devices=off ${HOST^}

With ZFS done, we might as well setup boot, EFI, and swap partitions too:

yes | mkfs.ext4 /dev/disk/by-partuuid/$PART2
mkdir /mnt/install/boot/
mount /dev/disk/by-partuuid/$PART2 /mnt/install/boot/
mkfs.msdos -F 32 -n EFI -i 4d65646f /dev/disk/by-partuuid/$PART1
mkdir /mnt/install/boot/efi/
mount /dev/disk/by-partuuid/$PART1 /mnt/install/boot/efi/
mkswap /dev/mapper/$PART3

At this time, I also often disable IPv6 as I’ve noticed that on some misconfigured IPv6 networks it takes ages to download packages. This step is both temporary (i.e., IPv6 is disabled only during installation) and fully optional:

sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
sysctl -w net.ipv6.conf.lo.disable_ipv6=1

To start the fun we need to debootstrap our OS. As of this step, you must be connected to the Internet:

apt update
apt dist-upgrade --yes
apt install --yes debootstrap
debootstrap --components=main,restricted,universe,multiverse \
    oracular /mnt/install/

We can use our live system to update a few files on our new installation:

echo $HOST > /mnt/install/etc/hostname
sed "s/kubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

At last, we’re ready to chroot into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install /usr/bin/env \
    PART1=$PART1 PART2=$PART2 PART3=$PART3 PART4=$PART4 \
    HOST=$HOST USERNAME=$USERNAME USERID=$USERID \
    bash --login

With our newly installed system running, let’s not forget to set up locale and time zone:

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales
dpkg-reconfigure tzdata

To download some stuff later, wget is useful.

apt install -y wget

In order for decryption to work, we do need to set up crypttab so our encrypted partitions:

echo -n > /etc/crypttab
echo "$PART3 PARTUUID=$PART3 none luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
echo "$PART4 PARTUUID=$PART4 none luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab

cat /etc/crypttab

To mount all those partitions, we also need some fstab entries too. ZFS entries are not strictly needed. I just like to add them in order to hide our LUKS encrypted ZFS from the file manager (habit from Nautilus days):

echo -n > /etc/fstab
echo "PARTUUID=$PART2    /boot     ext4 noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab
echo "PARTUUID=$PART1    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab
echo "/dev/mapper/$PART3 none      swap sw,nofail                                  0 0" >> /etc/fstab
echo "/dev/mapper/$PART4 none      auto nofail,nosuid,nodev,noauto                 0 0" >> /etc/fstab

cat /etc/fstab

Now we’re ready to onboard the latest Linux kernel. Since this is not a LTS release, generic kernel will do.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

On systems with a lot of RAM, I like to adjust memory settings a bit. This is inconsequential in the grand scheme of things, but I like to do it anyway. Think of it as wearing “lucky” socks:

echo "vm.swappiness=10" >> /etc/sysctl.conf
echo "vm.min_free_kbytes=1048576" >> /etc/sysctl.conf
cat /etc/sysctl.conf

Now we can create the boot environment:

apt install --yes zfs-initramfs cryptsetup keyutils plymouth-theme-spinner
update-initramfs -c -k all

And then, we can get grub going. Do note we also set up booting from swap (needed for hibernation) here too.

apt install --yes grub-efi-amd64-signed shim-signed

sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
    mem_sleep_default=deep \
    RESUME=UUID=$(blkid -s UUID -o value /dev/mapper/$PART3)\"/" \
    /etc/default/grub

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
    --bootloader-id=Ubuntu --recheck --no-floppy

cat /boot/grub/grub.cfg  | grep 'linux' | grep 'ZFS' | head -1 | xargs | cut -d' ' -f3-

I don’t like snap so I preemptively banish it from ever being installed:

apt remove --yes snapd 2>/dev/null

echo 'Package: snapd'    > /etc/apt/preferences.d/snapd
echo 'Pin: release *'   >> /etc/apt/preferences.d/snapd
echo 'Pin-Priority: -1' >> /etc/apt/preferences.d/snapd

apt update

And now, finally, we can install our minimal desktop environment:

apt install --yes kde-plasma-desktop man-db

Since I have a framework laptop, I like to allow trim operation on its external USB drives.

cat << EOF | tee /etc/udev/rules.d/42-framework-storage.rules
ACTION=="add|change", SUBSYSTEM=="scsi_disk", ATTRS{idVendor}=="13fe", ATTRS{idProduct}=="6500", ATTR{provisioning_mode}="unmap"
ACTION=="add|change", SUBSYSTEM=="scsi_disk", ATTRS{idVendor}=="32ac", ATTRS{idProduct}=="0005", ATTR{provisioning_mode}="unmap"
ACTION=="add|change", SUBSYSTEM=="scsi_disk", ATTRS{idVendor}=="32ac", ATTRS{idProduct}=="0010", ATTR{provisioning_mode}="unmap"
EOF

In order to have hibernation to work properly, we need to disable a few devices. Fortunately, we can create script to do it for us every time system goes to sleep.

cat << EOF | tee /usr/lib/systemd/system-sleep/framework
#!/bin/sh
case \$1 in
  pre)
    for DRIVER_LINK in \$(find /sys/devices/ -name "driver" -print); do
      DEVICE_PATH=\$(dirname \$DRIVER_LINK)
      if [ ! -f "\$DEVICE_PATH/power/wakeup" ]; then continue; fi
      DRIVER=\$( basename \$(readlink -f \$DRIVER_LINK) )
      if [ "\$DRIVER" = "i2c_hid_acpi" ] || [ "\$DRIVER" = "xhci_hcd" ]; then
        echo disabled > \$DEVICE_PATH/power/wakeup
     fi
    done
  ;;
esac
EOF

sudo chmod +x /usr/lib/systemd/system-sleep/framework

For hibernation, I like to change sleep settings so that hibernation kicks in automatically after 13 minutes of sleep.

sed -i 's/.*AllowSuspend=.*/AllowSuspend=yes/'                           /etc/systemd/sleep.conf
sed -i 's/.*AllowHibernation=.*/AllowHibernation=yes/'                   /etc/systemd/sleep.conf
sed -i 's/.*AllowSuspendThenHibernate=.*/AllowSuspendThenHibernate=yes/' /etc/systemd/sleep.conf
sed -i 's/.*HibernateDelaySec=.*/HibernateDelaySec=13min/'               /etc/systemd/sleep.conf

For that we also need to do a minor lid switch configuration adjustment. I also like to set my power button as a hibernation trigger.

apt install -y pm-utils
sed -i 's/.*HandlePowerKey=.*/HandlePowerKey=hibernate/'                          /etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitch=.*/HandleLidSwitch=suspend-then-hibernate/'           /etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitchExternalPower=.*/HandleLidSwitchExternalPower=ignore/' /etc/systemd/logind.conf
sed -i 's/.*HoldoffTimeoutSec=.*/HoldoffTimeoutSec=13s/'                          /etc/systemd/logind.conf

Since Firefox is a snapd package (and we banished it), we can install it manually.

add-apt-repository --yes ppa:mozillateam/ppa

cat << 'EOF' | sed 's/^    //' | tee /etc/apt/preferences.d/mozillateamppa
    Package: firefox*
    Pin: release o=LP-PPA-mozillateam
    Pin-Priority: 501
EOF

apt update && apt install --yes firefox

Chrome aficionados, can install it too:

pushd /tmp
wget --inet4-only https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt install -y ./google-chrome-stable_current_amd64.deb
popd

Lastly, we need to have a user too.

adduser --disabled-password --gecos '' -u $USERID $USERNAME
usermod -a -G adm,cdrom,dialout,dip,lpadmin,plugdev,sudo,tty $USERNAME
echo "$USERNAME ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USERNAME
passwd $USERNAME

It took a while, but we can finally exit our debootstraped environment:

exit

Let’s clean all mounted partitions and get ZFS ready for next boot:

zfs set devices=off ${HOST^}/System

sync
umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

After reboot, we should be done and our new system should boot with a password prompt.

reboot

Once system has booted, we can tell it to use local BIOS clock. It helps with dual-boot and I actually prefer to have local time in BIOS.

sudo timedatectl set-local-rtc 1 --adjust-system-clock

Of course, we can end with a hibernation test. If everything went correctly, this should work nicely.

sudo systemctl hibernate

Just the Hibernation Steps, Please

For a while now I use Ubuntu family of Linux on my Framework. While the exact flavor might change (Kubuntu rules! :)), two things remain the same. ZFS and hibernation. If you want to install system from scratch, you can find post for many different Ubuntu releases. What I don’t have is a separate post with just a hibernation steps. So, here it is.

First, let’s discuss prerequisites. You MUST have a separate swap partition if you want to hibernate to a file on ZFS. My setup is usually something like this:

PartitionTypeSize (MB)Description
EFIEF00FAT32255Small EFI partition
Boot8300EXT41,792Linux boot partition
Swap8200-65,536Swap, minimum 40% of RAM
(else)8309(whatever)one or more partitions for system and data

If you don’t have a separate partition, you can stop reading now and find somebody smarter. And no, the swap partition doesn’t need to be same size or larger than the amount of RAM you have. Minimum is actually 40% of your RAM (controlled by image_size parameter). Now, I’ve been guilty of having the swap the same size as RAM too. But that is not a requirement.

You also must have the Secure Boot disabled. If you don’t have it disabled, it will all seemingly work but system will never restore.

Steps I am giving here are for encrypted swap. I personally think that you should NEVER have unencrypted data on disk. NEVER. However, I am aware that most of people don’t care and see password entry as a chore. In one of the next posts, I will add instructions for non-encrypted swap too. So, if you are “one of those”, stay tuned.

Order of operations is quite fungible but, for the purpose of this guide, I will start with config files. There are two and whether you will modify them both depends on what you want to achieve.

In sleep.conf, I adjust suspend settings to allow for Suspend-then-Hibernate setup. My preference is to have system hibernate after 13 minutes of sleep but you can change that to your liking.

sudo sed -i 's/.*AllowSuspend=.*/AllowSuspend=yes/'                           /etc/systemd/sleep.conf
sudo sed -i 's/.*AllowHibernation=.*/AllowHibernation=yes/'                   /etc/systemd/sleep.conf
sudo sed -i 's/.*AllowSuspendThenHibernate=.*/AllowSuspendThenHibernate=yes/' /etc/systemd/sleep.conf
sudo sed -i 's/.*HibernateMode=.*/HibernateMode=platform shutdown/'           /etc/systemd/sleep.conf
sudo sed -i 's/.*HibernateDelaySec=.*/HibernateDelaySec=13min/'               /etc/systemd/sleep.conf

Second config file is to reassign power button to hibernation. This is just my preference and, if you want power button to stay as-is, you can omit this step. For this, you need to install pm-utils package, if not already present on your system. Again, these are settings I like so adjust as needed.

sudo apt install -y pm-utils
sudo sed -i 's/.*HandlePowerKey=.*/HandlePowerKey=hibernate/'                          /etc/systemd/logind.conf
sudo sed -i 's/.*HandleLidSwitch=.*/HandleLidSwitch=suspend-then-hibernate/'           /etc/systemd/logind.conf
sudo sed -i 's/.*HandleLidSwitchExternalPower=.*/HandleLidSwitchExternalPower=ignore/' /etc/systemd/logind.conf
sudo sed -i 's/.*HoldoffTimeoutSec=.*/HoldoffTimeoutSec=13s/'                          /etc/systemd/logind.conf

With this sorted out, you need to make sure computer doesn’t wake up. This step can be skipped quite often, but not so with Framework. With Framework laptops you need to manually disable wakeup for i2c_hid_acpi and xhci_hcd devices.

These commands will generate script I personally use and allow for its execution upon sleep.

cat << EOF | sudo tee /usr/lib/systemd/system-sleep/framework
#!/bin/sh
case \$1 in
  pre)
    for DRIVER_LINK in \$(find /sys/devices/ -name "driver" -print); do
      DEVICE_PATH=\$(dirname \$DRIVER_LINK)
      if [ ! -f "\$DEVICE_PATH/power/wakeup" ]; then continue; fi
      DRIVER=\$( basename \$(readlink -f \$DRIVER_LINK) )
      if [ "\$DRIVER" = "i2c_hid_acpi" ] || [ "\$DRIVER" = "xhci_hcd" ]; then
        echo disabled > \$DEVICE_PATH/power/wakeup
     fi
    done
  ;;
esac
EOF
sudo chmod +x /usr/lib/systemd/system-sleep/framework

Now, we can setup swap. For this I strongly recommend using variables as to avoid any naming issues. Replacewith your partition (e.g. /dev/disk/by-id/whatever-part3).

PART=<part>
UUID=`sudo blkid -s PARTUUID -o value $PART`

Assuming your swap is not initialized, you need to do so.

sudo cryptsetup luksFormat -q --type luks2 \
  --sector-size 4096 \
  --cipher aes-xts-plain64 --key-size 256 \
  --pbkdf argon2i /dev/disk/by-partuuid/$UUID
sudo cryptsetup luksOpen \
  --persistent --allow-discards \
  --perf-no_write_workqueue --perf-no_read_workqueue \
  /dev/disk/by-partuuid/$UUID $UUID
mkswap /dev/mapper/$UUID

Of course, adding this to both crypttab and fstab is also needed for swap to work properly.

echo "$UUID PARTUUID=$UUID none luks,discard,initramfs,keyscript=decrypt_keyctl" | sudo tee -a /etc/crypttab
echo "/dev/mapper/$UUID none swap sw,nofail 0 0" | sudo tee -a /etc/fstab

The final step is adding swap as RESUME into grub. Note that swap will be identified based on partition UUID.

sudo sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
  rtc_cmos.use_acpi_alarm=1 \
  RESUME=UUID=$(blkid -s UUID -o value /dev/mapper/$UUID)\"/" \
  /etc/default/grub

With these things in place, you should be able to use hibernate. To check, use systemctl.

sudo systemctl hibernate

And yes, you can probably use these steps with any laptop, not just Framework. However, I tested this on Framework and thus will not make other claims. :)

Sweet Dreams, My Dear Framework

Setting up sleep on my Framework 13 was a bit annoying but, once set, it worked perfectly. However, the same solution didn’t work on my Framework 16. While my installation is far from standard, the hibernation steps are quite straighforward:

  1. Setup swap partition RESUME variable in grub loader
  2. Adjust sleep.conf
  3. Disable wakeup for troublesome components so your laptop doesn’t wake immediately

And it was the step 3 that presented the problem - the darn thing kept waking up.

Since I sorted this out with Framework 13, I figured I can do the same for Framework 16. Even better, I found a forum post that actually told me which components need more of a sleep.

echo disabled > /sys/devices/pci0000:00/0000:00:08.1/0000:c1:00.3/usb1/1-4/1-4.3/power/wakeup
echo disabled > /sys/devices/platform/AMDI0010:03/i2c-1/i2c-PIXA3854:00/power/wakeup
echo disabled >/sys/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:4b/PNP0C09:00/PNP0C0D:00/power/wakeup

Yes, it was different than my approach of disabling them in udev rules, but the same idea overall. And it even has the same suspects as for Framework 13, most notably Touchpad (i2c_hid_acpi), lid switch (button), and USB xHCI (xhci_hcd). Full of hope, I tried that and my computer still woke up.

My next step was to check the file (yes, I blindly copy/pasted it) and problem was obvious. My devices were at a different path. So I adjusted, tried it again, finished up a blog post, and called it a day. I mean, everything was working perfectly. Or so I thought.

After a few days, I placed computer into hibernate only for the darn thing to wake up on me. What the heck? I though I solved that issue. So I checked and noticed my devices were at slightly different location. Hm, maybe in all the fuss around finishing up blog post I accidentally made an error. So I addjusted paths and, with everything working correctly, called it a day.

But, guess what, in a few days I got the same issue again. And this time I was certain I had it done correctly. However, device paths were changed again. With so many independent USB devices, plug-and-play was moving stuff around every time system was rebooted.

So, I needed to script my /usr/lib/systemd/system-sleep/framework a bit smarter. At the end I ended up with this:

#!/bin/sh
case $1 in
  pre)
    for DRIVER_LINK in $(find /sys/devices/ -name "driver" -print); do
      DEVICE_PATH=$(dirname $DRIVER_LINK)
      if [ ! -f "$DEVICE_PATH/power/wakeup" ]; then continue; fi
      DRIVER=$( basename $(readlink -f $DRIVER_LINK) )
      if [ "$DRIVER" = "i2c_hid_acpi" ] || [ "$DRIVER" = "xhci_hcd" ]; then
        echo disabled > $DEVICE_PATH/power/wakeup
     fi
    done
  ;;
esac

This will search for naughty devices every time hibernate is called upon and turn off wakeup. If PnP moves them, no worries, script will find them again.

And yes, the same script works for both Framework 13 and 16.


P.S.: While I mention button driveer in text, script actually doesn’t disable wakeup on lid switch. I kinda like computer to wake when I open it.

Meld Context Menu in KDE Plasma 6

Illustration

As a new convert to KDE Plasma 6, I am still in process of getting the whole config sorted out. And that includes thing that was annoyingly difficult to do on Gnome - context menu support for Meld.

While there is (was) an extension available, it’s not maintained and with Ubuntu 24.04 there came an end to it actually working. I did play with a few other potential solutions, but I wasn’t really successful in seamlessly integrating any of them into Nautilus.

So, when I got KDE Plasma 6 running, I decided to see how easy is to integrate Meld in context menu for its file manager “Dolphin”. I did find a few solutions on Internet but they were either bringing overly complicated menus or they were simply non-fuctional. So I decided to roll my own.

Great help here was dolphin’s service menu specification that (surprisingly) really does it job - covers all the options and leaves you smarter than you were before reading it. I know that should be the purpose of any documentation but Linux documentation often fails at that simple aspect. For my project, I wanted an option to either compare two directories or two files. And I wanted just a normal compare and not a thousand options in submenu.

For this I ended up creating two .desktop files and limit visibility of each to when two items are selected. One .desktop file is handling compare for two files while the other handles the same for two directories.

To bring the story to the end, here are the commands to recreate those files. Just create directory, write files, and make them executable. Easy-peasy.

mkdir -p ~/.local/share/kio/servicemenus
cat << 'EOF' | ~/.local/share/kio/servicemenus/meld.directory.service.desktop
[Desktop Entry]
Type=Service
MimeType=inode/directory
Actions=diffDirectories
X-KDE-Priority=TopLevel
X-KDE-RequiredNumberOfUrls=2

[Desktop Action diffDirectories]
Name=Compare Directories
Icon=org.gnome.Meld
Exec=meld %U
EOF
cat << 'EOF' | ~/.local/share/kio/servicemenus/meld.file.service.desktop
[Desktop Entry]
Type=Service
MimeType=application/octet-stream
Actions=diffFiles
X-KDE-Priority=TopLevel
X-KDE-RequiredNumberOfUrls=2

[Desktop Action diffFiles]
Name=Compare Files
Icon=org.gnome.Meld
Exec=meld %U
EOF
chmod +x ~/.local/share/kio/servicemenus/meld.directory.service.desktop
chmod +x ~/.local/share/kio/servicemenus/meld.file.service.desktop

Using Linux MPLAB X IDE on High-DPI Screen

These days scaling usually works even under Linux. However, there are always a few stubborn applications that evade scaling and unfortunately one of them is MPLAB X IDE I use for PIC microcontroller development.

While there are instructions on how to deal with it under Windows, it doesn’t really explain how to get an equivalent behavior under Linux other than running executable directly.

But, you can use the same fontsize trick on Linux too and make your life easier by modifying the desktop file manually. Just run the following command:

sudo sed -i -E 's|Exec=(.*)mplab_ide.*|Exec=\1mplab_ide --fontsize 17|' \
    /usr/share/applications/mplab_ide-v6.20.desktop

Assuming your version is 6.20 (otherwise adjust as needed), you will get everything sized about 50% larger (default font size is 11). Now your eyes can finally relax.

Manually Installing Encrypted ZFS on Ubuntu 24.04

For my daily driver I’m fortunate enough to have mirrored ZFS setup, my secondary machine has only a single SSD slot. While that makes mirror setup unproductive, I still want to use ZFS. Yes, it cannot automatically correct errors but it can at least help me know about them. And that’s before considering datasets, quotas, and beautiful snapshots.

For this setup I will use LUKS instead of the native ZFS encryption. Computer is fast enough that I don’t notice difference during daily work and total encryption is worth it for me.

As before, if you are beginner with ZFS, you might want to use Ubuntu’s built-in ZFS installatiion method. It will result in similar enough setup but without all these manual steps.

Why do I use this setup? Well, I like to setup my own partitions and I definitely love setting up my own datasets. In addition, this is the only way to make Ubuntu do a very minimal install. Lastly, I like to use hibernation with my computers and setting this during manual installation is often easier than sorting issues later.

With preamble out of the way, let’s go over all the steps to make it happen.

The first step is to boot into the USB installation and use “Try Ubuntu” option. Once on the a desktop, we want to open a terminal, and, since all further commands are going to need root access, we can start with that.

sudo -i

Next step should be setting up a few variables - disk, hostname, and username. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.

DISK=/dev/disk/by-id/<whatever>
HOST=radagast
USERNAME=josip

I want to partition disk into 4 partitions. The first two partitions are unencrypted and in charge of booting (boot + EFI). While I love encryption, I almost never encrypt the boot partitions in order to make my life easier as you cannot seamlessly integrate the boot partition password prompt with the later password prompt. Thus encrypted boot would require you to type the password twice (or thrice if you decide to use native ZFS encryption on top of that). Third partition is largest and will contain all user data. The last partition is actually swap that we need for hibernation support. Since we’re using SSD, its position doesn’t matter (on the hard drive you definitely want it way ahead) so I put it these days at the end to match my mirrored ZFS setup.

All these requirements come in the following few partitioning commands:

DISK_LASTSECTOR=$(( `blockdev --getsz $DISK` / 2048 * 2048 - 2048 - 1 ))
DISK_SWAPSECTOR=$(( DISK_LASTSECTOR - 64 * 1024 * 1024 * 1024 / 512 ))

blkdiscard -f $DISK 2>/dev/null
sgdisk --zap-all                                $DISK
sgdisk -n1:1M:+255M           -t1:EF00 -c1:EFI  $DISK
sgdisk -n2:0:+1792M           -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:$DISK_SWAPSECTOR -t3:8309 -c3:LUKS $DISK
sgdisk -n4:0:$DISK_LASTSECTOR -t4:8200 -c4:Swap $DISK
sgdisk --print                                  $DISK

The next step is to setup all LUKS partitions. If you paid attention, that means we need to repeat formatting a total of 2 times. Unless you want to deal with multiple password prompts, make sure to use the same password for each:

cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i $DISK-part3

cryptsetup luksFormat -q --type luks2 \
    --sector-size 4096 \
    --cipher aes-xts-plain64 --key-size 256 \
    --pbkdf argon2i $DISK-part4

Since creating encrypted partitions doesn’t mount them, we do need this as a separate step. I like to name my LUKS devices based on partition names so we can recognize them more easily:

cryptsetup luksOpen \
    --persistent --allow-discards \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    $DISK-part3 ${DISK##*/}-part3

cryptsetup luksOpen \
    --persistent --allow-discards \
    --perf-no_write_workqueue --perf-no_read_workqueue \
    $DISK-part4 ${DISK##*/}-part4

Finally, we can set up our ZFS pool with an optional step of setting quota to roughly 85% of disk capacity. Since we’re using LUKS, there’s no need to setup any ZFS keys. Name of the pool will match name of the host and it will contain several datasets to start with. Most of my stuff goes to either Data dataset for general use or to VirtualBox dataset for virtual machines. Consider this just a suggestion and a good starting point, adjust as needed:

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O quota=1600G \
    -O canmount=off -O mountpoint=none -R /mnt/install \
    ${HOST^} /dev/mapper/${DISK##*/}-part3

zfs create -o canmount=noauto -o mountpoint=/ \
    -o reservation=100G \
    ${HOST^}/System
zfs mount ${HOST^}/System

zfs create -o canmount=noauto -o mountpoint=/home \
           ${HOST^}/Home
zfs mount ${HOST^}/Home
zfs set canmount=on ${HOST^}/Home

zfs create -o canmount=noauto -o mountpoint=/Data \
           -o 28433:snapshot=360 \
           ${HOST^}/Data
zfs set canmount=on ${HOST^}/Data

zfs create -o canmount=noauto -o mountpoint=/VirtualBox \
           -o recordsize=32K \
           ${HOST^}/VirtualBox
zfs set canmount=on ${HOST^}/VirtualBox

zfs set devices=off ${HOST^}

With ZFS done, we might as well setup boot, EFI, and swap partitions too:

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI -i 4d65646f $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

mkswap /dev/mapper/${DISK##*/}-part4

At this time, I also often disable IPv6 as I’ve noticed that on some misconfigured IPv6 networks it takes ages to download packages. This step is both temporary (i.e., IPv6 is disabled only during installation) and fully optional:

sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
sysctl -w net.ipv6.conf.lo.disable_ipv6=1

To start the fun we need to debootstrap our OS. As of this step, you must be connected to the Internet:

apt update
apt dist-upgrade --yes
apt install --yes debootstrap
debootstrap noble /mnt/install/

We can use our live system to update a few files on our new installation:

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
rm /mnt/install/etc/apt/sources.list
cp /etc/apt/sources.list.d/ubuntu.sources /mnt/install/etc/apt/sources.list.d/ubuntu.sources
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

At last, we’re ready to chroot into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install /usr/bin/env \
    DISK=$DISK HOST=$HOST USERNAME=$USERNAME USERID=$USERID \
    bash --login

With our newly installed system running, let’s not forget to set up locale and time zone:

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux kernel. I find that hwe is a nice compromise between using generic and oem but you can use whichever you or your hardware prefers:

apt update
apt install --yes --no-install-recommends linux-generic-hwe-24.04 linux-headers-generic-hwe-24.04

Now we set up crypttab so our encrypted partitions are decrypted on boot:

echo "${DISK##*/}-part3 $DISK-part3 none \
      luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab

echo "${DISK##*/}-part4 $DISK-part4 none \
      luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab

cat /etc/crypttab

To mount all those partitions, we also need some fstab entries too. ZFS entries are not strictly needed. I just like to add them in order to hide our LUKS encrypted ZFS from the file manager:

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=3s 0 1" >> /etc/fstab

echo "/dev/mapper/${DISK##*/}-part3 \
    none auto nofail,nosuid,nodev,noauto 0 0" >> /etc/fstab

echo "/dev/mapper/${DISK##*/}-part4 \
    swap swap nofail 0 0" >> /etc/fstab

cat /etc/fstab

On systems with a lot of RAM, I like to adjust memory settings a bit. This is inconsequential in the grand scheme of things, but I like to do it anyway. Think of it as wearing “lucky” socks:

echo "vm.swappiness=10" >> /etc/sysctl.conf
echo "vm.min_free_kbytes=1048576" >> /etc/sysctl.conf

Now we can create the boot environment:

apt install --yes zfs-initramfs cryptsetup keyutils \
                  grub-efi-amd64-signed shim-signed

update-initramfs -c -k all

And then, we can get grub going. Do note we also set up booting from swap (needed for hibernation) here too. If you’re using secure boot, bootloaded-id HAS to be Ubuntu:

apt install --yes grub-efi-amd64-signed shim-signed
sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT.*/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash \
    RESUME=UUID=$(blkid -s UUID -o value /dev/mapper/${DISK##*/}-part4)\"/" \
    /etc/default/grub

update-grub

grub-install --target=x86_64-efi --efi-directory=/boot/efi \
             --bootloader-id=Ubuntu --recheck --no-floppy

I don’t like snap so I preemptively banish it from ever being installed:

apt remove --yes snapd 2>/dev/null
echo 'Package: snapd'    > /etc/apt/preferences.d/snapd
echo 'Pin: release *'   >> /etc/apt/preferences.d/snapd
echo 'Pin-Priority: -1' >> /etc/apt/preferences.d/snapd
apt update

And now, finally, we can install our minimal desktop environment:

apt install --yes ubuntu-desktop-minimal man

Since Firefox is a snapd package (banished), we can install it manually:

add-apt-repository --yes ppa:mozillateam/ppa
cat << 'EOF' | sed 's/^    //' | tee /etc/apt/preferences.d/mozillateamppa
    Package: firefox*
    Pin: release o=LP-PPA-mozillateam
    Pin-Priority: 501
EOF
apt update && apt install --yes firefox

Chrome aficionados, can install it too:

pushd /tmp
wget --inet4-only https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt install ./google-chrome-stable_current_amd64.deb
popd

For hibernation, I like to change sleep settings so that hibernation kicks in after 13 minutes of sleep:

sed -i 's/.*AllowSuspend=.*/AllowSuspend=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*AllowHibernation=.*/AllowHibernation=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*AllowSuspendThenHibernate=.*/AllowSuspendThenHibernate=yes/' \
    /etc/systemd/sleep.conf
sed -i 's/.*HibernateDelaySec=.*/HibernateDelaySec=13min/' \
    /etc/systemd/sleep.conf

For that we also need to do a minor lid switch configuration adjustment:

apt install -y pm-utils

sed -i 's/.*HandlePowerKey=.*/HandlePowerKey=hibernate/' \
    /etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitch=.*/HandleLidSwitch=suspend-then-hibernate/' \
    /etc/systemd/logind.conf
sed -i 's/.*HandleLidSwitchExternalPower=.*/HandleLidSwitchExternalPower=ignore/' \
    /etc/systemd/logind.conf

Lastly, we need to have a user too.

adduser --disabled-password --gecos '' $USERNAME
usermod -a -G adm,cdrom,dialout,dip,lpadmin,plugdev,sudo,tty $USERNAME
echo "$USERNAME ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USERNAME
passwd $USERNAME

It took a while, but we can finally exit our debootstrap environment:

exit

Let’s clean all mounted partitions and get ZFS ready for next boot:

sync
umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

After reboot, we should be done and our new system should boot with a password prompt.

reboot

Our newly installed system should boot now.


[2024-08-22: Automatically determining the last sector in DISK_LASTSECTOR variable] [2024-08-28: Removed manual user ID assignment]

Resolving ZFS Error on Alpine Linux

My alpine setup is modest and pretty usual. I installed it back in 3.19 times with the main partition using Ext4 and all data on ZFS. Why didn’t I install the root file system on ZFS? Well, I needed the machine quickly and installing using default settings instead of messing with ZFS was a faster way to do it. And, as it often happens, temporary installation became permanent.

As Alpine Linux is on 3.20.2 now, I felt my system might do well with a bit of upgrading. It’s easy after all. I just changed my /etc/apk/repositories ti point toward the latest stable repositories:

https://dl-cdn.alpinelinux.org/alpine/latest-stable/main
https://dl-cdn.alpinelinux.org/alpine/latest-stable/community

And followed this with standard update-upgrade dance:

apk update
apk upgrade

One short reboot later and my system was upgraded! Oh, yeah, and my data was gone. What gives?

Fortunately for my blood pressure, I quickly determined that my data was not really gone but just hiding as my ZFS modules weren’t loaded and all ZFS commands advised me to do modprobe zfs. I followed the same only to be greeted by an error message:

modprobe: ERROR: could not insert zfs: Invalid argument

I tried removing and readding packages, fixing them, and making many more small adjustments. Pretty much any command that the internet had to offer, I tried. But all those things always brought me back to the same cryptic message.

At the end, I decided to fall forward. If one upgrade broke the system, maybe another upgrade would solve it. And yes, someone smarter would probably go with a downgrade instead, but I don’t roll that way. It was either upgrade or reinstall for that naughty server.

Where do you upgrade from 3.20, you ask? Well, there’s always an “edge”. And upgrade to it was again just a minor change to /etc/apk/repositories followed by update/upgrade:

https://dl-cdn.alpinelinux.org/alpine/edge/main
https://dl-cdn.alpinelinux.org/alpine/edge/community

Wouldn’t you know it, edge was fine and my ZFS was buzzing along once more.

But, while I didn’t mind reinstalling this machine, keeping it on edge long-term (as my “temporary” projects tend to get), didn’t really give me a level of confidence I wanted. So I decided either 3.20 would work or I would reinstall it fully now I knew for sure my data was fine.

How do you downgrade to 3.20? Well, the first step is to change /etc/apk/repositories yet again:

https://dl-cdn.alpinelinux.org/alpine/latest-stable/main
https://dl-cdn.alpinelinux.org/alpine/latest-stable/community

The difference is that this time I wanted to force a downgrade of installed packages thus slightly modified commands:

apk update
apk upgrade -a

After a reboot, my system happily presented itself in all its 3.20 glory with ZFS loaded without any further issues.

What was the problem? I have no idea. However, apk package handling and painless upgrade/downgrade is growing on me.