Linux, Unix, and whatever they call that world these days

Case-insensitive ZFS

Don’t.

Well, this was a short one. :)

From the very start of its existence, ZFS supported case-insensitive datasets. In theory, if you share disk with Windows machine, this is what you should use. But reality is a bit more complicated. It’s not that setting doesn’t work. It’s more a case of working too well.

Realistically you are going to be running ZFS on some *nix machine and access it from Windows, it’ll be via Samba. As *nix API generally expects case-sensitivity, Samba will dynamically convert what it shares from the case-sensitive world into the case-insensitive one. If file system is case-insensitive, Samba will get confused and you will suddenly have issues renaming files that differ only in case.

For example, you won’t be able to rename test.txt into Test.txt. Before doing rename Samba will if the new file already exists (step needed if underlying system is case-insensitive) in order to avoid overwriting unrelated file. This second check will fail on case-insensitive dataset as ZFS will report Test.txt exists. Because of this check (that would be necessary on case-sensitive file system) Samba will incorrectly think that destination already exists and not allow the rename. Yep, any rename differing only in case will fail.

Now, this could be fixable. If Samba would recognize the file system is case-insensitive, it could skip that check. But what if you have case-sensitive file system mounted within case-insensitive dataset? Or vice-versa? Should Samba check on every access or cache results? For something that doesn’t happen on *nix often, this would be either flaky implementation or a big performance hit.

Therefore, Samba assumes that file system is case-sensitive. In 99% of cases, this is true. Unless you want to chase ghosts, just give it what it wants.

Changing ZFS Key Location

Back when I was creating my original pool, I decided to use password prompt as my encryption key unlocking method. And it was good. But then I wanted to automate this a bit. I wanted my key to be read of USB drive.

To do that one can simply prepare a new key and point the pool toward it.

dd if=/dev/urandom of=^^/usb/key.dat^^ bs=32 count=1
zfs change-key -o keylocation=file://^^/usb/key.dat^^ -o keyformat=raw Pool

Of course, it’s easy to return it back to password prompt too:

zfs change-key -o keylocation=prompt -o keyformat=passphrase Pool

Simple enough.

SDR on Ubuntu x86

[This is a post 3 in two-part series :), for hardware setup go here]

While running SDR radio on Raspberry was fine, I kinda wanted to move this to one of my x86 servers. Due to this, I had to revisit my old guide.

When device is plugged in, trace should be seen in dmesg. If everything is fine, you should see some activity.

dmesg | tail[4306437.661393] usbcore: registered new interface driver dvb_usb_rtl28xxu

To get SDR running, there is some work involved with its compilation. Note the DETACH_KERNEL_DRIVER=ON flag enabling SDR application to access device without disabling its driver. Rest is really similar to official instructions.

sudo apt-get install -y git build-essential cmake libusb-1.0-0-dev libglib2.0-dev
cd ~
git clone git://git.osmocom.org/rtl-sdr.git
cd rtl-sdr/
mkdir build
cd build
cmake ../ -DINSTALL_UDEV_RULES=ON -DDETACH_KERNEL_DRIVER=ON
make
sudo make install
sudo ldconfig

This is an ideal time to test it. As I have the iptables active, I manually enable port on external interface. Other than that I will not restrict application to a single IP but allow it to listen on all interfaces.

iptables -A INPUT -i ^^eth0^^ -p tcp --dport 1234 -j ACCEPT
/usr/local/bin/rtl_tcp -a 0.0.0.0

The last step is to enable running it as a service. We need to create a separate user, enable service, and finalize it all with reboot.

sudo adduser --disabled-password --gecos "" sdr
sudo usermod -a -G plugdev sdr

sudo cat > /lib/systemd/system/rtl_tcp.service <<- EOF
[Unit]
After=network.target

[Service]
Type=exec
ExecStart=/usr/local/bin/rtl_tcp -a 0.0.0.0
KillMode=process
Restart=on-failure
RestartSec=10
User=sdr

[Install]
WantedBy=multi-user.target
Alias=rtl_tcp.service
EOF

sudo systemctl enable /lib/systemd/system/rtl_tcp.service
sudo reboot

And that’s it. Now you can run SDR TCP server on your Ubuntu server.

Testing Native ZFS Encryption Speed (20.10)

[2020-11-02: There is a newer version of this post]

Illustration

Back in the days of Ubuntu 20.04, I did some ZFS native encryption testing. Results were not promising to say the least but they were done using ZFS 0.8.3 on Ubuntu 20.04. There was a hope that Ubuntu 20.10 bringing 0.8.4 would have a lot of performance improvements. So I repeated my testing.

First I tested CCM and saw that results were 10-15% lower than in 20.04. However, this was probably not due to ZFS changes as both Luks and no-encryption numbers dropped too. As my testing was done on a virtual machine, it might not be anything related to Ubuntu at all. For all practical purposes, you can view those results as unchanged.

However, when I tested GCM encryption speed, I had to repeat test multiple times because I couldn’t believe the results I was seeing. ZFS native encryption using GCM was only about 25% slower than no encryption at all and handily beating Luks numbers. Compared to the last year’s times, GCM encryption got a fivefold improvement. That’s what I call optimization.

Last year I suggested going with native ZFS encryption only when you are really interested in ZFS having direct physical access to drives or if you were interested in encrypted send/receive. For performance critical scenarios, Luks was the way to go.

Now I can honestly recommend going with the native ZFS encryption (provided you use GCM). It’s as fast as Luks, allows ZFS to handle physical drives directly, and simplifies the setup. The only scenario where Luks still matters is if you want to completely hide your disk content as native encryption does leak some metadata (e.g., dataset properties). And no, you don’t need to upgrade to 20.10 for the speed as some performance improvements have been backported to 20.04 too.

I have migrated my own main file server to ZFS native encryption some time ago mostly to give ZFS direct disk access and without much care for array speed. Now there is no reason not to use it on desktop either.


PS: You can take a peek at the raw data if you’re so inclined.

PPS: Test procedure is in the previous post so I didn’t bother repeating it here.

Windows Product Key in VirtualBox BIOS

As many who use virtual machines for testing, I often need to reinstall the same. One thing that annoys me when getting Windows reinstalled is the prompt for the product key. Yes, you can bypass it temporarily but you still need to enter it when you go and activate the Windows. My laptop has product key embedded in BIOs. Why cannot I do the same with virtual machine? Well, maybe I can.

Investigation started by looking into where exactly the key is located on physical hardware. It was relatively easy to discover this was in ACPI MSDM table. Microsoft even has instructions on how to create the table. If you look a bit further, you can even find description on the MSDM fields.

OffsetLengthNameValue
04SignatureAlways MSDM.
44LengthTotal length is always 0x55.
81RevisionAlways 3.
91ChecksumChecksum over the whole table.
106OEM IDAnything goes; pad with spaces.
168OEM Table IDAnything goes albeit ASCII text is customary.
244OEM RevisionAny number will do but keep it positive (little-endian).
284Creator IDAnything goes; pad with spaces.
324Creator RevisionAny number will do but keep it positive (little-endian).
364MSDM VersionAlways 1.
404ReservedAlways 0.
444Data typeAlways 1.
484ReservedAlways 0.
524Key lengthAlways 0x1D.
5629Product keyProduct key with dashes included.

It was relatively easy to figure what goes in which field except for the checksum. I didn’t find which checksum method it uses but realistically there are only two when you have a 8-bit checksum. Either it’s CRC or a simple sum of all fields. Well, it was a sum for this one. No matter what, sum of all fields has to be 0. If all fields except for checksum would count up to 250, the checksum would need to be 6 in order to overflow to 0. Actually a trivial thing to calculate.

With the file generated, it’s easy to add it to the VirtualBox VM.

vboxmanage setextradata "^^VM^^" "VBoxInternal/Devices/acpi/0/Config/CustomTable" ^^/msdm\_table.dat^^

Now your Windows 10 installation can proceed with the product key pre-entered and only activation pending.


PS: This just automatically fills product key during install. You still need to do the activation. This will NOT work with the pirated key.

PPS: Yes, this is essentially the same thing as adding PID.txt to the installation media. The same result but without media modification.

PPPS: And yes, of course I made a generator.

Product Key:

Resizing Fixed VirtualBox Disk

I like using fixed disks for VirtualBox VMs. It just gives me a warm fuzzy feelings when it comes to stability. However, that comes at a cost of not being able to resize it easily. If you try, you’ll get VBOX_E_NOT_SUPPORTED error.

vboxmanage modifyhd Test.vdi --resize 262144
 0%...
 Progress state: VBOX_E_NOT_SUPPORTED
 VBoxManage: error: Failed to resize medium
 VBoxManage: error: Resizing to new size 274877906944 is not yet supported
 VBoxManage: error: Details: code VBOX_E_NOT_SUPPORTED (0x80bb0009)
 VBoxManage: error: Context: "RTEXITCODE handleModifyMedium(HandlerArg*)" at line 816

But that doesn’t meant it’s impossible.

First we stop VM and clone its disk to a new image. This will automatically make it dynamic. And it will take a while.

sudo vboxmanage clonemedium "Test.vdi" "Test_new1.vdi"
 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
 Clone medium created in format 'VDI'. UUID: 96e20ba5-65f6-4248-b42e-ab48683a4cf9

With dynamic disk, we can now resize disk image.

sudo vboxmanage modifymedium disk "Test_new1.vdi" --resize 262144
 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

But resizing leaves us with dynamic disk which is not what I wanted. So we convert it back to fixed. Will take long time but we again have fixed disk once done.

sudo vboxmanage clonemedium "Test_new1.vdi" "Test_new2.vdi" --variant Fixed
 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
 Clone medium created in format 'VDI'. UUID: 2bc060b6-0637-43f9-8a55-8ac6bc69ea0d

Since we want direct replacement, we need to copy old UID to the new image.

UUID=`sudo vboxmanage showmediuminfo "Test.vdi" | egrep '^UUID' | awk '{print $2}'`
sudo vboxmanage internalcommands sethduuid "Test_new2.vdi" $UUID
 UUID changed to: 4f046c99-d76a-478f-95cf-f52b07927bd0

And now we can remove intermediate step.

rm "Test_new1.vdi"

Then we copy file system properties of old image to the new one (probably not needed but it doesn’t hurt).

sudo chmod --reference="Test.vdi" "Test_new2.vdi"
sudo chown --reference="Test.vdi" "Test_new2.vdi"

And finally we replace the old image with the new one.

mv "Test_new2.vdi" "Test.vdi"

This whole process will take ages and it will require enough space to copy disk twice. Increasing disk from 128 to 256 GB required 128+128+256 - essentially 4x the disk space. But sometime roundabout way is the only way. :)

Headless VirtualBox on Ubuntu Server 20.04

As expected, first we need to install VirtualBox.

sudo wget -q https://www.virtualbox.org/download/oracle_vbox_2016.asc -O- | sudo apt-key add -
echo "deb [arch=amd64] https://download.virtualbox.org/virtualbox/debian focal contrib" \
    | sudo tee /etc/apt/sources.list.d/virtualbox.list
sudo apt update
sudo apt-get install --yes virtualbox-6.1
sudo systemctl status vboxdrv

sudo usermod -aG vboxusers $USER

Since our server is headless, we will need a remote acces. This means installing Oracle’s extension pack with RDP support. You should be good installing this on your home system but you will need license for production deployments.

VBOXVER=`vboxmanage -v | cut -dr -f1`
wget -P /tmp \
    https://download.virtualbox.org/virtualbox/$VBOXVER/Oracle_VM_VirtualBox_Extension_Pack-$VBOXVER.vbox-extpack
vboxmanage extpack install /tmp/Oracle_VM_VirtualBox_Extension_Pack-$VBOXVER.vbox-extpack

With installation ready, we can approach creating VM. I’ll give an example of the Ubuntu Desktop but really anything goes.

mkdir -p /srv/virtualbox

vboxmanage createvm \
    --ostype Ubuntu_64 \
    --basefolder "^^/srv/virtualbox^^" \
    --register \
    --name "^^Test^^"

vboxmanage modifyvm "^^Test^^" \
    --memory 1024 \
    --nic1 nat \
    --vrde on --vrdeport ^^33890^^

vboxmanage createhd \
    --filename "^^/srv/virtualbox/Test/Test.vdi^^" \
    --format VDI --size ^^10240^^

vboxmanage storagectl "^^Test^^" \
    --name "SATA" \
    --add sata

vboxmanage storageattach "^^Test^^" \
    --storagectl SATA --port 0 --type hdd \
    --medium "/srv/virtualbox/Test/Test.vdi"

vboxmanage storageattach "^^Test^^" \
    --storagectl SATA --port 15 --type dvddrive \
    --medium ^^/tmp/ubuntu-20.04-desktop-amd64.iso^^

If you are using firewall, make sure to allow port through. For example, iptables would need the following adjustment.

iptables -A INPUT -p tcp --dport 33890 -j ACCEPT

With this, we’re finally ready to start the virtual machine.

vboxmanage startvm "^^Test^^" --type headless

To connect to it just use any Remote Desktop application.


PS: Authentication for RDP is highly recommended and you should probably check documentation to see what works best for you.

Manually Installing Ubuntu 20.04 on Surface Go

I love ZFS but it definitely doesn’t fit every situation. One situation it doesn’t fit is Surface Go. Not only device is low on RAM but it’s also low on disk space. And ZFS really hates when it doesn’t have enough disk space.

Now, one can install Ubuntu perfectly well without any shenanigans. Just follow a guide on how to boot install USB and you’re golden. But I like my installations to be a bit special. :)

After booting into Ubuntu desktop installation one needs a root prompt. All further commands are going to need root credentials anyhow.

sudo -i

The very first step should be setting up a few variables - disk, pool, host name, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.

DISK=/dev/disk/by-id/^^ata_disk^^
HOST=^^desktop^^
USER=^^user^^

Disk setup is really minimal .

blkdiscard $DISK

sgdisk --zap-all                       $DISK

sgdisk -n1:1M:+63M -t1:EF00 -c1:EFI    $DISK
sgdisk -n2:0:+448M -t2:8300 -c2:Boot   $DISK
sgdisk -n3:0:0     -t3:8309 -c3:Ubuntu $DISK

sgdisk --print                         $DISK

I usually encrypt just the root partition as having boot partition unencrypted does offer advantages and having standard kernels exposed is not much of a security issue.

cryptsetup luksFormat -q --cipher aes-xts-plain64 --key-size 512 \
    --pbkdf pbkdf2 --hash sha256 $DISK-part3

Since crypt device name is displayed on every startup, for Surface Go I like to use host name here.

cryptsetup luksOpen $DISK-part3 $HOST

Now we can prepare all needed partitions.

yes | mkfs.ext4 /dev/mapper/$HOST
mkdir /mnt/install
mount /dev/mapper/$HOST /mnt/install/

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

To start the fun we need debootstrap package.

apt install --yes debootstrap

And then we can get basic OS on the disk. This will take a while.

debootstrap focal /mnt/install/

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials:

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK HOST=$HOST USER=$USER \
    bash --login

Let’s not forget to setup locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes initramfs-tools cryptsetup keyutils grub-efi-amd64-signed shim-signed tasksel

Since we’re dealing with encrypted data, we should auto mount it via crypttab. If there are multiple encrypted drives or partitions, keyscript really comes in handy to open them all with the same password. As it doesn’t have negative consequences, I just add it even for a single disk setup.

echo "$HOST  UUID=$(blkid -s UUID -o value $DISK-part3)  none \
    luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
cat /etc/crypttab

To mount boot and EFI partition, we need to do some fstab setup too:

echo "UUID=$(blkid -s UUID -o value /dev/mapper/$HOST) \
    / ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we get grub started and update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

tasksel install ubuntu-desktop-minimal

Short package upgrade will not hurt.

apt dist-upgrade --yes

The only remaining task before restart is to create the user, assign a few extra groups to it, and make sure its home has correct owner.

sudo adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
passwd $USER

As install is ready, we can exit our chroot environment.

exit

And unmount our disk:

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}

After the reboot you should be able to enjoy your installation.

reboot

PS: If you are doing install on normal desktop, check similar ZFS-based installation guide.

Getting Rid of Aspeed Firmware Warning

Every time I would run update-initramfs, I would receive the following warning on my SuperMicro machine:

update-initramfs -u
 update-initramfs: Generating /boot/initrd.img-5.4.0-42-generic
 W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast

This issue is well-known and harmless. It’s just a hardcoded value that’s checked by ASPEED driver and, if file is not present, results in this warning.

If you are tired of it and you just want to make driver happy, create an empty file at that spot:

touch /lib/firmware/ast_dp501_fw.bin

And now the warning is gone. :)

IPv6 Prefix Delegation with NetPlan

No. That’s the short of it.

Longer version starts with me setting up my new Ubuntu server and deciding to use netplan to setup network interfaces. Nothing too big. Just a few bonds here and there. And all seemed fine until I tried to setup email and noticed the following message in syslog:

postfix/smtp[49721]: connect to alt1.aspmx.l.google.com[209.85.146.27]:25: Connection timed out
postfix/smtp[49721]: connect to aspmx5.googlemail.com[2607:f8b0:4002:c08::1b]:25: Network is unreachable

Once I double-checked, I noticed syslog was right - I only had link-local address assigned to interface. Ok, so DHCPv6 only turns on DHCP and not prefix delegation I use for my network. Easy-peasy, that’s surely matter of just turning on the correct setting…

Or finding a matching bug. Yep, prefix delegation that’s perfectly valid and well supported IPv6 address allocation method is not supported.

But ticket did solve my problem. As advised, I simply reverted to use networkd (don’t forget to use systemctl enable systemd-networkd).


PS: Here are my networkd files that work with IPv6 PD.

[NetDev]
Name=bond0
MACAddress=00:25:90:ba:31:40
Kind=bond

[Bond]
Mode=802.3ad
LACPTransmitRate=fast
MIIMonitorSec=100ms
TransmitHashPolicy=layer2+3
[Match]
Name=bond0

[DHCP]
RouteMetric=100
UseMTU=true

[Network]
DHCP=yes
IPv6PrefixDelegation=yes
IPv6PrivacyExtensions=true
LinkLocalAddressing=ipv6
ConfigureWithoutCarrier=yes
[Match]
PermanentMACAddress=00:25:90:ba:31:40

[Network]
Bond=lan
[Match]
PermanentMACAddress=00:25:90:ba:31:41

[Network]
Bond=lan