Small(er) Ubuntu Server 22.04 Install on Vultr

For my new Ubuntu Server deployment I decided to go with Intel High Frequency Vultr instance, mostly due to its larger disk allotment. However, going with default Vultr’s deployment image, I ended up with 18 GB of disk occupied. And yes, I could have removed extra stuff I didn’t need (e.g. /usr/local/cuda/ was the prime candidate). However, I decided to go a different route - manual installation.

Getting the Ubuntu ISO is easy enough as a wide selection is already available behind ISO Library tab on Server Image selection page. Combine that with noVNC console and you can just click your way through. However, one can also find Shell option within the Help menu giving you access to the bash prompt and allowing for more control.

While noVNC is decent, the lack of copy/paste makes it unwieldy when more then a few commands need to be entered. So, my first task was to SSH into the installation and continue from there. To do this, we have to allow for password login and set the password.

sed -i 's/^#PermitRootLogin.*$/PermitRootLogin yes/' /etc/ssh/sshd_config
systemctl restart sshd
passwd

Now we can connect using any SSH client and do the rest of steps from there.

I like to start by setting a few variables. Here I needed only DISK and HOST.

DISK=/dev/vda
HOST=^^ubuntu^^

Assuming this is a fresh VM, the disk should already be empty but I like to clean it again (just in case) and create a single partition. On servers I quite often skip swap partition and that’s the case here too.

blkdiscard -f $DISK
echo -e "o\nn\np\n1\n\n\nw" | fdisk $DISK
fdisk -l $DISK

Now we can format our partition and mount it into /mnt/install/.

# mkfs.ext4 ${DISK}1
# mkdir /mnt/install
# mount ${DISK}1 /mnt/install/

We will need to install debootstrap to get our installation going.

# apt update ; apt install --yes debootstrap

And now we can finally move installation files to our disk.

debootstrap $(basename `ls -d /cdrom/dists/*/ | grep -v stable | head -1`) /mnt/install/

Before proceeding, we might as well update a few settings.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

Finally we get to chroot into our newly installed system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK HOST=$HOST \
    bash --login

While optional, I like to update locale settings and set the time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales
dpkg-reconfigure tzdata

Installing kernel is next.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by our boot environment packages.

apt install --yes initramfs-tools grub-pc

Setting FSTab will ensure disk is mounted once done.

echo "PARTUUID=$(blkid -s PARTUUID -o value ${DISK}1) \
    / ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" > /etc/fstab
cat /etc/fstab

Of course, boot wouldn’t be possible without getting images ready.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

And finally, boot needs Grub installed to MBR.

update-grub
grub-install ${DISK}

While we could boot our system already, it’s a bit too bare for my taste so I’ll add a basic set of packages.

apt install --yes ubuntu-server-minimal man

Once done, a small update will not hurt.

apt update ; apt dist-upgrade --yes

Of course, we need to ensure we can boot into our system. While one could use passwd to set the root password, I like to use keys for access.

mkdir /root/.ssh
echo '^^ssh-ed25519 AAAA...^^' > /root/.ssh/authorized_keys
chmod 600 -R /root/.ssh

With all set, we can exit our chroot environment.

exit

We might as well unmount our custom install.

mount | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}

And finally we get to reboot. Please note that you should also go to VM Settings and unmount custom ISO to avoid getting into installation again.

reboot

If all steps worked, you will face a pristine Ubuntu installation measuring something around 5 GB. To make it even smaller, one can turn off the swap.

swapoff -a
vi /etc/fstab
rm /swapfile

Regardless, your system is now ready for abuse. :)

You Need to Load the Kernel First

I update my Linux servers regularly. What I do less regularly is restarting them. So I was not really surprised when one of them failed to boot with “you need to load the kernel first” message. What I usually do is: select the old kernel, boot the darn thing up, and then refresh grub. However, this time I overdid it so even the attempt to boot the old kernel produced the same message. It was time for troubleshooting.

While I knew which kernels wouldn’t boot, I didn’t actually know which kernels I have available. Fortunately, grub has a command line mode hidden behind ‘c’ key press. There I selected my disk and listed all files:

ls
set root=(hd1,2)
ls /

It helps if you know how your partitions are setup for this to work but, in the worst case scenario, you can also go over each partition to find vmlinuz files. This is also the reason why I leave the boot partition unencrypted. Had my boot partition been encrypted, this step would be impossible and more involved recovery would be needed.

In this case, I was able to find vmlinuz files and I saw that the newest kernel I had installed was 5.4.0.117. Armed with that knowledge I went back to the main grub screen.

On the main screen I went to the edit option hidden behind ‘e’ keypress. There it was simple to edit existing commands (both linux and initrd) to update the kernel version to match the one I had on my disk, followed by F10 keypress. Voila! My linux was booting.

Mind you, this wasn’t a permanent solution since the next reboot would leave me with the same issue. What I needed was to update grub (and I might as well update initramfs while I’m at it).

From the prompt, two commands were enough to make my next reboot a boring affair.

sudo update-initramfs -k all -u
sudo update-grub

And that’s it. Now my grub and my kernels are back in sync. At least for a while.

Text for OpenGL

I occasionally like to learn something I literally have no use for. It keeps me happy and entertained. This month I decided to deal with OpenGL. And no, I don’t consider learn OpenGL an useless skill. OpenGL is very much useful and, despite newer technologies, still here to stay. It’s just that, since I do no game development, I have no real use for it. But I wanted to dip my toes into that world regardless.

After getting few triangles on the screen, it came time to output text and I was stunned to learn OpenGL has no real text support. And no, OpenGL is not unique here as neither Vulkan or Metal provide much support. Rendering text is simply not an integral part of rendering pipeline. And, once one gives it a thought, it’s clear it doesn’t belong there.

That’s not to say there are no ways to render text. The most common one is treating text as a texture. The less common way is rasterizing font into triangles. Since I really love bitmap fonts and square is easily constructed from two right triangles, I decided to go the rustic route.

The first issue was which font to select. I wanted something old, rather complete, and free. Due to quirks in the copyright law, bitmap fonts are generally not considered copyrightable under US law. Mind you, that holds true only for their final bitmap form. Fonts that come to you as TTF or OTF are definitely copyrightable.

The other issue with selection was completeness. While selecting old ROM font supporting code page 437 (aka US) is easy, the support for various European languages is limited, to say the least. Fortunately, here I came upon Bedstead font family which covered every language I could think off with some extra. While era-faithful setup would include upscaling and even a rudimentary anti-aliasing, I decided to go with a raw 5x9 pixel grid.

For conversion I wrote BedsteadToVertices utility that simply takes all character bitmaps and extracts them into a Vector2 array of triangles. The resulting file is essentially a C# class returning buffer that can be directly drawn. Something like this:

var triangles = BedsteadVerticesFont.GetVertices(text,
                                                 offsetX: -0.95f,
                                                 offsetY: 0.95f,
                                                 scaleX: 0.1f,
                                                 scaleY: 0.2f);
gl.BindBuffer(BufferTargetARB.ArrayBuffer, BufferId);
gl.BufferData<float>(BufferTargetARB.ArrayBuffer, triangles, BufferUsageARB.DynamicDraw);
gl.DrawArrays(GLEnum.Triangles, 0, (uint)triangles.Length / 2);

The very first naïve version of this file ended up generating a 3.8 MB source file. Not a breaking deal but quite a bit larger than I was comfortable with. So I went with a low hanging fruit first. Using float arrays instead of Vector2 instantly dropped the file size to 2.3 MB. Dropping all floats to 4 decimal places dropped it further to 2.0 MB.

And no, I didn’t think about reducing the whitespace. Code generated files don’t need to be ugly and reducing space count to a minimum would do just that. Especially because removing spaces will result in the exactly same compiled code at the expense of readability. Not worth it.

However, merging consecutive pixels into one big rectangle was yet another optimization that’s both cheap in implementation and reduces file size significantly. In my case, the end result was 1 MB for 1,500 characters. And yes, this is still a big file but if you exclude all the beautiful Unicode non-ASCII characters, that can bring file size down to 61 KB. Had I wanted to go the binary route, that would be even smaller but I was happy enough with this not to bother.

While the original Bedstead font is monospaced, I decided to throw a wrench into this and remove all extra spacing where I could do so easily. That means that font still feels monospaced but you won’t have excessive spaces really visible. And yes, kerning certain letter pairs (e.g., rl) was too out of scope.

On the OpenGL side, one could also argue that this style of bitmap drawing would be an excellent territory for the use of indices to reduce triangle count. One would be right on technicality but I opted not to complicate my life for dubious gains as modern GPUs (even the integrated ones) are quite capable of handling extra few hundred of triangles.

In any case, I solved my problem and, as always, the source code is available for download.

[2022-06-07: With a bit of optimization, ASCII-only file is at 50 KB.]

Really Prolific?

Well, it’s been a while since the FTDI fuckup so I guess it was a time for another IC supplier to go bonkers. Yes, it’s again time for a chip manufacturer to mess with your computer drivers.

Story starts with me searching for 5V USB cable with a 3.3V signal. After finding a suitable device, I did what was needed and forgot about it for a while. A few days ago I needed USB type-A serial device to do a quick loopback test and grabber the same, previously working, device. While the serial port did appear, I couldn’t open it or send any data.

A quick trip to Device Manager has shown a problem: “THIS IS NOT PROLIFIC PL2303. PLEASE CONTACT YOUR SUPPLIER.” Yes, it’s the exact nonsense that FTDI pulled years ago - using Microsoft Windows Update mechanism for their authenticity enforcement.

Now, you might thing this is their right. And I can see how they might be annoyed with fake chips using their drivers. However, their beef should be with fake chip suppliers and not with the end customer. For me the concept of bricking device owned by an unsuspected user is a bridge too far.

My case is probably the standard one. I bought device without knowing it has a fake chip in it. I paid the seller, he paid his supplied, his supplied paid the manufacturer and so on. Now my device stopped working. Money is long gone and so is the supplier of the fake chips. I might have lost that money. If I can ask for refund, the seller might be out of money. Manufacturer might be out of money (especially if they didn’t know they’re dealing with fakes). The only person not out of money is probably the guy selling fakes in the first place.

While Prolific might look at me as a potential new customer since I am in the market for a new cable, I believe that’s the wrong assumption. I am never going to knowingly buy a Prolific device again. Why? Because there is no way that I, as a customer, can check if device is indeed original or not.

What I do know is that Prolific is ready to play shenanigans with Microsoft update and brick my devices down the road. Since I cannot verify their authenticity myself, buying any Prolific device is something that might bite me in the ass. Unless something changes, I won’t buy a single Prolific cable ever again. Their product is nothing special and there are many other manufacturers happy to take my money.

I hope that Microsoft will rollback driver since it’s their update that’s causing issues for the customer. I also hope that Prolific will see the error of their ways and stop bricking customer devices. I am hoping, but not holding my breath for either.

PS: And yes, FTDI did say they saw the error of their ways back in 2014. Only to pull the same shit again in 2016. They learned nothing. Chances are neither will Prolific.

PPS: In meantime, you can download the older driver (v3.8.39.0 worked for me) and use it instead.

Installing UEFI ZFS Root on Ubuntu 22.04

Before reading further you should know that Ubuntu has a ZFS setup option since 19.10. You should use it instead of the manual installation procedure unless you need something special. In my case that special something is the native ZFS encryption, UEFI boot, and custom partitioning I find more suitable for a single disk laptop.

After booting into Ubuntu desktop installation (via “Try Ubuntu” option) we want to open a terminal. Since all further commands are going to need root credentials, we can start with that.

sudo -i

The very first step should be setting up a few variables - disk, pool, host name, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system. I like to use upper-case for ZFS pool as that’s what will appear as password prompt. It just looks nicer and ZFS doesn’t care either way.

DISK=/dev/disk/by-id/^^ata_disk^^
POOL=^^Ubuntu^^
HOST=^^desktop^^
USER=^^user^^

General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are not planning to have multiple kernels, decreasing boot partition size might be a good idea (512 MB is ok). This time I decided to also add a small swap partition. While hosting swap on top of the pool itself is a perfectly valid scenario, I actually found it sometime causes issues. Separate partition seems to be slightly better.

blkdiscard -f $DISK

sgdisk --zap-all                      $DISK
sgdisk -n1:1M:+63M  -t1:EF00 -c1:EFI  $DISK
sgdisk -n2:0:+960M  -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:+4096M -t3:8200 -c3:Swap $DISK
sgdisk -n4:0:0      -t4:BF00 -c4:Pool $DISK
sgdisk --print                        $DISK

Finally we’re ready to create system ZFS pool. Note that you need to encrypt it at the moment it’s created. And yes, I still like lz4 the best.

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
    -O canmount=off -O mountpoint=none -R /mnt/install $POOL $DISK-part4

On top of the pool we can create a root dataset.

zfs create -o canmount=noauto -o mountpoint=/ $POOL/root
zfs mount $POOL/root

Over time I went back and forth whether to use a separate home dataset or not. In this iteration, a separate dataset it is. :)

zfs create -o canmount=noauto -o mountpoint=/home $POOL/home
zfs mount $POOL/home
zfs set canmount=on $POOL/home

Assuming we’re done with datasets, we need to do a last minute setting change.

zfs set devices=off $POOL

Assuming UEFI boot, two additional partitions are needed. One for EFI and one for booting. Unlike what you get with the official guide, here I don’t have ZFS pool for boot partition but a plain old ext4. I find potential fixup works better that way and there is a better boot compatibility. If you are thinking about mirroring, making it bigger and ZFS might be a good idea. For a single disk, ext4 will do.

# yes | mkfs.ext4 $DISK-part2
# mkdir /mnt/install/boot
# mount $DISK-part2 /mnt/install/boot/

# mkfs.msdos -F 32 -n EFI -i 4d65646f $DISK-part1
# mkdir /mnt/install/boot/efi
# mount $DISK-part1 /mnt/install/boot/efi

To start the fun we need debootstrap package. Starting this step, you must be connected to the Internet.

apt update && apt install --yes debootstrap

Bootstrapping Ubuntu on the newly created pool comes next. This will take a while.

debootstrap $(basename `ls -d /cdrom/dists/*/ | grep -v stable | head -1`) /mnt/install/

We can use our live system to update a few files on our new installation.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials. Don’t worry if this returns errors - that just means you are not using wireless.

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK USER=$USER \
    bash --login

Let’s not forget to setup locale and time zone.

locale-gen --purge "en\_US.UTF-8" # update-locale LANG=en\_US.UTF-8 LANGUAGE=en\_US # dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes zfs-initramfs cryptsetup keyutils grub-efi-amd64-signed shim-signed

Since data is encrypted, we might as well use random key to encrypt our swap.

echo "swap $DISK-part3 /dev/urandom \
    swap,cipher=aes-xts-plain64,size=256,plain" >> /etc/crypttab
cat /etc/crypttab

To mount EFI, boot, and swap partitions, we need to do some fstab setup.

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "/dev/mapper/swap none swap defaults 0 0" >> /etc/fstab
cat /etc/fstab

We might as well activate the swap now.

/etc/init.d/cryptdisks restart && sleep 5
swapon -a

Now we get grub started and update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

apt install --yes ubuntu-desktop-minimal

A short package upgrade will not hurt.

add-apt-repository universe
apt update && apt dist-upgrade --yes

If one is so inclined, /home directory can get a separate dataset too but I usually skip it these days. I just proceed to create the user, assign a few extra groups to it, and make sure its home has correct owner.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
echo "$USER ALL=NOPASSWD:ALL" > /etc/sudoers.d/$USER
passwd $USER

As install is ready, we can exit our chroot environment.

exit

And cleanup our mount points.

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
umount /mnt/install/home
umount /mnt/install
zpool export -a

After the reboot you should be able to enjoy your installation.

reboot

PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 21.10 and 20.04

PPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, 19.04, and 18.10.