IDE0180: Use Tuple to Swap Values

As I upgraded my Visual Studio 2022 Preview, I noticed a new code suggestion: IDE0180. It popped next to a reasonably standard variable swap:

var tmp = a;
a = b;
b = tmp;

The new syntax it recommended was much nicer, in my opinion:

(a, b) = (b, a);

I love the new syntax!

Except it’s not new - it has been available since C# 7 (we’re at 10 now, just for reference). I just somehow missed it. The only reason I noticed it now was due to a suggestion. I guess better late than never. :)

Native PNG in C# .NET

It all started with a simple barcode. I was refactoring my old Code128 barcode class to work in .NET 5 and faced an interesting issue - there was no option to output barcode as an image. Yes, on Windows you could rely on System.Drawing but .NET is supposedly multiplatform environment these days. And no, System.Drawing is not supported in Linux - you only get error CS0234: The type or namespace name 'Drawing' does not exist in the namespace 'System' (are you missing an assembly reference?.

If you’re thinking to yourself “Wait, I remember Microsoft’s System.Drawing.Common working on Linux”, you don’t have a bad memory. However, Microsoft since made a small update and, with a stroke of a pen, decided to abandon it. Yes, you can use runtime options as a workaround but official support is no longer there.

Alternatives do exist. Most notably both ImageSharp and SkiaSharp would satisfy any graphic need I might have. But both also present quite a big dependency to pull for what is essentially a trivial task. I mean, how difficult can it be to implement PNG writer?

It turns out, not difficult at all. If you go over specification you’ll notice there are only three chunks you need to support: IHDR, IDAT, and IEND. If you know how to write those three, you can output PNG image that’s readable by all. Literally the only two things that are not straightforward were deflate compression that required an extra header and dealing with CRC.

Most of my debugging time was actually spend dealing with output not showing properly in IrfanView. I could read my output image in Paint.NET, Paint, Gimp, and multitude of other programs I’ve tried. It took me a while before I figured out that IrfanView 4.54 is actually one with the issue. Update to the latest version sorted that one out.

In any case, I successfully added PNG support to by barcode class.

And then I started thinking… Why not make a simple PNG image reader/writer? Well, answers are numerous. First of all, alternatives exist and a lone developer can never hope to have such level of completeness. Secondly, I don’t deal with graphics on a daily basis and thus features would be few and far between. And lastly, while PNG is a relatively simple specification to start with, it has a lot of optional complexity. And some of that complexity would require me to learn much more than I ever want to know (yes, ICC profiles - I’m looking at you).

However, logic be damned. What’s the better use of an afternoon than writing a PNG parser?

In any case, the final product is here and while as simple as it gets it’s relatively feature complete when it comes to loading plain old PNG images. The only mandatory feature I failed to implement is support for interlaced images.

Expectedly, the end result is usable for changing a pixel or two but not for much more as all infrastructure is missing. I will add some of the missing functionality in the future, resize and interlacing support coming first, but I consider this class reasonably feature complete for what I both need and can do without making a full blown library. And I haven’t had as much fun programming in a while.

Why My Serial Port Device Is Detected as a Mouse

When designing electronics, probably the easiest way to make it communicate with a computer is to connect it as a serial port. However, occasionally you might see your device detected as a serial mouse instead of a plain serial port. When this happens the device will obviously not work and you might even see your mouse cursor jump around.

This behavior is an artefact of how Windows detect a serial mouse since that was written way before plug’n’play times. Widnows will first raise DTR and RTS signals and then monitor if there’s an M or B character coming at 1,200 baud. If either character appears, congratulations - your device is now a mouse. And anything it sends will be processed as if it was a mouse action.

There’s a brute force solution of disabling serial mouse in registry and that would honestly be the easiest way around. However, what if you cannot modify the registry? What if you need to make your device work in the face of an unknown machine? Well, there are certain tactics you can employ to either make this behavior unlikely or avoid it altogether.

The simplest solution of not sending M or B will unfortunately not work as easily. Yes, if your device works at the 1,200 baud rate, avoiding those characters will be fine. But if your device works at faster baud rate (and it probably does) the other end receiving at 1,200 will misunderstand its output. Depending when the actual bit sample is taken and what your device is actually sending, you might see an offending character appear either every time or every 10 times. Even worse, the frequency of the issue might change depending on the computer or even the room temperature. So, not really a solution.

The second obvious choice is detecting DTR and RTS combination and pausing whatever output your device gives until that state passes. If you can afford this, it’s a great and surefire way to sort the issue out as DTR and RTS are not baud rate sensitive. The only problem happens when you need those signals for flow control or for something completely custom. Even if you’re not using them they’ll still “cost” you two pins.

The most practical way I found of stopping this issue is by having device not speak until spoken to. If you design your serial protocol so that a device must receive a valid command before it will send any data, it will pass serial mouse detection unharmed. And, if you really need a streaming output (a perfectly valid requirement), just use a command to turn it on. If you can additionally somehow get CRC check into your protocol, the world is your oyster.

Installing UEFI ZFS Root on Ubuntu 21.10

Before reading further you should know that Ubuntu has a ZFS setup option since 19.10. You should use it instead of the manual installation procedure unless you need something special. In my case that special something is the native ZFS encryption and custom partitioning I find more suitable for laptop.

After booting into Ubuntu desktop installation we want to get a root prompt. All further commands are going to need root credentials anyhow.

sudo -i

The very first step should be setting up a few variables - disk, pool, host name, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system. I like to use upper-case for ZFS pool as that’s what will appear as password prompt. It just looks nicer and ZFS doesn’t care either way.

DISK=/dev/disk/by-id/^^ata_disk^^
POOL=^^Ubuntu^^
HOST=^^desktop^^
USER=^^user^^

General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are not planning to have multiple kernels, decreasing boot partition size might be a good idea (512 MB is ok).

blkdiscard -f $DISK

sgdisk --zap-all                     $DISK

sgdisk -n1:1M:+63M -t1:EF00 -c1:EFI  $DISK
sgdisk -n2:0:+960M -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:0     -t3:BF00 -c3:Root $DISK

sgdisk --print                       $DISK

Finally we’re ready to create system ZFS pool. Note that you need to encrypt it at the moment it’s created.

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
    -O canmount=off -O mountpoint=none $POOL $DISK-part3

On top of this encrypted pool we could create our root set (as I did in previous guides) or just use default dataset itself. I found that actually works better for me. In either case, our new root file system will end up at /mnt/install.

zfs set canmount=noauto $POOL
zfs set mountpoint=/ $POOL

mkdir /mnt/install
mount -t zfs -o zfsutil $POOL /mnt/install

Assuming UEFI boot, two additional partitions are needed. One for EFI and one for booting. Unlike what you get with the official guide, here I don’t have ZFS pool for boot partition but a plain old ext4. I find potential fixup works better that way and there is a better boot compatibility. If you are thinking about mirroring, making it bigger and ZFS might be a good idea. For a single disk, ext4 will do.

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

To start the fun we need debootstrap package.

apt install --yes debootstrap

Bootstrapping Ubuntu on the newly created pool is next. This will take a while.

debootstrap $(basename `ls -d /cdrom/dists/*/ | grep -v stable | head -1`) /mnt/install/

We can use our live system to update a few files on our new installation.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials. Don’t worry if this returns errors - that just means you are not using wireless.

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK POOL=$POOL USER=$USER \
    bash --login

Let’s not forget to setup locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes zfs-initramfs grub-efi-amd64-signed shim-signed tasksel

To mount boot and EFI partition, we need to do some fstab setup.

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we get grub started and update our boot environment.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

tasksel install ubuntu-desktop-minimal

Short package upgrade will not hurt.

apt dist-upgrade --yes

We can omit creation of the swap dataset but I personally find a small one handy.

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
    -o sync=always -o primarycache=metadata -o secondarycache=none $POOL/Swap
mkswap -f /dev/zvol/$POOL/Swap
echo "/dev/zvol/$POOL/Swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

If one is so inclined, /home directory can get a separate dataset too but I usually skip it these days. I just proceed to create the user, assign a few extra groups to it, and make sure its home has correct owner.

adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
passwd $USER

As install is ready, we can exit our chroot environment.

exit

And cleanup our mount points.

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
umount /mnt/install
zpool export -a

After the reboot you should be able to enjoy your installation.

reboot

PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 22.04 and 20.04

PPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, 19.04, and 18.10.

ZFS Pool on SSD

I am a creature of habit. Long time ago I found ZFS setup that works for me and didn’t change much since. But sometime I wonder if those settings still hold with SSDs in game. Most notably, are 4K blocks still the best?

Since I already “had” to update my desktop to Ubuntu 21.10, I used that opportunity to clear my disks and have it installed from scratch. And it would be a shame not to run some tests first on my XPG SX6000 Pro - SSD I use for pure data storage. After trimming this DRAM-less SSD, I tested the pool across multiple recordsize values and at ashift values of 12 (4K block) and 13 (8K block).

My goal was finding a good default settings for both bulk storage and virtual machines. Unfortunately, those are quite opposite requirements. Bulk storage benefits greatly from good sequential access while virtual machines love random IO more. Fortunately, with ZFS, one can accomplish both using two datasets with different recordsize values. But ashift value has to be the same.

Due to erase block sizes getting larger and larger, I expected performance to be better with 8K “sectors” (ashift=13) than what I usually used (ashift=12). But I was surprised.

First of all, results were all over the place but it seems that ashift=12 is still a valid starting point. It might be due to my SSD having smaller than expected erase page but I doubt it. My thoughts go more toward SSDs being optimized for the 4K load. And the specific SSD I used to test with is DRAM-less thus allowing any such optimizations to be even more visible.

Optimizations are probably also the reason for 128K performing so well in the random IO scenarios. Yes, for sequential access you would expect it, but for random access it makes no sense how fast it is. No matter what’s happening, it’s definitely making recordsize=128K still the best general choice. Regardless, for VMs, I created a sub-dataset with much smaller 4K records (and compression off) just to lower write-amplification a bit.

The full test results are in Google Sheets. For testing I used fio’s fio-rand-RW.fio and fio-seq-RW.fio profiles.