Quickly Patching a Failing Ansible Setup

In my network, I use Ansible to configure both servers and clients. And yes, that includes Windows clients too. And it all worked flawlessly for a while. Out of nowhere, one Wednesday, my wife’s Surface Pro started failing its Ansible setup steps with Error when collecting bios facts.

For example:

[WARNING]: Error when collecting bios facts: New-Object : Exception calling ".ctor" with "0" argument(s): "String was not recognized as a valid DateTime."  At line:2 char:21  + ...         $bios = New-Object -TypeName
Ansible.Windows.Setup.SMBIOSInfo  +                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~      + CategoryInfo          : InvalidOperation: (:) [New-Object], MethodInvocationException      +
FullyQualifiedErrorId : ConstructorInvokedThrowException,Microsoft.PowerShell.Commands.NewObjectCommand      at <ScriptBlock>, <No file>: line 2

And yes, the full list of exceptions was a bit longer, but they all had one thing in common. They were pointing toward SMBIOSInfo.

The first order of business was to find what the heck was being executed on my wife’s Windows machine. It took some process snooping to figure out that setup.ps1 was the culprit. Interestingly, this was despite ansible_shell_type being set to cmd. :)

On my file system, I found that file at two places. However, you’ll notice that if you delete one in the .ansible directory, it will be recreated from the one in /usr/lib.

  • /usr/lib/python3/dist-packages/ansible_collections/ansible/windows/plugins/modules/setup.ps1
  • /root/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/setup.ps1

Finally, I was ready to check the script for errors, and it didn’t take me long to find the one causing all the kerfuffle I was experiencing.

The issue was with the following code:

string dateFormat = date.Length == 10 ? "MM/dd/yyyy" : "MM/dd/yy";
DateTime rawDateTime = DateTime.ParseExact(date, dateFormat, null);
return DateTime.SpecifyKind(rawDateTime, DateTimeKind.Utc);

That code boldly assumed the BIOS date uses a slash / as a separator. And that is true most of the time, but my wife’s laptop reported its date as 05.07.2014. Yep, those are dots you’re seeing. Even worse, the date was probably in DD.MM.YYYY format, albeit that’s a bit tricky to prove conclusively. In any case, ParseExact was throwing the exception.

My first reaction was to simply return null from that function and not even bother parsing the BIOS date as I didn’t use it. But then I opted to just prevent the exception as maybe that information would come in handy one day. So I added a TryParse wrapper around it.

DateTime rawDateTime;
if (DateTime.TryParseExact(date, dateFormat, null,
    System.Globalization.DateTimeStyles.None, out rawDateTime)) {
    return DateTime.SpecifyKind(rawDateTime, DateTimeKind.Utc);
} else {
    return null;
}

This code retains status quo. If it finds the date in either MM/dd/yyyy or MM/dd/yy format, it will parse it correctly. Any other format will simply return null, which is handled elsewhere in the code.

With this change, my wife’s laptop came back into the fold, and we lived happily ever after. The end.


PS: Yes, I have opened a pull request for the issue.

ZFS Root Setup with Alpine Linux

Running Alpine Linux on ZFS is nothing new as there are multiple guides describing the same. However, I found official setups are either too complicated when it comes to the dataset setup or they simply don’t work without legacy boot. What I needed was a simplest way to bring up ZFS on UEFI systems.

First of all, why ZFS? Well, for me it’s mostly the matter of detecting issues. While my main server is reasonably well maintained, rest of my lab consists of retired computers I stopped using a long time ago. As such, it’s not rare that I have hardware faults and it happened more than once that disk errors went undetected. Hardware faults will still happen with ZFS but at least I will know about them immediately and without corrupting my backups too.

In this case, I will describe my way of bringing up the unencrypted ZFS setup with a separate ext4 boot partition. It requires EFI enabled BIOS with secure boot disabled as Alpine binaries are not signed.

Also, before we start, you’ll need Alpine Linux Extended ISO for ZFS installation to work properly. Don’t worry, the resulting installation will still be a minimal set of packages.

Once you boot from disk, you can proceed with the setup as you normally would but continue with [none] at the question about installation disk.

setup-alpine

Since no answer was given, we can proceed with manual steps next. First, we can set up a few variables. While I usually like to use /dev/disk/by-id for this purpose, Alpine doesn’t install eudev by default. In order to avoid depending on this, I just use good old /dev/sdX paths.

DISK=/dev/sda
POOL=Tank

Of course, we need some extra packages too. And while we’re at it, we might as well load ZFS drivers.

apk add zfs sgdisk e2fsprogs util-linux grub-efi
modprobe zfs

With this out of way, we can partition the disk out. In this example, I use three separate partitions. One for EFI, one for /boot, and lastly, one for ZFS.

sgdisk --zap-all             $DISK
sgdisk -n1:1M:+127M -t1:EF00 $DISK
sgdisk -n2:0:896M   -t2:8300 $DISK
sgdisk -n3:0:0      -t3:BF00 $DISK
sgdisk --print               $DISK
mdev -s

While having a separate dataset for different directories sometimes makes sense, I usually have rather small installations. Thus, putting everything into a single dataset actually makes sense. Most of the parameters are the usual suspects but do note I am using ashift 13 instead of the more common 12. My own testing has shown me that on SSD drives, this brings slightly better performance. If you are using this on spinning rust, you can use 12, but 13 will not hurt performance in any meaningful way, so might as well leave it as is.

zpool create -f -o ashift=13 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O canmount=noauto -O mountpoint=/ -R /mnt ${POOL} ${DISK}3

Next is the boot partition, and this one will be ext4. Yes, having ZFS here would be “purer,” but I will sacrifice that purity for the ease of troubleshooting when something goes wrong.

yes | mkfs.ext4 ${DISK}2
mkdir /mnt/boot
mount -t ext4 ${DISK}2 /mnt/boot/

The last partition to format is EFI, and that has to be FAT32 in order to be bootable.

mkfs.vfat -F 32 -n EFI -i 4d65646f ${DISK}1
mkdir /mnt/boot/efi
mount -t vfat ${DISK}1 /mnt/boot/efi

With all that out of the way, we can finally install Alpine onto our disk using the handy setup-disk script. You can ignore the failed to get canonical path error as we’re going to manually adjust things later.

BOOTLOADER=grub setup-disk -v /mnt

With the system installed, we can chroot into it and continue the rest of the steps from within.

mount --rbind /dev  /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys  /mnt/sys
chroot /mnt /usr/bin/env DISK=$DISK POOL=$POOL ash --login

For grub, we need a small workaround first so it properly detects our pool.

sed -i "s|rpool=.*|rpool=$POOL|"  /etc/grub.d/10_linux

And then we can properly install the EFI bootloader.

apk add efibootmgr
mkdir -p /boot/efi/alpine/grub-bootdir/x86_64-efi/
grub-install --target=x86_64-efi \
  --boot-directory=/boot/efi/alpine/grub-bootdir/x86_64-efi/ \
  --efi-directory=/boot/efi \
  --bootloader-id=alpine
grub-mkconfig -o /boot/efi/alpine/grub-bootdir/x86_64-efi/grub/grub.cfg

And that’s it. We can now exit the chroot environment.

exit

Let’s unmount all our partitions.

umount -Rl /mnt
zpool export -a

And, after reboot, your system should come up with ZFS in place.

reboot

Quick and Dirty ChatGPT Proofreader

While I find ChatGPT’s reliability dubious when it comes to difficult real-life questions, I found one niche where it functions almost flawlessly - proofreading.

For many non-native speakers (or me at least), pinning down all details of English language (especially getting those pesky indefinite articles at correct places) might be difficult. ChatGPT, at least to my untrained eye, seems to do a really nice job when it comes to correcting the output.

And yes, one can use its chat interface directly to do the proofreading, but ChatGPT’s API is reasonably cheap so you might as well make use of it.

var apiEndpoint = "https://api.openai.com/v1/chat/completions";
var apiKey = "sk-XXX";

var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Authorization
    = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", apiKey);

var inputText = File.ReadAllText("<inputfile>");
inputText = "Proofread text below. Output it as markdown.\n\n"
    + inputText.Replace("\r", "");

var requestBody = new {
    model = "gpt-3.5-turbo",
    messages = new[] {
        new {
            role = "user",
            content = inputText,
        }
    }
};

var jsonRequestBody = JsonSerializer.Serialize(requestBody);
var httpContent = new StringContent(jsonRequestBody,
                                    Encoding.UTF8, "application/json");

var httpResponse = await httpClient.PostAsync(apiEndpoint, httpContent);
var responseJson = await httpResponse.Content.ReadAsStringAsync();
dynamic responseObject = JsonSerializer.Deserialize<dynamic>(responseJson);

string outputText = responseObject.GetProperty("choices")[0]
    .GetProperty("message").GetProperty("content").GetString();

Console.WriteLine(outputText);

And yes, this code doesn’t really check for errors and requires a lot more “plumbing” to be a proper application but it does actually work.

Happy proofreading!

Qemu on Ubuntu

For a long time, I used VirtualBox as my virtualization software of choice. Not only did I have experience with it, but it also worked flawlessly, whether on Windows or on Linux. That is until I faced The virtual machine has terminated unexpectedly during startup because of signal 6 error. No matter what, I couldn’t get VirtualBox working on D34010WYK NUC with Ubuntu 22.04 LTS. Well, maybe it was time to revisit my virtualization platform of choice.

My needs were modest. It had to work without a screen (i.e., some form of remote access), it had to support NAT network interface, and it had to support sharing a directory with host (i.e., shared folder functionality).

When I went over all contenders that could work on top of an existing Linux installation, that left me with two — VirtualBox and QEMU. Since I already had issues with VirtualBox, that left me only with QEMU to try.

I remember using QEMU back in the day a lot before I switched to VirtualBox, and it was OK but annoying to set up and it allowed no host directory sharing. Well, things change. Not all — it’s still focused on command-line — but among features, it now had directory sharing.

To install QEMU we need a few prerequisites but actually less than I remember:

apt install -y libvirt-daemon libvirt-daemon-system bridge-utils qemu-system-x86
systemctl enable libvirtd
systemctl start libvirtd
modprobe kvm-intel
adduser root kvm

When it comes to VM setup, you only need to create a disk (I prefer a raw one, but qcow2 and vmdk are also an option):

qemu-img create -f raw disk.raw 20G

After that, you can use qemu-system-x86_64 to boot VM for the first time:

qemu-system-x86_64 -cpu max -smp cpus=4 -m 16G \
  -drive file=disk.raw,format=raw \
  -boot d -cdrom /Temp/alpine-virt-3.17.1-x86_64.iso \
  -vnc :0 --enable-kvm

This will create a VM with 4 CPUs, 16 GB of RAM, and assign it the disk we created before. In addition, it will boot from CDROM containing installation media. To reach its screen we can use VNC on port 5900 (default).

Once installation is done, we power off the VM, and any subsequent boot should omit CDROM:

qemu-system-x86_64 -cpu max -smp cpus=4 -m 16G \
  -drive file=disk.raw,format=raw \
  -virtfs local,path=/shared,mount_tag=sf_shared,security_model=passthrough \
  -vnc :0 --enable-kvm

And that’s all there is to it. As long as our command is running, our VM is too. Of course, if we want to run it in the background, we can do that too by adding the -daemonize parameter.

Unlike with VirtualBox, within the host, we don’t need any extra drivers or guest additions (at least not for Alpine Linux). We just need to mount the disk using 9p file system:

mount -t 9p -o trans=virtio sf_shared /media/sf_shared

If we want to make it permanent, we can add the following line into /etc/fstab:

sf_shared /media/sf_shared 9p trans=virtio,allow_other 0 0

With this, we have an equivalent setup to the one I used VirtualBox for. And all that with less packages installed and definitely with less disk space used.

And yes, I know QEMU is nothing new, and I remember playing with it way before I went with VirtualBox. What changed is support QEMU enjoys within guest VMs and how trouble-free its setup got.

In any case, I’ve solved my problem.

Mikrotik and ED25519 Keys

Well, it seems miracles do happen. According to the 7.9 testing release notes, Mikrotik will finally support ED25519 host keys. But, is this even important? I would argue yes.

First of all, ED25519 keys are MUCH shorter and significantly faster while providing higher security margin than 2048-bit RSA keys. If you want to use the same key to centrally manage your network and you have some underpowered clients, you will definitely feel RSA slowness when establishing connection - especially when dealing with high-ping situations. And shorter keys are not anything to frown upon either as they get much easier to copy/paste than wall of text RSA provides.

Secondly, security of ED25519 seems quite robust and sits somewhere between 2048-bit and 4096-bit RSA key. Unless there is a major breakthrough in cracking ED25519, this is good enough for foreseeable future. When/if quantum computers become a reality, both RSA and ED25519 are fcked so you’re in a losing battle. However, ED25519 keys seem to have a quantum-resistant NTRU-X25519 key exchange in OpenSSH while there is nothing similar for RSA.

Albeit I’m not cryptographer, I do listen to a lot of smart ones and most of them assume any quantum scaling breakthrough necessary to break ED25519 keys will buy a few years at most for RSA algorithm. In short, while both RSA and ED25519 may be doomed at undefined time in the future, it seems unnecessary to avoid faster algorithm ED25519 is today.

Lastly, for me this will mean I can use a single management key once more as, at this time, I’m using ED25519 for most of my needs with RSA being exclusively kept for the purpose of managing Mikrotik. Finally, I’ll be able to use one key to rule them all.

Good luck with upgrade!


[2023-05-04: Unfortunately, ED25519 support is partial at best. If you try to assign key to a user, you’ll get unable to load key file (wrong format or bad passphrase]

[2023-08-18: Well, while ED25519 support has been with us since 7.9, one couldn’t import any ED25519 keys. If 7.12 beta 1 release notes are to be believed (“ssh - added support for user ed25519 public keys”), we should finally have it done fully and properly. Let’s see…]

[2023-11-15: At last, ED25519 is supported by Mikrotik as of RouterOS 7.12]