ZFS Encryption Speed (Ubuntu 23.10)

There is a newer version of this post

As it became a custom, I retest ZFS native encryption performance with each new Ubuntu release. Here we have results for Ubuntu 23.10 on kernel 6.5 using ZFS 2.2.

Testing was done on a Framework laptop with an i5-1135G7 processor and 64GB of RAM. Once booted into installation media, I execute the script that creates 42 GiB RAM disk that hosts all data for six 6 GiB files. Those files are then used in RAIDZ2 configuration to create a ZFS pool. The process is repeated multiple times to test all different native ZFS encryption modes in addition to a LUKS-based test. This whole process is repeated again with AES disabled and later with a reduced core count.

Since I am testing on the same hardware as for 23.04 and using essentially the same script, I expected similar results but I was slightly surprised to see both raw read and write speed has been reduced by more than 10%. I am not sure if this is due to the new kernel, new BIOS, or some other combination of changes but the performance hit seems quite significant.

However, what I’m most interested in is not necessarily the actual speed but how it’s impacted by encryption. As compared to last year, it seems each GCM encryption mode has taken a few percent hit. We’re still talking about 2 GiB/s for both read and write so I’m not too worried.

Interestingly, while key size had more impact before, it seems that with 23.10 you can count on the same speed regardless if you select 128, 192, or 256 bit key.

If you don’t have AES support and you need CCM, the news is not that good as that code path has gotten significantly worse. Unless you’re stuck on an ancient CPU this is irrelevant I guess as you should never opt for CCM in the first place.

Using ZFS on top of LUKS has gotten slightly better when it comes to writes where it actually lagged the most behind the native ZFS. The improvement is significant but we’re still talking about 30% lower speeds. On read size, there are no changes and it’s the only area where LUKS wins over the native ZFS encryption.

For this release, I also experimentally tried to get power usage for each test run. I did the same by disconnecting the battery and measuring the power the laptop was drawing. This is not the most precise way of measuring it so I might be off but it looked as ZFS encryption was as efficient as it gets when it comes to the power usage.

To summarize, the native ZFS encryption is still live and kicking in Ubuntu 23.10 and might even provide some power usage advantages as compared to LUKS.

PS: You can find older tests for Ubuntu 23.04, 22.10, 22.04, 20.10, and 20.04.

Counting Geigers

A long while ago I got myself a Geiger counter, soldered it together, stored it in a drawer, and forgot about it for literally more than a year. However, once I found it again, I did the only thing that could be done - I connected it to a computer and decided to track its values.

My default approach would usually be to create an application to track it. But, since I connected it to a Linux server, it seemed appropriate to merge this functionality into a bash script that already sends ZFS data to my telegraf server.

Since my Geiger counter has UART output, the natural way of collecting its output under Linux was a simple cat command:

cat /dev/ttyUSB0

That gave me a constant output of readings, about once a second:

CPS, 0, CPM, 12, uSv/hr, 0.06, SLOW

CPS, 1, CPM, 13, uSv/hr, 0.07, SLOW

CPS, 0, CPM, 13, uSv/hr, 0.07, SLOW

CPS, 1, CPM, 14, uSv/hr, 0.07, SLOW

CPS, 1, CPM, 15, uSv/hr, 0.08, SLOW

CPS, 0, CPM, 15, uSv/hr, 0.08, SLOW

In order to parse this line easier, I wanted two things. First, to remove extra empty lines caused by CRLF line ending, and secondly to have all values separated by a single space. Simple adjustment sorted that out:

cat /dev/ttyUSB0 | sed '/^$/d' | tr -d ','

While this actually gave me a perfectly usable stream of data, it would never exit. It would just keep showing new data. Perfectly suitable for some uses, but I wanted my script just to take the last data once a minute and be done with it. And no, you cannot just use tail command alone for this - it needs to be combined with something that will stop the stream - like timeout.

timeout --foreground 1 cat /dev/ttyUSB0 | sed '/^$/d' | tr -d ',' | tail -1

If we place this into a variable, we can extract specific values - I just ended up using uSv/h but the exact value might depend on your use case.

OUTPUT=`timeout --foreground 1 cat /dev/ttyUSB0 | sed '/^$/d' | tr -d ',' | tail -1`
CPS=`echo $OUTPUT | awk '{print $2}'`
CPM=`echo $OUTPUT | awk '{print $4}'`
USVHR=`echo $OUTPUT | awk '{print $6}'`
MODE=`echo $OUTPUT | awk '{print $7}'`

With those variables in hand, you can feed whatever upstream data sink you want.

Randomizing Serial Number During MPLAB X Build

Quite often, especially when dealing with USB devices, a serial number comes really handy. One example is my TmpUsb project. If you don’t update its serial number, it will still work when only one is plugged in. But plug in two, and all kinds of shenanigans will ensue.

This is the exact reason why I already created a script to randomize its serial number. However, that script had one major fault - it didn’t work under Linux. So, it came time to rewrite the thing and maybe adjust it a bit.

The way the original script worked was as a post-build step in MPLAB X project that would patch the Intel Hex object file and that’s something I won’t change as it integrates flawlessly with programming steps.

The resulting serial number was 12 hexadecimal digits in length (48 bits or random data) and that was probably excessive. Even worse, that led to an overly complicated script that was essentially a C# program to patch hex after the modification was done since any randomization always impacted more than 1 line. As I wanted a solution that could work anywhere the Bash can run (even Windows), I wanted to make my life easier by limiting any change to a single line.

Well, first things first, to limit serial number length, I had to figure out how much of a serial number can fit in one. Looking at the Intel hex produced by MPLAB X, we can see that each line is 16 bytes, which means any serial number intended for USB consumption can be up to 8 characters. However, what if other code pushes the serial number further down the line? Well, now you get only a single character.

What we need is a way to fix the serial number location. The solution to this is in the __at keyword. Using it, we can align our string descriptors wherever we want them. In the TmpUsb example, that would be something like this

const struct {
    uint8_t bLength;
    uint8_t bDscType;
    uint16_t string[7];
} sd003 __at(0x1000) = {
    sizeof(sd003),
    USB_DESCRIPTOR_STRING,
    { '2','8','4','4','3','4','2' }
};

The whole USB descriptor structure has to fit into 16 bytes as to limit any subsequent modification to the single line. The first 2 bytes are length and type bytes, leaving us with 14 bytes for our serial. Since USB likes 16-bit unicode, this means we have 7 characters to play with. If we stay in the hexadecimal realm, this provides 28 bits of randomization. Not bad, but we can slightly improve on it by expanding character selection a bit.

That’s where base32 comes in. It’s a nice enough encoding that isn’t case sensitive and it omits most easily confused characters. And yes, it would take 40 bits to fully utilize base32 but trimming it at 7 will leave you with 35 bits which is plenty.

How do I get this serial number in an easy way? Well, getting 5 bytes using dd and then passing it to base32 will do.

NEW_SERIAL=`dd if=/dev/urandom bs=5 count=1 2>/dev/null | base32 | cut -c 1-7`

If we pass the non-random serial number in, with some mangling to get it expanded, it is trivial to swap it with the new one using sed:

LINE=`cat "$INPUT" | grep "$SERIAL_UNICODE"`
NEW_LINE=`echo -n "$LINE" | sed "s/$SERIAL_UNICODE/$NEW_SERIAL_UNICODE/g"`
sed -i "s/$LINE/$NEW_NEW_LINE/g" "$INPUT"

But no, this is no good. Changing content of a line will invalidate checksum character that comes at the end. To make this work we need to adjust that checksum. Thankfully, checksum is just a sum of all characters preceding with a bit of inversion as a last step. Something like this:

CHECKSUM=0
for ((i = 1; i < $(( ${#NEW_LINE} - 2 )); i += 2)); do
    BYTE_HEX="${NEW_LINE:$i:2}"
    NEW_NEW_LINE="$NEW_NEW_LINE$BYTE_HEX"
    BYTE_VALUE=$(printf "%d" 0x$BYTE_HEX)
    CHECKSUM=$(( (CHECKSUM + BYTE_VALUE) % 256 ))
done
CHECKSUM_HEX=`printf "%02X" $(( (~CHECKSUM + 1) % 256 )) | tail -c 2`

Once all this is combined into a script, we can call it by giving it a few arguments, most notably, project directory (${ProjectDir}), image path (${ImagePath}), and our predefined serial number (2844342):

bash "${ProjectDir}/../package/randomize-usb-serial.sh"
  "${ProjectDir}"
  "${ImagePath}"
  "2844342"

Script is available for download here but you can also see it in action as part of TmpUsb.

MPLABX and the Symbol Lookup Error

After performing the latest Ubuntu upgrade, my MPLAB X v6.15 installation stopped working. It would briefly display the splash screen and then crash. When attempting to run it manually, I encountered a symbol lookup error in libUSBAccessLink_3_38.so.

$ /opt/microchip/mplabx/v6.15/mplab_platform/bin/mplab_ide

/opt/microchip/mplabx/v6.15/sys/java/zulu8.64.0.19-ca-fx-jre8.0.345-linux_x64/bin/java: symbol lookup error: /tmp/mplab_ide/mplabcomm4864006927221691126/mplabcomm5312997795113373971libUSBAccessLink_3_38.so: undefined symbol: libusb_handle_events_timeout_completed

Since there were a few links to the old MPLAB directory, I cleaned up /usr/lib a bit, but that didn’t help. What did help was removing the older libusb altogether.

sudo rm /usr/local/lib/libusb-1.0.so.0

With that link out of the way, everything started working once again.

Overthinking the LED Blinking

The very first thing I add to every piece of hardware I make is an LED. It not only signals that the device is on, but you can also use one to show processing or even do some simple debugging. What I often go for is a simple resistor+LED pair. And here is where the first decision comes - do I go for high-side or low-side drive?

High side drive is when your output pin goes into the anode of the LED. The exact resistor location doesn’t really matter, and either before or after the LED results in the same behavior. When your pin goes high, the LED lights up. When the pin goes low, the LED turns off.

Low side drive is pretty much the same idea but with logic reversed. When the pin goes low, the LED turns on, while the pin going high turns it off.

With both being essentially the same, why would you select one over the other? Honestly, there is no difference worth mentioning. It all goes to personal preference and/or what is easier to route on the PCB.

For open-drain LED control we make use of the tri-state output available on most microcontrollers (high-Z state simulated using a switch). In addition to the standard low and high setup, we have an additional one sometimes called high-Z (or high impedance). In Microchip PIC, this is done by manipulating the TRIS pin. If the output TRIS is 1 (high-Z), the LED will be off. If TRIS is 0 and the output is low (the only valid open-drain state), the LED has a path to the ground and lights up.

This is quite similar to the previously mentioned low-side drive and has pretty much the same effect but with a slightly different way of controlling the pin. I would argue that both are essentially the same if your circuit uses just a single voltage. Where things change is when you have different voltages around as open-drain will work for driving LED on any rail (ok, some limits apply) while low side drive will not turn off fully if, e.g., you are controlling 12V LED with a 5V microcontroller.

But this open-drain setup got me thinking whether I can have use something similar to have LED be on by default? Why, you wonder? Well, my first troubleshooting step is to see if input voltage is present. My second troubleshooting step is to see if microcontroller is booting. Quite often, those tasks fall onto two different LEDs. If I could have LED on by default, that would tell me if there’s any voltage to start with and then blinking boot sequence would tell me microcontroller is alive and programmed.

Well, there is one way to do it, and I call it default on open-drain. Unprogrammed Microchip PIC has pins in tri-state (high-Z) configuration. Thus, current will flow through LED as soon as voltage is applied. However, if pin goes low, current will flow to the pin and thus LED will go off. Exactly what we wanted.

Albeit, nothing comes without the cost and here “cost” is twofold. The first issue is current consumption. While our “on” state is comparable to what we had with other setups, our “off” state uses even more current. For blinking it might not matter but it should be accounted for if you have it off for longer duration.

Second potential issue comes if our ping ever goes high since that will have LED pull as much current as pin can provide. While Microchip’s current limiting is usually quite capable, you are exceeding current limit and that’s never good for a long-term PIC health. And chips that have no current limiting are going to fry outright. It is critical for functioning of this LED setup that the only two states are ever present: either low or high-Z.

To avoid this danger, we can do the same setup using two resistor split. Assuming both resistors together are about the same overall value, in “on” state this circuit matches what we had before. In “off” state, we unfortunately increased current even further but, if we ever switch pin to high, we are fully protected. Even better, this setup allows for three LED states: “on”, “off”, and “bright”. This feature alone might be worth increased current consumption.

In order to deal with higher current consumption, we could reorganize resistors a bit. This circuit behaves the same as the previous one if microcontroller’s pin is high-Z (or not present). If microcontroller pin goes high, “bright” state gets even brighter as you’re essentially putting two resistors in parallel.

When pin goes low, we have those resistors in series and thus current consumption, while still present, is not exceedingly higher than what “on” state requires. And yes, having made what’s essentially a voltage divider does mean that LED is not really fully off. However, it goes dark enough that nobody will be able to tell the difference when blinking starts.

Which one is best?

Well, I still use a bog-standard resistor-LED drive the most (either low, high, or open-drain). It’s simple, it’s fool-proof and definitely has the lowest current consumption among examples given. However, for boards where remote debugging is a premium but space doesn’t allow for bigger display, I find that having three brightness settings at a cost of a single resistor is quite a useful bargain.