Endianness Fun in C#

Those doing parsing of network protocols will be familiar with BitConverter. For example, to read an integer that’s written in big-endian (aka network) order, one could write something like this:

Array.Reverse(buffer, offset, 4);
Console.WriteLine(BitConverter.ToInt32(buffer, offset));

If one is dealing with multiplatform code, they could even go with a slightly smarter code:

if (BitConverter.IsLittleEndian) { Array.Reverse(buffer, offset, 4); }
Console.WriteLine(BitConverter.ToInt32(buffer, offset));

However, that’s the old-style way of doing things. For a while now, .NET also offers BinaryPrimitives class in System.Buffers.Binary namespace. While this class was originally designed to be used with protocol buffers (which explains the namespace), there is no reason why you couldn’t use it anywhere else. Actually, considering how versatile the class is, I am stunned they didn’t just add it in the System namespace.

In any case, reading our 32-bit number is now as easy as:

Console.WriteLine(BinaryPrimitives.ReadInt32BigEndian(buffer));

And yes, this doesn’t accept offset argument. Boo-hoo! Just use Span and slice it.

var span = new ReadOnlySpan<byte>(buffer);
Console.WriteLine(BinaryPrimitives.ReadInt32BigEndian(span.Slice(offset)));

Much easier than remembering if you need to reverse array or not. :)

Calculating Mean for Microchip PIC

Averaging ADC readings on microcontroller is often needed but also often quite problematic. The easiest way to do it is remembering the last N values, summing them together and simply dividing by count. Simple, reliable, and uses a BUNCH of memory. If you want to store 50 16-bit values, you will need 100 bytes. When dealing with Microchip PIC, that’s a fortune.

However, if you’re ok with a mean rather with an average, there’s an easier way (and I would argue that mean is a better measure anyways in 90% of cases). You can use Welford’s online algorithm to get mean value using only 3 bytes of static storage and a few bytes more when executing. Of course, use of such algorithm in microcontroller environment will require a bit of a trade off (the least of them is me using average and mean interchangeably further down in post).

For my case, I assumed I need it for unsigned 10-bit ADC values. In the context of Walford algorithm, this means my intermediate values can be up to 15-bits (16-bit signed integer is needed internally) and this leaves 5 bits to use as a fixed decimal point. My case also called just “averaging” a handful of values, so I was ok with a 8-bit counter.

This means my variables are as follows:

uint8_t walfordCount = 0;
int16_t walfordMeanEx = 0;

As compared to the standard Walford’s algorithm, a few things are different. First of all, if we have more than N entries, it will reduce count to half over and over again. This trickery is allowing for “sorta” running average. And yes, you shouldn’t really use Walford’s for running average but it’s close enough. Secondly, when adding the value into the internal variable, we convert it to the signed integer and shift it 5-bits to simulate a decimal point. The only expensive (in CPU terms) operation here is dividing by count as everything else is quite cheap (shift, subtractions, and comparisons).

void walford_add(uint16_t value) {
    walfordCount += 1;
    if (walfordCount > WALFORD_CUTOFF) { walfordCount >>= 1; }
    int16_t delta = (int16_t)(value << FIXED_DECIMAL) - walfordMeanEx;
    walfordMeanEx += delta / walfordCount;
}

To get a value out, we just shift variable 5-bits back to remove the “decimal point”.

uint16_t walford_getMean(void) {
    int16_t value = (walfordMeanEx >> FIXED_DECIMAL);
    return (value > 0) ? (uint16_t)value : 0;
}

And that’s it. It’s a cheap (memory-wise) way to get an average on a 8-bit microcontrollers. As compromises go, I had worse.

In any case, you can download code here.

Disk Preclear

Some time ago I got into a habit of shucking. And each shucked drive I first fill with a random data. Purpose of this is twofold; the primary reason being security as disk filled with random data doesn’t “leak” information when you place encrypted file system on it (except when that file system is ZFS with native encryption but that’s a long rant for some other time).

The second reason is drive verification before I add it to my pool. While disks are hearty bunch these days, they do fail. And they usually fail either really early or really late. Giving it a bit of exercise early in their lifetime can sometime provoke failure. And if that failure happens you can easily do RMA. Doing the warranty replacement is also possible once they are shucked but might involve messing with manufacturer and their idea of “intended purpose” that’s part of their terms of warranty. It’s easier this way.

Previously I used a good old dd on my server for the same purpose but that actually became slightly annoying as I filled all the bays. Just to initialize a new drive, I had to remove the old drive and hope other disks are ok while I randomize the new one (can take a few days). Furthermore, that also meant I was increasing the CPU load (remember random numbers are not cheap) and temperature without writing any useful data. A task that’s more suitable for my Windows desktop.

With .NET 6 out, I figured I could give it a ride and write a Windows Forms application.

In a default state application will randomly traverse disk writing random data and verifying what’s written. While access might be random, every sector is accessed once and only once even in this mode. This is my favorite mode by far and I find it strangely calming due to how it looks graphically. But then again, I was always a fan of just looking into the defragmentation screen. :)

If you want the same effect you would get using dd with urandom you can switch to the sequential access. The same random data but written quite a bit faster since drive doesn’t need to seek all over the place. Of course, if security is not much of a concern, once could also fill disk with zeros or a “standard” 0x55AA pattern. While slightly faster, those modes are quite a bit less exciting.

I created this application to fill the need I had and no other application seemed to do exactly what I wanted. If someone else finds it useful, it’s awesome. If not, I still had fun creating it and that’s all that matters in the end.

Application download and link to its source code is available here.

IDE0180: Use Tuple to Swap Values

As I upgraded my Visual Studio 2022 Preview, I noticed a new code suggestion: IDE0180. It popped next to a reasonably standard variable swap:

var tmp = a;
a = b;
b = tmp;

The new syntax it recommended was much nicer, in my opinion:

(a, b) = (b, a);

I love the new syntax!

Except it’s not new - it has been available since C# 7 (we’re at 10 now, just for reference). I just somehow missed it. The only reason I noticed it now was due to a suggestion. I guess better late than never. :)

Native PNG in C# .NET

It all started with a simple barcode. I was refactoring my old Code128 barcode class to work in .NET 5 and faced an interesting issue - there was no option to output barcode as an image. Yes, on Windows you could rely on System.Drawing but .NET is supposedly multiplatform environment these days. And no, System.Drawing is not supported in Linux - you only get error CS0234: The type or namespace name 'Drawing' does not exist in the namespace 'System' (are you missing an assembly reference?.

If you’re thinking to yourself “Wait, I remember Microsoft’s System.Drawing.Common working on Linux”, you don’t have a bad memory. However, Microsoft since made a small update and, with a stroke of a pen, decided to abandon it. Yes, you can use runtime options as a workaround but official support is no longer there.

Alternatives do exist. Most notably both ImageSharp and SkiaSharp would satisfy any graphic need I might have. But both also present quite a big dependency to pull for what is essentially a trivial task. I mean, how difficult can it be to implement PNG writer?

It turns out, not difficult at all. If you go over specification you’ll notice there are only three chunks you need to support: IHDR, IDAT, and IEND. If you know how to write those three, you can output PNG image that’s readable by all. Literally the only two things that are not straightforward were deflate compression that required an extra header and dealing with CRC.

Most of my debugging time was actually spend dealing with output not showing properly in IrfanView. I could read my output image in Paint.NET, Paint, Gimp, and multitude of other programs I’ve tried. It took me a while before I figured out that IrfanView 4.54 is actually one with the issue. Update to the latest version sorted that one out.

In any case, I successfully added PNG support to by barcode class.

And then I started thinking… Why not make a simple PNG image reader/writer? Well, answers are numerous. First of all, alternatives exist and a lone developer can never hope to have such level of completeness. Secondly, I don’t deal with graphics on a daily basis and thus features would be few and far between. And lastly, while PNG is a relatively simple specification to start with, it has a lot of optional complexity. And some of that complexity would require me to learn much more than I ever want to know (yes, ICC profiles - I’m looking at you).

However, logic be damned. What’s the better use of an afternoon than writing a PNG parser?

In any case, the final product is here and while as simple as it gets it’s relatively feature complete when it comes to loading plain old PNG images. The only mandatory feature I failed to implement is support for interlaced images.

Expectedly, the end result is usable for changing a pixel or two but not for much more as all infrastructure is missing. I will add some of the missing functionality in the future, resize and interlacing support coming first, but I consider this class reasonably feature complete for what I both need and can do without making a full blown library. And I haven’t had as much fun programming in a while.