Welford's Algorithm

Most of the time, general statistics calculations are easy. Take all data points you have, find the average, the standard deviation, and you have 90% of stuff you need to present a result in a reasonable and familiar manner. However, what if you have streaming data?

Well, then you have a Welford’s method. This magical algorithm enables you to calculate both average and standard deviation as data arrives without wasting a lot of memory accumulating it all.

So, of course, I wrote a C# code for it. To use it, just add something like this:

var stats = new WelfordVariance();
while(true) {
    stats.Add(something.getValue());
    output.SetMean(stats.Mean);
    output.SetStandardDeviation(stats.StandardDeviation);
}

Algorithm is not perfect and will sometime slightly differ from classically calculated standard deviation but it’s generally within a spitting distance and uses minimum of memory. It’s hard to find better bang-for-buck when it comes to large datasets.

One Taken Every Second

In order to keep an eye on my garage I placed a Wyze camera in it. So when I noticed one day that somebody has swiped my tool bag, I thought I’ll find the culprit quickly. Well, it was not meant to be.

I had recording turned on but only a 32 GB card in it. And I noticed tool bag missing only after two weeks or so. So any recording was already gone by the time I took a look. Since only other people that had access to the garage were Amazon delivery guys, I could narrow it down but not sufficiently to do anything about it.

So I went to look into a cheap solution to record images long term and RTSP immediately came to mind. Even better, Wyze cameras already support it (albeit via a special firmware).

My idea was to simply record an image every second and save it on my NAS using ffmpeg. While this was a simple task in theory, it proved to be a bit annoying to find parameters that would be stable enough. Most notably, sometime it would just stop ingesting new frames and thus require restart.

After testing a lot of different parameters, I came with the following code:

while (true); do
    ffmpeg \
        -probesize 1000000 \
        -analyzeduration 1000000 \
        -flags low_delay \
        -fflags discardcorrupt \
        -re \
        -rtsp_transport tcp \
        -stimeout 10000000 \
        -allowed_media_types video \
        -i rtsp://${CAMERA_USER}:${CAMERA_PASS}@${CAMERA_IP}/live \
        -f image2 \
        -strftime 1 \
        -vf fps=fps=1 \
        -async 1 \
        -vsync 1 \
        -threads 1 \
        -use_wallclock_as_timestamps 1 \
        "${BASE_DIRECTORY}/${CAMERA_IP}~%Y-%m-%d-%H-%M-%S.jpg"
    sleep 1
done

Using this setup ffmpeg will keep taking image every second. If it gets stuck, it will exit and then it’s on while to restart the capture again. One can then use a separate process to convert these files into a mjpeg file but that’s story for another day.

Inappropriate Ioctl for Device

After disconnecting a serial USB cable from my Ubuntu Server 20.04, I would often receive “Inappropriate ioctl for device” error when trying to redirect output to serial port.

stty -F /dev/ttyACM0 -echo -onlcr
 stty: /dev/ttyACM0: Inappropriate ioctl for device

Quick search yielded multiple results but nothing that actually worked for me. Most promising were restarting udev and manual driver unbind but they didn’t really solve anything end-to-end. The only solution was to reboot.

However, after a bit of playing with unloading drivers, I did find solution that worked. Unload driver, manually delete device, and finally load driver again.

modprobe -r cdc_acm
rm -f /dev/ttyACM0
modprobe cdc_acm

I am not sure why unloading driver didn’t remove device link itself, but regardless, I could finally get it to work without waiting for reboot.

Watching Sector Count - Take 2

As I shucked my 12 TB drive and got it working, I noticed my reporting script was reporting it as 11997 GB. What the heck. Even with power-of-two shenanigans, I would expect capacity to follow LBA Count for Disk Drives Standard (LBA1-03) I already wrote about.

When I checked disk, I saw the following:

Based on my calculator, I expected to see size of 12,002,339,414,016 bytes. Was WDC placing non-standard capacity drives in their enclosures? Or did I miss something? Well, I missed something. :)

There is a later version of sector count standard coming from SFF Committee as SFF-8447. And this standard makes a difference between low capacity (80 - 8,000 GB) and high capacity (>8,000 GB) disk drives.

For lower capacity drives, formulas are ones we already know (first one is for 512-byte, second one for 4K sector):

97,696,368 + (1,953,504 * (CapacityInGB - 50)) -or- 12,212,046 + (244,188 * (CapacityInGB - 50))

Drives larger than 8 TB have the following formulas (512-byte, 4K sector sizes):

ceiling(CapacityInBytes / 512, 221) -or- ceiling(CapacityInBytes / 4096, 218)

Armed with both formulas, we can update the sector count calculator - find it below.

Killing a Connection on Ubuntu Server 20.04

If you really want to kill a connection on a newer kernel Ubuntu, there is a ss command. For example, to kill connection toward 192.168.1.1 with dynamic remote port 40000 you can use the following:

ss -K dst 192.168.1.1 dport = 40000

Nice, quick, and it definitelly beats messing with routes and waiting for a timeout. This is assuming your kernel was compiled with CONFIG_INET_DIAG_DESTROY (true on Ubuntu).

To get a quick list of established connections for given port, one can use netstat with a quick’n’dirty grep:

$ netstat -nap | grep ESTABLISHED | grep ^^<port>^^