HexDump's Illegal Seek

After I upgraded to Ubuntu 21.04, my TmpUsb script suddenly started reporting the following hexdump: stdin: Illegal seek.

Line causing issue was the one determining partition serial number:

dd if=/dev/sda bs=512 skip=1 count=1 | hexdump -s39 -n4 -e '4/1 "%02X"'

It seems that hexdump got a bit too stricter with its input parameters and now disallows skipping bytes in fifo stream. I haven’t investigated much but my guess is that skipping 39 bytes probably messes with its internal buffer. In any case, dd has no such issues so the same code can be done without skipping in hexdump.

dd if=/dev/sda bs=1 skip=551 count=4 | hexdump -n4 -e '4/1 "%02X"'`

The best part is that this is compatible with older versions too.

Dazed and Confused, but Trying to Continue

From time to time, I would see the following slightly poetic statement on my console.

Uhhuh. NMI received for unknown reason 31 on CPU 3.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue

The first Internet search brought me sadness and dismay - my hardware was failing. It took going a bit deeper to find that AMD servers have that issue quite often even in the absence of real hardware failure.

Solution? Disable darn C-States. It’s a server after all.

PS: And no, even if you want to keep C-States, reason 31 is nothing to worry about - it’s been happening on my system for 2 years before this and I had no issues with it. It’s just annoyance and nothing more.

Recording Both Microphone and Speaker Under Ubuntu 21.04

When one records audio under Linux, issue that quite a few applications have is recording both microphone input and headphone output. And that’s true for SimpleScreenRecorder, otherwise really good recording application.

However, Linux always has a way around those restrictions and those can be actually found on SimpleScreenRecorder pages if you look deep enough.

pactl load-module module-null-sink \
  sink_name=duplex_out sink_properties=device.description="\"Duplex\ Output\""
pactl load-module module-null-sink \
  sink_name=app_out sink_properties=device.description="\"Application\ Output\""
pactl load-module module-loopback source=app_out.monitor
pactl load-module module-loopback source=app_out.monitor sink=duplex_out
pactl load-module module-loopback sink=duplex_out
pactl set-default-sink app_out

Trick is to essentially create two new output devices (i.e. sinks). One of them (app_out) will just be a target toward which applications should direct their output. Magic happens with the second output (duplex_out) which combines application output and what comes from microphone.

Now when you record audio, you can just point application to Duplex Output and record both sides.

PS: To make these changes permanent, they can be entered into /etc/pulse/default.pa. Of course, quoting rules are bit different so adjust accordingly if you have a space in your description.

…
# Recording setup
load-module module-null-sink sink_name=duplex_out sink_properties=device.description="Duplex\ Output"
load-module module-null-sink sink_name=app_out sink_properties=device.description="Application\ Output"
load-module module-loopback source=app_out.monitor
load-module module-loopback source=app_out.monitor sink=duplex_out
load-module module-loopback sink=duplex_out
set-default-sink app_out

DevStack on VirtualBox Ubuntu 20.04

The first step for DevStack inside of VirtualBox is creating the virtual machine. There are two obvious changes that you need to make and those are increasing processor count and assigned memory as high as you can afford it. The other two are a bit more “sneaky”.

We really have to enable Nested VT-x/AMD-V under System, Processor and if we want to access our system we should really set network forwarding rules for port 80 (HTTP for Dashboard) and port 22 (SSH, optional but really helpful). I usually set them to be 56080 and 56022 respectively under my localhost but the actual numbers can be of your choosing. And yes, there are other ways to setup networking but NAT with forwarding is mine.

With the virtual machine set, the next step toward DevStack is installing OS. While official guidelines prefer Ubuntu 18.04, I like to go with a slightly newer Ubuntu 20.04 Server. Whole installation is essentially one big click-next event with the only non-default part being installation of OpenSSH.

Once OS is installed, I also like to add my user to password-less sudoers and do any needed updates:

echo "$USER ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/$USER

sudo apt update
sudo apt dist-upgrade --yes

And now finally we can follow the DevStack instructions with customized host IP (otherwise you’ll get “Could not determine host ip address” error) and admin password.

sudo useradd -s /bin/bash -d /opt/stack -m stack
echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
sudo su - stack

git clone https://github.com/openstack-dev/devstack.git -b $STACK_BRANCH devstack/
STACK_BRANCH=stable/^^wallaby^^

cd devstack
cp samples/local.conf .
sed -i 's/#HOST_IP=w.x.y.z/HOST_IP=^^10.0.2.15^^/' local.conf
sed -i 's/ADMIN_PASSWORD=nomoresecret/ADMIN_PASSWORD=^^devstack^^/' local.conf
echo "#Enable heat services" >> local.conf
echo "enable_service h-eng h-api h-api-cfn h-api-cw" >> local.conf
echo "#Enable heat plugin" >> local.conf
echo "enable_plugin heat https://git.openstack.org/openstack/heat $STACK_BRANCH" >> local.conf

./stack.sh

Once done, GUI is available at http://localhost:56080/dashboard/.

Wait For Mountpoint

I have quite a few scripts running on my home server and they love writing on disk. They love it so much that, after reboot, they don’t necessarily wait for mount points to appear - they just start writing. Unfortunately, such eagerness also means that my other scripts mounting ZFS might find directory already in use and give up.

What I needed was a way to check if mount point is already there before starting with write. The easiest approach for me was using mountpoint command.

TEST_PATH=/test/directory
while(true); do  # wait for mount point
    mountpoint "$TEST_PATH" >/dev/null
    if [[ $? != 0 ]]; then
        sleep 1
        continue
    fi
    break
done

Script fragment above will check if given directory has something mounted and, if not, wait for 1 more second. Once test succeeds, it will break out of the infinite loop and proceed with whatever follows.

Easy enough.

One Taken Every Second

In order to keep an eye on my garage I placed a Wyze camera in it. So when I noticed one day that somebody has swiped my tool bag, I thought I’ll find the culprit quickly. Well, it was not meant to be.

I had recording turned on but only a 32 GB card in it. And I noticed tool bag missing only after two weeks or so. So any recording was already gone by the time I took a look. Since only other people that had access to the garage were Amazon delivery guys, I could narrow it down but not sufficiently to do anything about it.

So I went to look into a cheap solution to record images long term and RTSP immediately came to mind. Even better, Wyze cameras already support it (albeit via a special firmware).

My idea was to simply record an image every second and save it on my NAS using ffmpeg. While this was a simple task in theory, it proved to be a bit annoying to find parameters that would be stable enough. Most notably, sometime it would just stop ingesting new frames and thus require restart.

After testing a lot of different parameters, I came with the following code:

while (true); do
    ffmpeg \
        -probesize 1000000 \
        -analyzeduration 1000000 \
        -flags low_delay \
        -fflags discardcorrupt \
        -re \
        -rtsp_transport tcp \
        -stimeout 10000000 \
        -allowed_media_types video \
        -i rtsp://${CAMERA_USER}:${CAMERA_PASS}@${CAMERA_IP}/live \
        -f image2 \
        -strftime 1 \
        -vf fps=fps=1 \
        -async 1 \
        -vsync 1 \
        -threads 1 \
        -use_wallclock_as_timestamps 1 \
        "${BASE_DIRECTORY}/${CAMERA_IP}~%Y-%m-%d-%H-%M-%S.jpg"
    sleep 1
done

Using this setup ffmpeg will keep taking image every second. If it gets stuck, it will exit and then it’s on while to restart the capture again. One can then use a separate process to convert these files into a mjpeg file but that’s story for another day.

Inappropriate Ioctl for Device

After disconnecting a serial USB cable from my Ubuntu Server 20.04, I would often receive “Inappropriate ioctl for device” error when trying to redirect output to serial port.

stty -F /dev/ttyACM0 -echo -onlcr
 stty: /dev/ttyACM0: Inappropriate ioctl for device

Quick search yielded multiple results but nothing that actually worked for me. Most promising were restarting udev and manual driver unbind but they didn’t really solve anything end-to-end. The only solution was to reboot.

However, after a bit of playing with unloading drivers, I did find solution that worked. Unload driver, manually delete device, and finally load driver again.

modprobe -r cdc_acm
rm -f /dev/ttyACM0
modprobe cdc_acm

I am not sure why unloading driver didn’t remove device link itself, but regardless, I could finally get it to work without waiting for reboot.

Killing a Connection on Ubuntu Server 20.04

If you really want to kill a connection on a newer kernel Ubuntu, there is a ss command. For example, to kill connection toward 192.168.1.1 with dynamic remote port 40000 you can use the following:

ss -K dst 192.168.1.1 dport = 40000

Nice, quick, and it definitelly beats messing with routes and waiting for a timeout. This is assuming your kernel was compiled with CONFIG_INET_DIAG_DESTROY (true on Ubuntu).

To get a quick list of established connections for given port, one can use netstat with a quick’n’dirty grep:

$ netstat -nap | grep ESTABLISHED | grep ^^<port>^^

Cleaning Disk

Some time ago I explained my procedure for initializing disks I plan to use in ZFS pool. And the first step was to fill them with random data from /dev/urandom.

However, FreeBSD /dev/urandom is not really the speed monster. If you need something faster but still really secure, you can go with a random AES stream.

openssl enc -aes-128-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | hexdump)" \
    -pbkdf2 -nosalt </dev/zero | dd of=/dev/diskid/^^DISK-ID-123^^ bs=1M

Since the key is derived from random data, in theory it should be equally secure but (depending on CPU), multiple times faster than urandom.

Basic XigmaNAS Stats for InfluxDB

My home monitoring included pretty much anything I wanted to see with one exception - my backup NAS. You see, I use embedded XigmaNAS for my backup server and getting telegraf client onto it is problematic at best. However, who needs Telegraf client anyhow?

Collecting stats themselves is easy enough. Basic CPU stats you get from Telegraf client usually can be easily read via command line tools. As long as you keep the same tags and fields as what Telegraf usually sends you can nicely mingle our manually collected stats with what proper client sends.

And how do we send it? Telegram protocol is essentially just a set of lines pushed using HTTP POST. Yes, if you have a bit more secure system, it’s probably HTTPS and it might even be authenticated. But it’s still POST in essence.

And therein lies XigmaNAS’ problem. There is no curl or wget tooling available. And thus sending HTTP POST on embedded XigmaNAS is not possible. Or is it?

Well, here is the beauty of HTTP - it’s just a freaking text over TCP connection. And ancient (but still beloved) nc tool is good at exactly that - sending stuff over network. As long as you can “echo” stuff, you can redirect it to nc and pretend you have a proper HTTP client. Just don’t forget to set headers.

To cut the story short - here is my script using nc to push statistics from XigmaNAS to my Grafana setup. It’ll send basic CPU, memory, temperature, disk, and ZFS stats. Enjoy.