Linux, Unix, and whatever they call that world these days

DaVinci Resolve 18 on Ubuntu 23.10 with Intel GPU

While I successfully installed DaVinci Resolve 18 on my Ubuntu 23.10 machine, I simply couldn’t make it start. Quick excursion to the command line, showed the following error:

$ /opt/resolve/bin/resolve
/opt/resolve/bin/resolve: symbol lookup error: /lib/x86_64-linux-gnu/libpango-1.0.so.0: undefined symbol: g_string_free_and_steal

Yep, something was wrong with libraries. Fortunately, after checking quite a few wrong advices on Internet (I know, who would have thought), I solved it by a simple file copy:

sudo cp /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 /opt/resolve/libs/

Illustration

While this did solve the non-starting issue, I now got Unsupported GPU processing mode error. After searching a bit around the net, consensus seemed to be that you have to have external GPU for Resolve to work under Linux. But that didn’t seem right. Why would DaVinci Resolve require external GPU under Linux while being happy with internal GPU under Windows?

This seemed unreasonable so I tried installing OpenCL for my Intel GPU.

sudo apt install intel-opencl-icd

And with this simple change, I got to run Resolve to start. But I still couldn’t edit.

So I tried installing Intel’s own Open CL drivers:

mkdir neo
cd neo
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.14828.8/intel-igc-core_1.0.14828.8_amd64.deb
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.14828.8/intel-igc-opencl_1.0.14828.8_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/23.30.26918.9/intel-level-zero-gpu-dbgsym_1.3.26918.9_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/23.30.26918.9/intel-level-zero-gpu_1.3.26918.9_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/23.30.26918.9/intel-opencl-icd-dbgsym_23.30.26918.9_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/23.30.26918.9/intel-opencl-icd_23.30.26918.9_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/23.30.26918.9/libigdgmm12_22.3.0_amd64.deb
sudo apt install ./*.deb

Nope - start but doesn’t allow any editing.

After tryign a few more things, I decided to give up. :( I guess you really cannot run DaVinci Resolve on Intel Xe.

Watchdog setup for Supermicro server

EPYC server processors are really nice, when they work. However, lately with kernel 6.2 I started getting dreadful “CPU stuck” errors that lead to hanging system. Normal person might revert to an older kernel. Me? I decided to turn on the watchdog.

In case you don’t know, watchdog is a functionality that, once turned on, will require your system to notify it every once in a while that it’s still active. If notification is not received within given time interval, system is assumed stuck and thus it gets rebooted. Best of all, this is done on a hardware level and thus no hanging application or CPU will prevent it.

To confuse things a bit, my Supermicro M11SDV-4CT-LN4F server, seems to have two watchdog systems. One is part of Epyc platform itself and controlled via BIOS setting. That one has 5 minute interval and no matter what I couldn’t get it working properly. I mean, I could get it running but, since there was no easy way to reset it, system would reboot every 5 minutes, no matter what.

The second watchdog is the part of AST2500 chipset that handles other IPMI functions. And this one was well supported from Linux command line using ipmitool utility. To see its status, just ask ipmitool for that information:

ipmitool mc watchdog get

But there is no option to turn it on. However, one can always send raw commands and I was fortunate to see that somebody already did. Not to get into too much details, the last two numbers in the string of hexadecimal values are the only thing you generally want to change - time interval. In example below, I decided to go for 610 seconds (0x17D4 in 0.1 s units).

ipmitool raw 0x06 0x24 0x04 0x01 0x00 0x00 0xD4 0x17

This will start a ticking bomb that will, if not defused within the given interval, reboot your computer. So, why did I select 10 minutes and 10 seconds? As many things, this was completely subjective.

Well, no matter what, I wanted this watchdog not to interfere with my normal server operation. Since a normal reboot takes about 5 minutes, I wanted to have 5 minutes on counter even if I reboot system myself just before watchdog would reset. So, if I select 10 minute interval and reset it every 5 minutes, this gives me 5 minutes of extra time I might need for reboot. But why extra 10 seconds? Well, in case I mess with my watchdog settings and I miss reset at 5 minute mark, I wanted to give an extra chance of reset at 10 minute mark without having to deal with a reboot race condition.

And how might one actually setup watchdog and its reset within Linux? Well, crontab, of course. These two entries were all it took:

@reboot
/usr/bin/ipmitool raw 0x06 0x24 0x04 0x01 0x00 0x00 0xD4 0x17

0,5,10,15,20,25,30,35,40,45,50,55 * * * *
/usr/bin/ipmitool mc watchdog reset

This will turn on watchdog upon every system reset (and yes, once watchdog goes off, you do need to manually turn it back on) and every 5 minutes system will reset its counter if nothing goes awry.

Simple and effective.

ZFS Encryption Speed (Ubuntu 23.10)

There is a newer version of this post

As it became a custom, I retest ZFS native encryption performance with each new Ubuntu release. Here we have results for Ubuntu 23.10 on kernel 6.5 using ZFS 2.2.

Testing was done on a Framework laptop with an i5-1135G7 processor and 64GB of RAM. Once booted into installation media, I execute the script that creates 42 GiB RAM disk that hosts all data for six 6 GiB files. Those files are then used in RAIDZ2 configuration to create a ZFS pool. The process is repeated multiple times to test all different native ZFS encryption modes in addition to a LUKS-based test. This whole process is repeated again with AES disabled and later with a reduced core count.

Illustration

Since I am testing on the same hardware as for 23.04 and using essentially the same script, I expected similar results but I was slightly surprised to see both raw read and write speed has been reduced by more than 10%. I am not sure if this is due to the new kernel, new BIOS, or some other combination of changes but the performance hit seems quite significant.

However, what I’m most interested in is not necessarily the actual speed but how it’s impacted by encryption. As compared to last year, it seems each GCM encryption mode has taken a few percent hit. We’re still talking about 2 GiB/s for both read and write so I’m not too worried.

Interestingly, while key size had more impact before, it seems that with 23.10 you can count on the same speed regardless if you select 128, 192, or 256 bit key.

If you don’t have AES support and you need CCM, the news is not that good as that code path has gotten significantly worse. Unless you’re stuck on an ancient CPU this is irrelevant I guess as you should never opt for CCM in the first place.

Using ZFS on top of LUKS has gotten slightly better when it comes to writes where it actually lagged the most behind the native ZFS. The improvement is significant but we’re still talking about 30% lower speeds. On read size, there are no changes and it’s the only area where LUKS wins over the native ZFS encryption.

For this release, I also experimentally tried to get power usage for each test run. I did the same by disconnecting the battery and measuring the power the laptop was drawing. This is not the most precise way of measuring it so I might be off but it looked as ZFS encryption was as efficient as it gets when it comes to the power usage.

To summarize, the native ZFS encryption is still live and kicking in Ubuntu 23.10 and might even provide some power usage advantages as compared to LUKS.


PS: You can find older tests for Ubuntu 23.04, 22.10, 22.04, 20.10, and 20.04.

O Brother, Where Art Thou? (Printer Edition)

While most of my server machines are Linux and I am daily driving Ubuntu on my laptop, I still use Windows for some things. For example, scanning and printing has been thus far in the exclusive Windows domain.

Well, not anymore. Time has come to connect Ubuntu 23.04 desktop to my trusty Brother MFC-J475DW printer/scanner.

I first went to the Brother website and surprisingly they provide Linux drivers for something that’s now ancient device. And that’s where pleasant surprise stopped as instructions didn’t really work. But, with a bit of adjustments, it did eventually work out.

First of all, I connect to this printer via network and thus you’ll see IP in instructions. I’ll use 192.168.0.10 as an example - if your printer uses different address, adjust accordingly.

Secondly, you need the following packages (and yes, I’m not downloading fax drivers):

Thirdly, we need a few extra packets as a prerequisite.

sudo apt-get install lib32z1 lprng cups

For printer, we need both LPR and CUPS drivers:

sudo dpkg -i --force-all mfcj475dwlpr-3.0.0-1.i386.deb
sudo sed -i 's|\:lp.*|\:rm=192.168.0.10\\\n\t:rp=lp\\|g' /etc/printcap
sudo systemctl restart lprng

sudo dpkg -i --force-all mfcj475dwcupswrapper-3.0.0-1.i386.deb
lpadmin -p MFC-J475DW -E -v lpd://192.168.0.10/MFC-J475DW -P /usr/share/cups/model/Brother/brother_mfcj475dw_printer_en.ppd

For scanner, steps are even simpler:

sudo dpkg -i --force-all brscan4-0.4.11-1.amd64.deb
sudo brsaneconfig4 -a name=MFC-J475DW model=MFC-J475DW ip=192.168.0.10

If all went ok, you should be able to print and scan now.


PS: Huge props to Brother for providing Linux drivers for pretty much all their devices.

Xeoma in Docker (but Not in Kubernetes)

Illustration

After running my own homebrew solution involving ffmpeg and curl scriptery for a while, I decided it’s time to give up on that and go with a proper multi-camera viewing solution.

I had narrowed my desired setup to a few usual suspects: ZoneMinder, Shinobi, BlueIris, and one that I hadn’t heard about before: Xeoma.

I wanted something that would record (and archive) 2 cameras on my local network, could run in Kubernetes, and would allow for a mobile application some time down the road.

BlueIris was an immediate no-go for me as it’s Windows-only software. There’s a docker version of it but it messes with Wine. And one does not simply mess with Wine.

I did consider ZoneMinder and Shinobi, but both had setups that were way too complex for my mini Kubernetes (Alpine K3s). Mind you, there were guides out there but none of them started without a lot of troubleshooting. And even when I got them running, I had a bunch of issues still lingering. I will probably revisit ZoneMinder at one point in the future, but I didn’t have enough time to properly mess with it.

That left Xeoma. While not a free application, I found it cheap enough for my use case. Most importantly, while updates were not necessarily free, all licenses were perpetual. There’s no monthly fee unless you want to use their cloud.

Xeoma’s instructions were okay, but not specific for Kubernetes. However, if one figures out how to install stuff in docker, it’s trivial to get it running in Kubernetes. Here is my .yaml file:

---
apiVersion: apps/v1
kind: Deployment

metadata:
  namespace: default
  name: xeoma
  labels:
    app: xeoma

spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: xeoma
  template:
    metadata:
      labels:
        app: xeoma
    spec:
      containers:
      - name: xeoma
        image: coppit/xeoma:latest
        env:
        - name: VERSION
          value: "latest"
        - name: PASSWORD
          value: "changeme"
        volumeMounts:
          - name: config-volume
            mountPath: /config
          - name: archive-volume
            mountPath: /archive
      volumes:
        - name: config-volume
          hostPath:
            path: /srv/xeoma/config/
            type: Directory
        - name: archive-volume
          hostPath:
            path: /srv/xeoma/archive
            type: Directory

---
apiVersion: v1
kind: Service

metadata:
  namespace: default
  name: xeoma
  labels:
    app: xeoma

spec:
  type: LoadBalancer
  selector:
    app: xeoma
  ports:
  - name: server
    protocol: TCP
    port: 8090
    targetPort: 8090
  - name: web
    protocol: TCP
    port: 10090
    targetPort: 10090

And yes, this might not be the best setup - especially using directory volume mounts - but I find it very useful in my home lab. For example, the backup becomes just a trivial rsync affair.

With Kubernetes running, my next task was to select a license level. While there is a great licensing overview page, it still took me probably more than half an hour to finally select a version.

Free and Starter were immediately discounted due to their archive retention only going up to 5 days, and I wanted much more. The limit of 3 modules is not hugely problematic for my case, but later I found that might be a bit too low (due to how chaining works) even for me. I likewise removed Pro from consideration as it was way more expensive and it actually didn’t offer anything that I needed for my monitoring setup.

So my decision was between Lite and Standard. As I only needed 2 cameras at this time, Lite made a bit more sense. Out of things I was interested in, you do lose user profiles (i.e., everybody logs in as the same user) and the module for issue detection (e.g., camera offline) is strangely missing. But those weren’t deal breakers.

Do note that Lite also doesn’t offer upgrades to the software. The version you have is the version you’re stuck with. For a professional setup, I would definitely go with Standard, but again, for my home use case, I don’t need updates all the time.

So, I got the key, plugged it into the software, played a bit, decided to restart the server, and… my license was gone. One thing I couldn’t notice in trial mode was that the license was connected to the MAC address of the pod the software was running on. And the pod will get a new MAC address each time you restart it.

I tried quite a few tricks to make the MAC address stick: manually setting the address in /etc/network/interfaces, messing with ifconfig, docker-in-docker sidecar… No matter what I did, I couldn’t get the licensing to work in combination with Kubernetes.

And therein lies the danger of any licensing. If you are too strict, especially in a virtual world, real users get impacted while crackers are probably just fine…

In their defense, you can also get demo licenses. After I figured out what the issue was, having 10 demo licenses to mess with allowed me to play with the system a bit more than what I would be able to do with my perpetual license. Regardless, I was defeated - Kubernetes was not to be. I strongly recommend you to obtain Demo licenses if you have any unusual setup.

Regardless of the failed Kubernetes setup, I also had a good old docker on the same machine. With a few extra line items, that one worked wonderfully. My final setup for docker was the following command:

docker run --name=xeoma \
  -d \
  -p 8090:8090 \
  -p 10090:10090 \
  -v /srv/xeoma/config:/config \
  -v /srv/xeoma/archive:/archive \
  -e VERSION=https://felenasoft.com/xeoma/downloads/2023-08-10/linux/xeoma_linux64.tgz \
  -e PASSWORD=admin \
  --hostname=xeoma \
  --mac-address 08:ae:ef:44:26:57 \
  --restart on-failure \
  coppit/xeoma

Here, -d ensures that the container is “daemonized” and, as such, goes into the background.

Port 8090 as output is mandatory for setup, while the web interface running on 10090 can be removed if one doesn’t plan to use it. I decided to allow both.

The directory setup is equivalent to what I planned to use with Kubernetes. I simply expose both the /config and /archive directories.

Passing the URL as the VERSION environment variable is due to Lite not supporting upgrades. Doing it this way ensures we always get the current version. Of course, readers from the future will need to find their URL on the History of changes webpage. Changing the PASSWORD environment variable is encouraged.

In order for licensing to play nicely, we need the --mac-address parameter to be set to a fixed value. The easiest way forward is just generating one randomly. And no, this is not MAC address connected to my license. :)

For restarts, I found on-failure setting works best for me as it will start the container when the system goes up, in addition to restarting it in the case of errors.

And lastly, the Docker image coppit/xeoma is given as a blueprint for our setup.

And this Docker solution works nicely for me. Your mileage (and needs) may vary.

Disabling Caps Lock Under Linux

Different keyboard layouts on different laptops bring different annoyances. But there is one key that annoys me on any keyboard: CAPS LOCK. There is literally no reason for that key to exist. And yes, I am using literally appropriately here. The only appropriate action is to get rid of it.

If you’re running any systemd-enabled Linux distribution that is easy enough. My approach is as follows:

echo -e "evdev:atkbd:*\n KEYBOARD_KEY_3a=f15" \
  | sudo tee /etc/udev/hwdb.d/42-nocapslock.hwdb

To apply, either reboot the system or reload with udevadm:

sudo udevadm -d hwdb --update
sudo udevadm -d control --reload
sudo udevadm trigger

Congrats, your keyboard is now treating CapsLock as F15 (aka the highest F key you can assign keyboard shortcuts too in Gnome settings). Of course, you can select and other key of your liking. For that, you can take a look at SystemD GitHub for ideas. Of course, setting it to nothing (i.e., reserved) is a valid choice as well.


PS: If you want to limit change to just your laptop (e.g., if you’re propagating changes via Ansible and you don’t want to touch your desktop), you can check content of /sys/class/dmi/id/modalias for your computer IDs. Then you can limit your input appropriately. For example, limiting change to my Framework 13 laptop would look something like this:

evdev:atkbd:dmi:bvn*:bvr*:bd*:br*:svnFramework:*
 KEYBOARD_KEY_3a=f20

PPS: In case Caps Lock is not 3a key on your computer, you might need to adjust files appropriately. To figure out which key it is, run evtest. When you press Caps Lock, you’ll get something like this:

-------------- SYN_REPORT ------------
type 4 (EV_MSC), code 4 (MSC_SCAN), value 3a
type 1 (EV_KEY), code 58 (KEY_CAPSLOCK), value 0

Value you want is after MSC_SCAN.


PPPS: Another way to debug keyboard is by using libinput (part of libinput-tools package):

sudo libinput debug-events --show-keycodes

PPPPS: And yes, you can remap other keys too. F1 is my second “favorite”, close after Caps Lock.

Unreal Tournament 2004 Server on Kubernetes

From time to time I play Unreal Tournament 2004 with my kids. Aging game, being older than either of them, is not an obstacle to having fun. However, the inability to host a networked game is. As Windows updates and firewall rules change, we end up figuring out who gets to host the game every few months. Well, no more!

To solve my issues, I decided to go with Laclede’s LAN Unreal Tournament 2004 Dedicated Freeplay Server. This neat docker package has everything you need to host games on Linux infrastructure and, as often happens with docker, is trivial to run. If this is what you need, stop reading and start gaming.

But I wanted to go a small step further and set up this server to run on Kubernetes. Surprisingly, I didn’t find any YAML to do the same, so I decided to prepare one myself.

skopeo copy docker://lacledeslan/gamesvr-ut2004-freeplay:latest docker-archive:gamesvr-ut2004-freeplay.tar
---
apiVersion: apps/v1
kind: Deployment

metadata:
  namespace: default
  name: ut2004server
  labels:
    app: ut2004server

spec:
  replicas: 1
  selector:
    matchLabels:
      app: ut2004server
  template:
    metadata:
      labels:
        app: ut2004server
    spec:
      containers:
        - name: ut2004server
          image: lacledeslan/gamesvr-ut2004-freeplay:latest
          workingDir: "/app/System"
          command: ["/app/System/ucc-bin"]
          args:
            [
              "server",
              "DM-Antalus.ut2?AdminName=admin?AdminPassword=admin?AutoAdjust=true?bPlayerMustBeReady=true?Game=XGame.XDeathMatch?MinPlayers=2?WeaponStay=false",
              "nohomedir",
              "lanplay",
            ]
          envFrom:
          resources:
            requests:
              memory: "1Gi"
              cpu: "500m"

---
apiVersion: v1
kind: Service

metadata:
  namespace: default
  name: ut2004server
  labels:
    app: ut2004server

spec:
  type: LoadBalancer
  selector:
    app: ut2004server
  ports:
    - name: game
      protocol: UDP
      port: 7777
      targetPort: 7777
    - name: web
      protocol: TCP
      port: 8888
      targetPort: 8888

Happy gaming!


PS: While the server image has been available for a while now and it seems that nobody minds, I would consider downloading an image copy, just in case.

Running Comfast CF-953AX on Ubuntu 22.04

Based on list of wireless cards supported on Linux, I decided to buy Comfast CF-953AX as it should have been supported since Linux kernel 5.19. And HWE kernel on Ubuntu 22.04 LTS brings me right there. With only the source of that card being AliExpress, it took some time for it to arrive. All that wait for nothing to happen once I plugged it in.

In order to troubleshoot the issue, I first checked my kernel, and it was at the expected version.

Linux 5.19.0-38-generic #39~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC

Then I checked with lsusb my devices, and there it was.

Bus 002 Device 003: ID 3574:6211 MediaTek Inc. Wireless_Device

However, checking for network using lshw -C network showed nothing. As always, looking stuff up on the Internet brought a bit more clarity to the issue. Not only the driver wasn’t loaded but the USB VID:PID combination was unrecognized. The solution was simple enough: load the driver and teach it the new VID:POD combination.

sudo modprobe mt7921u
echo 3574 6211 | sudo tee /sys/bus/usb/drivers/mt7921u/new_id

Running lshw has found the card.

  *-network
       description: Wireless interface
       physical id: 5
       bus info: usb@2:1
       logical name: wlxe0e1a9389d77
       serial: e0:e1:a9:38:9d:77
       capabilities: ethernet physical wireless
       configuration: broadcast=yes driver=mt7921u driverversion=6.2.0-20-generic firmware=____010000-20230302150956
       multicast=yes wireless=IEEE 802.11

Well, now to make changes permanent, we need to teach Linux a new rule:

sudo tee /etc/udev/rules.d/90-usb-3574:6211-mt7921u.rules << EOF
ACTION=="add", \
    SUBSYSTEM=="usb", \
    ENV{ID_VENDOR_ID}=="3574", \
    ENV{ID_MODEL_ID}=="6211", \
    RUN+="/usr/sbin/modprobe mt7921u", \
    RUN+="/bin/sh -c 'echo 3574 6211 > /sys/bus/usb/drivers/mt7921u/new_id'"
EOF

After that, we should update our initramfs.

sudo update-initramfs -k all -u

And that’s it. Our old 22.04 just learned a new trick.

Visual Code Ctrl+. Not Working Under Ubuntu

Every time I reinstall Ubuntu, I am faced with the same issue. Whenever I press Ctrl+. in Visual Studio Code, I get an underscored letter “e” instead of the expected refactoring menu. And it takes me a while to remember the solution: running ibus-setup and removing emoji keyboard shortcuts.

However, there is also a scriptable solution. Just use gsettings and remove hotkeys from the command line.

gsettings set org.freedesktop.ibus.panel.emoji hotkey "[]"
gsettings set org.freedesktop.ibus.panel.emoji unicode-hotkey "[]"

Running Supermicro IPMIView 2.0 on Ubuntu

If you have a Supermicro motherboard, you have probably already downloaded the IPMIView utility. This utility allows access to motherboard sensors but most importantly, it includes KVM functionality, enabling you to access your computer as if it had a screen. Pure magic when it works. However, when using Ubuntu 23.04, I encountered difficulties running it due to the Malformed \uxxxx encoding error.

An internal LaunchAnywhere application error has occured and this application cannot proceed. (LAX)

Stack Trace:
java.lang.IllegalArgumentException: Malformed \uxxxx encoding.
    at java.base/java.util.Properties.loadConvert(Properties.java:678)
    at java.base/java.util.Properties.load0(Properties.java:455)
    at java.base/java.util.Properties.load(Properties.java:381)
    at com.zerog.common.java.util.PropertiesUtil.loadProperties(Unknown Source)
    at com.zerog.lax.LAX.<init>(Unknown Source)
    at com.zerog.lax.LAX.main(Unknown Source)

Nevertheless, since Supermicro provides all the necessary components, bypassing this error is actually quite simple. Just run the main .jar file directly.

cd <directory>
./jre/bin/java -jar IPMIView20.jar

And that’s it.