Recovering ZFS

Well, after using ZFS for years, it was only a matter of time before I encountered an issue. It all started with me becoming impatient with my Ubuntu 20.04 desktop. The CPU was at 100% and the system was actively writing to disk (courtesy of ffmpeg), but I didn’t want to wait. So, I decided to do a hard reset. What’s the worst that could happen?

Well, boot process took a while and I was presented with bunch of entries looking like this:

INFO: task z_wr_iss blocked for more than 122 seconds.
      Tainted: P      0E      6.8.0-35-generic #35-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
INFO: task z_metaslab blocked for more than 122 seconds.
      Tainted: P      0E      6.8.0-35-generic #35-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
INFO: task vdev_autotrim blocked for more than 122 seconds.
      Tainted: P      0E      6.8.0-35-generic #35-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
…

At first, I thought the system would recover on its own as this wasn’t the first time I had mistreated it. However, leaving it alone did nothing. So, it was time for recovery.

The first step was getting to the GRUB menu. Pressing <ESC> multiple times during boot simply dropped me to a prompt. And yes, you can load initramfs manually from there, but it’s a bit tedious. However, in this case, I just typed normal to get the menu, followed by <E> to edit the line starting with “linux”. There, I appended break, telling the system to drop me into initramfs prompt so that I could manually load ZFS.

From here, there was another hurdle to bypass. While this stopped before ZFS was loaded, it also stopped before my disks were decrypted. Had I used native ZFS encryption, this wouldn’t be a problem, but I wanted LUKS, so now I had to load them manually. As I had a mirror, I used the following to open both:

DISK1=/dev/disk/by-id/nvme-<disk1>
DISK2=/dev/disk/by-id/nvme-<disk2>

cryptsetup luksOpen $DISK1-part4 ${DISK1##*/}-part4
cryptsetup luksOpen $DISK2-part4 ${DISK2##*/}-part4

Finally, I was ready to import the pool and decided to do it in read-only mode:

zpool import -o readonly=on <pool>

And surprisingly, it worked. That also meant that my recovery efforts didn’t need to go too far. So, I decided to try importing it again but in read/write mode:

zpool export <pool>
zpool import <pool>

And then I was greeted with an ominous message:

PANIC: zfs: adding existent segment to range tree

However, the import didn’t get stuck as such, and my data was still there. So, I decided to give it a good scrub:

zpool scrub <pool>

While the scrub didn’t find any errors, going over all data seemed to have resulted in data structures “straightening out” and thus everything looked as good as before.

One reboot later, and I got into my desktop just fine.

PS: If that failed, I would have probably gone with zpool import -F <pool>.

PPS: If that also failed, disabling replays would be my next move.

echo 1 > /sys/module/zfs/parameters/zil_replay_disable
echo 1 > /sys/module/zfs/parameters/zfs_recover

PPPS: You can also add those parameters to “linux” grub line (zfs.zil_replay_disable=1 zfs.zfs_recovery=1).

Rendering KiCAD PCB to PNG

I have pretty much automated creation of KiCAD export files thanks to its command line interfacehttps://docs.kicad.org/8.0/en/cli/cli.html). However, there was always one thing missing - image for each PCB. Yes, not really necessary for manufacturing but it’s really handy when you want to just quickly check things without starting KiCAD and/or for any inventory system.

Before KiCAD 8, getting a PCB image required exporting to .step file and then converting stuff around. It got complicated enough with all the prerequisites that I have essentially given up. Fortunately, that’s not true anymore, and now we can get .wrl files that can be often used directly.

kicad-cli pcb export vrml --output "board.wrl" "board.kicad_pcb"

While exported files don’t include lights, this usually doesn’t matter to viewers who add their own. However, for our processing, we want to add light that is just slightly to the top-left (at 0.1 -0.1 -1) of the camera (at 0 0 -1).

head -1 "board.wrl" > "board.front.wrl"
cat <<EOF >> "board.front.wrl"
Transform {
    children [
        DirectionalLight {
            on TRUE
            intensity 0.63
            ambientIntensity 0.21
            color 1.0 1.0 1.0
            direction 0.1 -0.1 -1
        }
EOF
cat "board.wrl" >> "board.front.wrl"
echo "] }" >> "board.front.wrl"

And yes, this method is a bit crude but it does work.

In order to reduce this 3D model into 2D image, ray tracing comes in handy. However, for that we need an external tool. I found that RayHunter worked great from the command line. While you only need that tool, you might want to check Castle Model Viewer as it can show you parameters in a bit more interactive way. Please note you also need libpng-dev package for PNG output.

cd ~/Downloads
wget https://master.dl.sourceforge.net/project/castle-engine/rayhunter/rayhunter-1.3.4-linux-x86_64.tar.gz
tar xzvf rayhunter-1.3.4-linux-x86_64.tar.gz
sudo install -m 0755 ~/Downloads/rayhunter/rayhunter /usr/local/bin/rayhunter

sudo apt install --yes libpng-dev

With prerequisites out of way, now we can finally export our board, looking directly at its front side. Camera position is a bit further away than I usually need it for my board but resolution of 4320x4320 pixels is large enough to then later crop the unneeded pixels.

rayhunter classic 7 \
    4320 4320 \
    "board.front.wrl" \
    "board.front.png" \
    --camera-pos 0 0 6 \
    --camera-dir 0 0 -1 \
    --scene-bg-color 1 1 1

You can see what each parameter does in rayhunter documentation

If all went fine, this will give you PCB board nicely rendered but with quite a lot of whitespace. In order to remove those extra pixels, I like to use ImageMagick.

sudo apt install -y imagemagick

Using convert command I trim all the whitespace, resize it to 1060x1060 pixels, add 10 pixels boarder on each side, and finally extend it to the full 1080x1080 size.

convert \
    "board.front.png" \
    -trim \
    -resize 1060x1060 -normalize -density 600 \
    -bordercolor white -border 10 \
    -gravity center -extent 1080x1080 \
    "board.front.png"

Congrats, your PCB image should be looking good just about now.

PS: You can do this from any PCB orientation by just adjusting camera and light position.

Supermicro IPMI Password Reset

For a long while I had issues with my AMD Epyc Supermicro board. While the issues “smelled” like memory, it wasn’t easy to pinpoint why exactly the system would get stuck couple times a week. It all pointed either toward the motherboard or, more likely, the processor itself. Either way, I decided to obtain an equivalent board.

Being cheap bastard, I didn’t want to buy it new but decided to peruse local Craigslist (no luck) and eBay. Being my lucky day, I got a really good price on a similar but not exactly the same board. Yes, the new board was Intel, used slightly more watts, less memory bandwidth, and fewer LAN ports. That said, it allowed me to use the same ECC memory I used on the AMD board, and those 2 LAN ports were of the 10G variant which does make me want to buy a new switch. Suffice to say, I don’t consider it a downgrade.

But, as always when used boards are bought, one doesn’t necessarily have all the passwords. In my case, the missing password was for the remote management interface (IPMI). Clearing CMOS was of no help.

I did search the Internet for solutions but none worked exactly as written. After slight modifications, I managed to erase my IPMI configuration so I might as well share what worked for me in 2024. I am sure that in a couple of years, these instructions will be incomplete too but that’s a problem for a future me.

First of all, you need IPMI tools. I downloaded them directly from Supermicro as these low-level operations tend not to work properly when using generic tools you might have with your Linux distribution. And yes, these instructions will be Linux-based. If you have something else, check the tools regardless as they contain executables for other OS’ too.

For Linux, we want to extract the IPMICFG-Linux.x86_64 file and then allow for its execution by setting the x bit:

chmod +x ./IPMICFG-Linux.x86_64

An excellent way to check if the tool is working (and to double-check if you’re on the correct host) is to see what the current IPMI settings actually are. Run:

./IPMICFG-Linux.x86_64 -m

Finally, we can fully reset IPMI by completely erasing its configuration (-fde) followed by a factory reset of the actual use Some guides would have you just perform a user reset but that didn’t work for me. I had to reset both and in this order:

./IPMICFG-Linux.x86_64 -fde -d
./IPMICFG-Linux.x86_64 -fd 3 -d

Finally, the usual ADMIN/ADMIN can be used to get into IPMI.