Driving the Plunger (part 2)

This is part 2 out of 3 in the series (see part 1 or 3).

So, we’re continuing on PlungerDriver development. If you’re not up-to-date, do check part 1.

For electronics, I decided early to keep it rather simple and use only components I had lying around. I mean, I was waiting for motors but there was no reason why I couldn’t just cobble something up.

The first decision I almost always like to make is how I will control the darn thing. In this case I saw myself using combination of a rotary button and a small OLED screen.

The rotary button I had was the one I used for a few projects in the past, which was Bourns PEC12R, more precisely its PEC12R-4220F-S0024 variant. It’s 12mm square, has detents (which I love), and you can also press it to get button functionality. Essentially, you have left/right/button controls. Ideal for something like this.

The screen I also had decided early on. Small 128x64 I2C OLEDs are easy to obtain and easy to drive. Even better, I had quite a few of different ones laying around from previous projects (e.g., USB OLED).

Input-wise, I wanted to connect it all via just a standard 3.81 mm Phoenix-style MC connector. These are really easy to work with, available from multiple manufacturers, and I have a load of these always in my storage.

For output, I decided to use JST XH connector. Using a different connector here was just to make it really clear what goes where. If I can do it, I always try to do different connectors for each functionality board might have. Makes mistake a lot less likely to happen.

As a trigger for paste dispensing, I decided to use both an on-board button and an external connector. While having a button on board was a handy thing to have during the testing, I really wanted an external connection for a foot pedal. Not only are they easy and cheap to get, but they’re the only practical way to trigger something while keeping both of your hands free. And yes, in order to avoid potential errors, I used a JST PH connector here.

The last selection was a PIC microcontroller. And yes, I knew it would be a PIC since those are the only ones I have in my storage. I did play with some other microcontrollers out there, but my pipeline and knowledge are so heavily skewed toward PIC that I always select it for projects like this. And here I opted for PIC16F1454. While many other Microchip devices would work equally well, I had a load of PIC16F1454 in my storage.

PCBs were made by PCBWay and provided to me for free. While they do sponsor me occasionally, I actually did quite a few orders with them where I paid my own money to get stuff made. They were fast (well, much faster than me doing my part) and boards ended up being exactly what I ordered. Unfortunately, that meant they had a few errors - none of which were PCBWay’s fault.

The first issue I noticed was actually reaching programming ICSP header. OLED simply didn’t leave enough space to reach the darn thing. Fortunately, I use Pogo pins with my PICkit so I could reach the header from the bottom. Had I been using just a standard header, it would have been really a deal breaker.

Second, a more serious issue was with the motor driver simply not working. All voltages were just fine when the motor was not connected. However, connect the motor and it would go nowhere. It took me a bit of troubleshooting to notice I messed up Vref voltage divider and that was causing any load to trip the current limit function on my motor driver.

As I already soldered the whole board, I had to unsolder the OLED to reach resistors underneath and place a bodge wire. Since I use lead-free solder, it took a considerable amount of heat to fix that and the board survived that without any issues. The same couldn’t be said for my OLED display which didn’t work after I placed it back. So I had to unsolder it again in order to solder a replacement.

Why I mention this? Well, playing with higher temperature quite often damages the soldermask. And I had that experience with many PCB manufacturers, including PCBWay. However, either PCBWay changed their soldermask formulation to handle the higher temperature or this red soldermask I selected is a bit more resilient as after cleaning it with alcohol, I couldn’t even tell it was ever worked on.

And that’s all for this installment of the project. See you next time when I get the firmware running and see how this works in practice.

Driving the Plunger (part 1)

This is part 1 out of 3 in the series (see part 2).

After playing with TB67H450FNG motor driver for a project of mine, I was really taken away by this little jelly bean chip. While rather basic, it’s really protective of its user. Not only does it offer a nice current limiting functionality defined by a single resistor, but it also has basic back-EMF current sink built in. Even if that goes wrong, there is a thermal shutdown circuit to save the day. And all that in the small 9-pin SOP package.

Yes, you read it right - 9 pin. While 8 pins are exactly where you expect them, the 9th pin is exposed beneath the chip itself and it’s pain to solder. Before you ask, yes, solder it you must.

At first glance, soldering it should not be too difficult. Just put some paste on pad, hit it from above (or below) with some heat and things will work out. And that is true if you place just enough solder paste. Place too much and you’ll get shorted pins from the back that are really annoying to get rid of. Place too little and it will not make contact thus leading to overheating and thermal shutdown. While not catastrophic, I can assure you troubleshooting this is not fun.

What I needed for this was a way to always place the same amount of paste on the bottom pad. My hands, while marvelous in their own right, were just not precise enough for this duty. What I needed was a way to push some paste out from a syringe in a repeatable manner.

As I was playing with a motor driver, thoughts immediately went toward using it to drive some gears and push a small rod into the back of the syringe in order to get its rotation into a linear motion.

With that idea on my mind, I went on to selecting components. I already knew which motor driver I wanted to use but it took me a while to decide on the motor.

The first thought was to use a 3D printer stepper driver I had around. It would be ideal for this as you can control it really precisely, it holds position well, and it can handle some torque even without gearing. That idea lasted until I took it into my hand. That darn thing was heavy. When I accounted for all parts I needed for pushing into the syringe, darn things would be way too top heavy for my taste. And that’s even without taking bulkiness into account.

No, I wanted something a bit slender and thankfully, such motors exist. Even more importantly for my case, they already had a shaft on saving me an extra step. After measuring my paste syringes, I decided onto 6V 15 rpm motor with 55 mm M3 thread.

With an M3 rod integrated into the motor, my only task left for the mechanical portion of the project was to design a plunger. And I did so in TinkerCAD. It was just a simple body taking the motor from one side and taking a custom-designed plunger from the other.

With all the mechanical stuff out of the way, it was time to design the electronics. But that’s a story for the next week.

Watchdog setup for Supermicro server

EPYC server processors are really nice, when they work. However, lately with kernel 6.2 I started getting dreadful “CPU stuck” errors that lead to hanging system. Normal person might revert to an older kernel. Me? I decided to turn on the watchdog.

In case you don’t know, watchdog is a functionality that, once turned on, will require your system to notify it every once in a while that it’s still active. If notification is not received within given time interval, system is assumed stuck and thus it gets rebooted. Best of all, this is done on a hardware level and thus no hanging application or CPU will prevent it.

To confuse things a bit, my Supermicro M11SDV-4CT-LN4F server, seems to have two watchdog systems. One is part of Epyc platform itself and controlled via BIOS setting. That one has 5 minute interval and no matter what I couldn’t get it working properly. I mean, I could get it running but, since there was no easy way to reset it, system would reboot every 5 minutes, no matter what.

The second watchdog is the part of AST2500 chipset that handles other IPMI functions. And this one was well supported from Linux command line using ipmitool utility. To see its status, just ask ipmitool for that information:

ipmitool mc watchdog get

But there is no option to turn it on. However, one can always send raw commands and I was fortunate to see that somebody already did. Not to get into too much details, the last two numbers in the string of hexadecimal values are the only thing you generally want to change - time interval. In example below, I decided to go for 610 seconds (0x17D4 in 0.1 s units).

ipmitool raw 0x06 0x24 0x04 0x01 0x00 0x00 0xD4 0x17

This will start a ticking bomb that will, if not defused within the given interval, reboot your computer. So, why did I select 10 minutes and 10 seconds? As many things, this was completely subjective.

Well, no matter what, I wanted this watchdog not to interfere with my normal server operation. Since a normal reboot takes about 5 minutes, I wanted to have 5 minutes on counter even if I reboot system myself just before watchdog would reset. So, if I select 10 minute interval and reset it every 5 minutes, this gives me 5 minutes of extra time I might need for reboot. But why extra 10 seconds? Well, in case I mess with my watchdog settings and I miss reset at 5 minute mark, I wanted to give an extra chance of reset at 10 minute mark without having to deal with a reboot race condition.

And how might one actually setup watchdog and its reset within Linux? Well, crontab, of course. These two entries were all it took:

@reboot
/usr/bin/ipmitool raw 0x06 0x24 0x04 0x01 0x00 0x00 0xD4 0x17

0,5,10,15,20,25,30,35,40,45,50,55 * * * *
/usr/bin/ipmitool mc watchdog reset

This will turn on watchdog upon every system reset (and yes, once watchdog goes off, you do need to manually turn it back on) and every 5 minutes system will reset its counter if nothing goes awry.

Simple and effective.

ZFS Encryption Speed (Ubuntu 23.10)

There is a newer version of this post

As it became a custom, I retest ZFS native encryption performance with each new Ubuntu release. Here we have results for Ubuntu 23.10 on kernel 6.5 using ZFS 2.2.

Testing was done on a Framework laptop with an i5-1135G7 processor and 64GB of RAM. Once booted into installation media, I execute the script that creates 42 GiB RAM disk that hosts all data for six 6 GiB files. Those files are then used in RAIDZ2 configuration to create a ZFS pool. The process is repeated multiple times to test all different native ZFS encryption modes in addition to a LUKS-based test. This whole process is repeated again with AES disabled and later with a reduced core count.

Since I am testing on the same hardware as for 23.04 and using essentially the same script, I expected similar results but I was slightly surprised to see both raw read and write speed has been reduced by more than 10%. I am not sure if this is due to the new kernel, new BIOS, or some other combination of changes but the performance hit seems quite significant.

However, what I’m most interested in is not necessarily the actual speed but how it’s impacted by encryption. As compared to last year, it seems each GCM encryption mode has taken a few percent hit. We’re still talking about 2 GiB/s for both read and write so I’m not too worried.

Interestingly, while key size had more impact before, it seems that with 23.10 you can count on the same speed regardless if you select 128, 192, or 256 bit key.

If you don’t have AES support and you need CCM, the news is not that good as that code path has gotten significantly worse. Unless you’re stuck on an ancient CPU this is irrelevant I guess as you should never opt for CCM in the first place.

Using ZFS on top of LUKS has gotten slightly better when it comes to writes where it actually lagged the most behind the native ZFS. The improvement is significant but we’re still talking about 30% lower speeds. On read size, there are no changes and it’s the only area where LUKS wins over the native ZFS encryption.

For this release, I also experimentally tried to get power usage for each test run. I did the same by disconnecting the battery and measuring the power the laptop was drawing. This is not the most precise way of measuring it so I might be off but it looked as ZFS encryption was as efficient as it gets when it comes to the power usage.

To summarize, the native ZFS encryption is still live and kicking in Ubuntu 23.10 and might even provide some power usage advantages as compared to LUKS.

PS: You can find older tests for Ubuntu 23.04, 22.10, 22.04, 20.10, and 20.04.

Counting Geigers

A long while ago I got myself a Geiger counter, soldered it together, stored it in a drawer, and forgot about it for literally more than a year. However, once I found it again, I did the only thing that could be done - I connected it to a computer and decided to track its values.

My default approach would usually be to create an application to track it. But, since I connected it to a Linux server, it seemed appropriate to merge this functionality into a bash script that already sends ZFS data to my telegraf server.

Since my Geiger counter has UART output, the natural way of collecting its output under Linux was a simple cat command:

cat /dev/ttyUSB0

That gave me a constant output of readings, about once a second:

CPS, 0, CPM, 12, uSv/hr, 0.06, SLOW

CPS, 1, CPM, 13, uSv/hr, 0.07, SLOW

CPS, 0, CPM, 13, uSv/hr, 0.07, SLOW

CPS, 1, CPM, 14, uSv/hr, 0.07, SLOW

CPS, 1, CPM, 15, uSv/hr, 0.08, SLOW

CPS, 0, CPM, 15, uSv/hr, 0.08, SLOW

In order to parse this line easier, I wanted two things. First, to remove extra empty lines caused by CRLF line ending, and secondly to have all values separated by a single space. Simple adjustment sorted that out:

cat /dev/ttyUSB0 | sed '/^$/d' | tr -d ','

While this actually gave me a perfectly usable stream of data, it would never exit. It would just keep showing new data. Perfectly suitable for some uses, but I wanted my script just to take the last data once a minute and be done with it. And no, you cannot just use tail command alone for this - it needs to be combined with something that will stop the stream - like timeout.

timeout --foreground 1 cat /dev/ttyUSB0 | sed '/^$/d' | tr -d ',' | tail -1

If we place this into a variable, we can extract specific values - I just ended up using uSv/h but the exact value might depend on your use case.

OUTPUT=`timeout --foreground 1 cat /dev/ttyUSB0 | sed '/^$/d' | tr -d ',' | tail -1`
CPS=`echo $OUTPUT | awk '{print $2}'`
CPM=`echo $OUTPUT | awk '{print $4}'`
USVHR=`echo $OUTPUT | awk '{print $6}'`
MODE=`echo $OUTPUT | awk '{print $7}'`

With those variables in hand, you can feed whatever upstream data sink you want.