ResetBox

Illustration

It all started with an itch to scratch. My modem occasionally misbehaves and needs a reboot. Unfortunately, I have it mounted way back in the 19" rack and accessing it requires removing back panel. And no, I cannot easily unplug it’s connected to UPS and wires are all tucked in the back. What I needed was a button in one of rack spacers allowing for reboot of equipment in the back.

Since I couldn’t find something like this readily available, I decided to build it.

First I enumerated all the equipment I have. One modem (12 V), one NUC (19 V), and finally two wireless access points (48 V). I was at crossroads. As voltage range is quite wide I had to choose if I want to make device work of specific voltage (cheaper) or to have a single device covering all the bases (more expensive). While going cheaper was tempting, I decided to go with wide input after all. It meant I can design thing only once and use it later for some other project without having to check voltage rating each time.

This decision also made me select DC-DC converter module. Making my own wide range converter would be possible but getting it just right would take annoyingly long time and actual cost wouldn’t be much lower. Yes, at $5 R-78HE5.0-0.3 is not cheap but it’s simplicity is worth every penny. If I ever decided to produce this box in bigger quantities, this would probably be the first thing to go in order to lower expenses and simplify design.

While not necessarily mandatory, I also opted to include EMC filter for DC-DC converter. Considering low currents it’s probably an overkill but I really didn’t want to find out later that I’m spraying too much noise around my rack. Design improvement would be to either remove filter all together (probably possible due to low current requirements) or make it much simpler and hopefully omitting SMD inductor. I hate to solder that stuff.

It took me a while to find a button as I wanted momentary switch with LED (excellent for giving status) and it had to look nice in rack. I finally found Adafruit’s S 560. This button is awesome in person.

To connect button, I selected JST S3B-XH. I needed the minimum of three wires (GND, LED, and switch input) and I already had bunch of premade cables made in the course of another project. If I had only this project to worry about, I would probably select 4-wire connector as it would make soldering a bit easier. Not it’s too hard now - just wire GND to both LED and NC input and you’re golden.

Voltage connector choice was extremely easy as literally all devices I wanted to control already used barrel connector. Since 2.1 mm barrel had became a defacto standard for network cameras, this also meant I could count on connecting it all with factory cables making the whole setup a bit nicer.

Speaking of which, I mulled long time about whether I should support only barrel setup with positive voltage on tip and negative voltage on sleeve or the inverse one too. All network devices I ever owned have positive tip. However, I decided to allow negative tip setup to. I am 95% sure this was completely unnecessary precaution but at least I don’t need to worry about it.

As far as saving money goes, supporting only positive tip barrel connectors, would allow me to remove diode bridge on input and allow me to replace mechanical relay with MOSTFET. As I’m making only 3 devices, for now I value flexibility the most. But decision to include these still bothers me a bit.

Housing hardware fell to Hammond 1593KBK. This is a reasonably sized and more important reasonably priced case with side panels. While I usually replace such panels with custom PCB with cutouts in place of connectors, this time I opted for 3D printed version. It’s definitely cheaper than PCB but it does require you to have printer. And no, it’s not as nice as PCB panel but it looks good enough - especially since device will sit in the back of the rack.

Lastly, I had to select microprocessor that would handle all logic. As most of my devices, I simply went with selecting Microchip’s part. I needed two outputs (LED and relay) and a single input (switch). In order to avoid complications, I like to keep programming lines separate too (MCLR, PGD, and PGC). Yes, you can share these lines with logic if you’re careful but having them reserved is one way to make sure. Once you include power supply, there is variety of 8-pin microcontrollers to chose from.

Finally I went with PIC12F1501-I/SN. It has 8 pins which is what we needed. SOIC package makes it easy to solder and internal oscillator minimizes extra components needed.

Code running on chip is as easy as it gets. What follows is the main loop.

void main(void) {
    io_led_on();
    while(true) {
        if (io_switch()) {
            if (debounceCounter < DEBOUNCE_MAX) {
                debounceCounter++;
            } else {
                intensityCounter++;
                if ((intensityCounter % 8) == 0) { io_led_on(); } else { io_led_off(); }
            }
        } else {
            if (debounceCounter == DEBOUNCE_MAX) {
                io_relay_on();
                for (int i = 0; i < RESET_COUNTER; i++) {
                    io_led_on(); wait_short();
                    io_led_off(); wait_short();
                }
                io_relay_off();
            }
            io_led_on()
            debounceCounter = 0;
        }
    }
}

If button is pressed, counter (debounceCounter variable) gets increased while button is pressed and until it reaches maximum. Side effect of handling counter like this is automatic debouncing and making sure short, accidental, swipes don’t activate reboot functionality. Once at maximum, counter stops and we dim the LED so user knows reboot will follow once button is depressed. Dimming is actually done by poor man’s PWM - just turn LED manually on 1/8 of interval (when counter is divisible by 8).

Second path executes only when the button is not pressed. If debounce counter is at maximum, a simple reset sequence starts. Here we disconnect relay (ironically by turning it on as it’s normally-connected setup) and LED blinks for 4 seconds. Then we turn relay back off so that power returns to the device.

It’s probably the simplest code I’ve written in quite a while.

If you want to check it a bit more, feel free to look up both board and code at GitHub.

Multiple IPv6 Networks with Mikrotik

Illustration

Setting up IPv6 for home network is simple enough with Mikrotik. I actually already wrote about it and that IPv6 guide is still perfectly valid. However, what if you have multiple bridges and a single IPv6 address won’t do (e.g. home and guest network)?

Well, first you need to be lucky enough to have provider willing to give you multiple /64 prefixes as stateless address autoconfiguration (SLAAC) cannot work with prefix longer than that.

For this to work you need a prefix hint to let your provider know you want something bigger. Despite RIPE recommending at least /56 prefix length, my Comcast-provided connection gives /60 at most even if you request a shorter prefix.

/ipv6 dhcp-client
add interface=^^ether1^^ pool-name=general-pool6 request=prefix prefix-hint=::/48 add-default-route=yes

Once you receive IP allocation, check on Status tab you indeed got something shorter than /64. If you got only /64 allocation, you will not be able to have two independent networks. In my case, /60 enables me up to 16 networks.

Next we add additional router address to use for advertisement.

/ipv6 address
add address=::1 from-pool=general-pool6 interface=^^bridge2^^ advertise=yes

Lastly, we need to setup router announcements.

/ipv6 nd
add interface=^^bridge2^^ ra-interval=20s-60s

And that’s it. Now your guests can enjoy IPv6 too.

Colored CPU usage in TMUX

Illustration

Those spending a lot of time in Linux command line know the value of a good terminal multiplexer. Even in the age of GUI terminal windows, using tmux will speed your work while many customizations it offers will make for comfortable environment.

Knowing its flexibility, for my Minecraft server, I wanted tmux status line to include something I never tried before - CPU usage. Yes, one could use top, htop, glances, or any other of the many monitoring tools but I didn’t want the whole screen occupied by it. I wanted just a small unobtrusive note in the corner.

There are multitude of ways to retrieve CPU usage but I found vmstat works well for me and thus the first version just placed its parsed output in the bottom-right corner:

set-option -g status-interval 1
set-window-option -g status-right ' #( vmstat 1 2 | tail -1 | awk "{ printf 100-\$15; }" )% '

While this satisfied my original requirements, I started thinking about colors. Could I make it more colorful?

Search immediately found me a few tmux plugins and one of them looked really promising. While using it would be perfectly fine for my use case, I don’t like dependencies and thus I tried to find tmux-only solution for colors.

A bit of testing later and I noticed shell commands (enclosed in #()) are processed before formatting directives. So, if I manage to return the formatting from shell, tmux will do further processing and colors will be there. A bit of awk-ing later and this is what I came up with.

set-option -g status-interval 1
set-window-option -g status-right ' #( vmstat 1 2 | tail -1 | awk "{ USAGE=100-\$15; if (USAGE < 20) { printf \"#[fg=green,bright]\"; } else if (USAGE < 80) { printf \"#[fg=yellow,bright]\"; } else { printf \"#[bg=red,fg=white,bright]\"; }; print \" \" USAGE \"% \" }" )'

It’s still vmstat based output but now awk appends color formatting strings for different usage levels. If usage is lower than 20%, a bright green foreground is used; lower than 80% results in a bright yellow text; and anything higher results in a bright red background.

It’s a simple color-coded way of showing CPU usage at a glance.


PS: If you want extra status line text after CPU percentage, consider adding #[bg=default,fg=default] to stop color “bleeding” to remainder of the line.

LACP Between Mikrotik and XigmaNAS

Illustration

My Mikrotik RB100AHx4 network router has only 1 Gbps ports and recently I have started noticing the need for a bit more bandwidth toward my NAS. As this router has no SFP+ ports, there is no easy way to increase speed. However, since my NAS machine has 4 network ports, there is a way forward - LACP.

While there are many ways to aggregate network links, I found 802.3ad link aggregation is the way to go. Not only all modern network devices support it but it also comes almost without any costs in the terms of CPU processing. To make it better, it also allows unbalanced links (e.g. router is LACP while NAS is just a single port setup). While this is not a desired end state, it makes both initial setup and potential later troubleshooting much easier. While not the focus of my setup, additional resilience of link to the cable damage is nothing to frown about either.

To create link aggregation on Mikrotik’s side, one has to first remove all ports from existing bridge (Bridge, Ports). Then we can create the new bond interface (Interfaces, Bonding) with mode set to 802.3ad and transmit hash policy set to layer 2 and 3. All interfaces we want to bond together should be listed under slaves. Assuming we want a 3-port bond, it would be something like this:

/interface bonding
add name=bonding-nas slaves=^^ether1^^,^^ether2^^,^^ether3^^ mode=802.3ad transmit-hash-policy=layer-2-and-3

Once bond interface is created, just add it to a bridge as you would any other interface (Bridge, Ports) and your work on Mikrotik side is done.

Illustration

On XigmaNAS, setup is equally easy. Due to LACP gracefully handling our currently asymmetric link, we can simply connect to the web GUI as we normally would. There go into Network, Interface Management, LAGG tab. Add a new interface selecting LACP (Link Aggregation Control Protocol) as protocol and by clicking all the ports you want to participate. Then go back to Network, Interface Management and select newly created interface (lagg0) from LAN’s dropdown menu. Once you save the settings, you can go ahead and reboot your NAS.

Enjoy your bandwidth and resilience increase!

While LACP is nice don’t expect wonders from it. As far as resiliency goes, things are as you would expect them - if you set it up with 3 ports as I did, you can lose 2 cables before connectivity is gone. For bandwidth things are not as simple. Just having 802.3ad link makes no difference if you have just a single connection. All packets are still going to travel over only a single cable (mandatory to avoid packet reordering). Since most test tools use just one connection, you will see no improvement.

Even when you are dealing with multiple connections, you depend on the gods of hashing. As your hash space is only N wide (3 for examples above), you might have multiple connections sharing the same hash and thus the same physical link while rest of links stay bored. Only if you have many different connections you can count on approximately equal traffic distribution and full benefits. For my NAS setup this is not really an issue as I can count on multiple connections from either one (SMB3) or multiple machines.

Not a magic bullet but it’s a cheap way to double bandwidth as long as you are aware of limitations.


PS: Once setup is done, you can try setting up fast LACP rate (lacp-rate=1s). This will enable for faster detection of link failures when interface itself is up but there is a problem between devices (e.g. you have damaged cable or misbehaving switch in between). In reality, you probably don’t need it but you might as well turn it on if network equipment supports it.

PPS: Asynchronous port setup is beneficial only during deployment. If you set 4-port LACP on one side and 2-port LACP on another, you are not reaping any benefits for two extra ports. When you are done with configuration you always want the same port count on both network devices.

Curious Case of Missing DHCP Renewal

Most of my network is based on Mikrotik routers and they serve me well. Even though I have decent mix of different operating systems present and I never had any issues. Until I got Ubuntu 19.10 Server up and running.

At the first glance, all worked perfectly and then suddenly machine would stop responding. Quick check from console interface pointed toward the culprit - there was no IP address present. Check on Mikrotik for DHCP leases was showing “waiting” state. Manually running dhclient on Ubuntu did temporarily resolve problem only to have address gone at the next DHCP expiration interval.

For some reason Ubuntu 19.10 Server would do DHCP once and then give up on further renewals. It was as if dhclient was crashing. Check of the syslog has shown curious entries:

systemd-networkd: eno1: DHCPv4 address 192.168.1.103/24 via 192.168.1.1
systemd-networkd: eno1: Could not set DHCPv4 route: Network is unreachable
systemd-networkd: eno1: Failed

I had DHCP classless static route option set on my Mikrotik and it seems dhclient really didn’t like my setup with multiple CIDRs. As soon as I removed all entries except for the default one, it started working properly.

On a hunch, I tried changing the network order to have default route as the first one and it worked. What also worked was omitting default gateway altogether and only having narrower CIDR definitions.

There is nothing in RFC3442 forcing this order and either ordering works for all other clients - including non-server Ubuntu 19.10. I guess something in Ubuntu Server configuration is making dhclient misbehave when things are not exactly how it wants it.

However, considering either ordering is perfectly valid, I decided to take a simpler route and update my classless route definition.