Seattle Code Camp 2018

Registrations for Seattle Code Camp 2018 are open. If you come on September 15th you can hear me speak about cryptography failures. All intention is for this to be a lightweight and funny talk but let’s see how funny crosses the language barrier. :)

If you attend, feel free to say hi, whether you attend my talk or not. :)

Installing Git on Embedded XigmaNAS

Having Git on NAS server comes in handy and I was quite annoyed when my workaround that worked on NAS4Free 11.1 stopped working with the advent of XigmaNAS 11.2. Well, I guess it was time to make some adjustments.

I won’t get into how the whole process works as I already explained it previously. Suffice to say it stopped working on 11.2 due to even tighter disk space restrictions it brought. While originally I just didn’t have space to install software, this time there wasn’t enough space to even get needed packages.

So I adjusted my unionfs creation process to include two extra directories (/var/db/pkg/ and /var/cache/pkg/):

rm -rf /mnt/.unionfs/* |& sed '/^$/d' |& sed -e 's/^/  /'
for LOCATION in "usr/local" "var/db/pkg" "var/cache/pkg"
do
  if mount -t unionfs | grep "/mnt/.unionfs/$LOCATION" >/dev/null
  then
    echo -en "${ESCAPE_OUTPUT}"
    mount -t unionfs |& grep "/mnt/.unionfs/$LOCATION" | sed -e 's/^/  /'
    echo -en "${ESCAPE_RESET}"
  else
    echo -en "${ESCAPE_ISSUE}"
    mkdir -p "/mnt/.unionfs/$LOCATION/" |& sed '/^$/d' |& sed -e 's/^/  /'
    mkdir -p "/$LOCATION/"
    mount_unionfs "/mnt/.unionfs/$LOCATION/" "/$LOCATION/" |& sed '/^$/d' |& sed -e 's/^/  /'
    echo -en "${ESCAPE_RESET}"

    echo -en "${ESCAPE_OUTPUT}"
    mount -t unionfs |& grep "/mnt/.unionfs/$LOCATION" | sed -e 's/^/  /'
    echo -en "${ESCAPE_RESET}"
  fi
done

Adjusting initialization script with this allowed me to install Git once again.

EEPROM Access Routines For PIC18

While I was playing with CAN bus, I needed to write something to EEPROM memory - to keep it safe between reboots. Imagine my surprise when calls to eeprom_read and eeprom_write returned errors:

warning: unsupported: The Read_b_eep routine is no longer supported. Please use the MPLAB X MCC.
warning: unsupported: The Busy_eep routine is no longer supported. Please use the MPLAB X MCC.
warning: unsupported: The Write_b_eep routine is no longer supported. Please use the MPLAB X MCC.

It seems PIC18 series doesn’t support old EEPROM routines that worked on PIC16 but, at the same time, my PIC is old enough not to be properly supported in the suggested MCC. I guess it was time to write PIC18 version of these functions myself.

Reading over documentation for PIC18F25K80, I was pleasantly surprised that EEPROM writing hasn’t really changed. My assembly code I wrote back in PIC16F84 days was quite close to what’s needed these days too. The only significant change was expanding EEADR to allow for more than 256 bytes. Yes, there are some shenanigans with setting EEPGD/CFGS bits but that’s done only once.

My replacement for eeprom_read function was similarly named eeprom18_read and it’s really a drop-in replacement. Not to be the one to dwell in past, I opted to change my ASM code to XC8 flavored C. It’s essentially just setting up EEPROM address, followed by read request, a short pause to get the data, and then reading the data itself.

unsigned char eeprom18_read(unsigned int offset) {
  EECON1bits.EEPGD = 0; // accesses data EEPROM memory
  EECON1bits.CFGS = 0;  // accesses data EEPROM memory

  EEADRH = offset >> 8;
  EEADR = offset;

  EECON1bits.RD = 1; // initiates an EEPROM read
  Nop();             // it can be read after one NOP instruction

  return EEDATA; }

Writing was a bit more involved but not too much so. After setting EEPROM address and data, it was important to allow EEPROM write and disable interrupts as ancient EEPROM unlock sequence (0x55 0xAA) must not be interrupted. After write is initiated, code will wait for it to complete and then restore interrupts. For safety it will also disable further writes to EEPROM until it’s needed again.

void eeprom18_write(unsigned int offset, unsigned char value) {
  EECON1bits.EEPGD = 0;  // accesses data EEPROM memory
  EECON1bits.CFGS = 0;   // accesses data EEPROM memory

  EEADRH = offset >> 8;
  EEADR = offset;

  EEDATA = value;

  unsigned char oldGIE = GIE;  // interrupts be disabled during this code segment
  GIE = 0;

  EECON1bits.WREN = 1;  // allows write cycles

  EECON2 = 0x55;  // write sequence unlock
  EECON2 = 0xAA;  // write sequence unlock

  EECON1bits.WR = 1;     // initiates a data EEPROM erase/write cycle
  while(EECON1bits.WR);  // waits for write cycle to complete

  EECON1bits.WREN = 0;  // disable write
  GIE = oldGIE;         // restore interrupts
}

Encrypted ZFS (A Slightly Parallel Edition)

Initial encryption of ZFS pool does require a bit of work - especially when it comes to initial disk randomization. Yes, you could skip it but then encrypted bits are going to stick out. It’s best to randomize it all before even doing anything ZFS related.

The first problem I had with the old setup was the need to start randomizing each disk separately. Since operation takes a while (days!), this usually resulted in me starting all dd commands concurrently thus starving it of resources (mostly CPU for random number generation).

As my CPU can generate enough random data to saturate two disks, it made sense to use parallelize xargs using the serial number (diskid) of each disk as an input. While using /dev/sd* would work, I tend to explicitly specify disks serial number as it’s not destructive if ran on the wrong machine. I consider it a protection against myself. :)

The final command still takes ages but it requires only one window and it will take care to keep only two disks busy at a time:

echo "^^DISK-ID-123^^ ^^DISK-ID-456^^ ^^DISK-ID-789^^ ^^DISK-ID-012^^" | \
  tr ' ' '\n' | xargs -I '{}' -P 2 \
  dd if=/dev/urandom of=/dev/diskid/{} bs=1M

After drives are “cleaned”, I do the encryption (one-by-one this time):

echo "^^DISK-ID-123^^ ^^DISK-ID-456^^ ^^DISK-ID-789^^ ^^DISK-ID-012^^" | \
  tr ' ' '\n' | xargs -I '{}' \
  geli init -e AES-XTS -l 128 -s 4096 '/dev/diskid/{}'

There used to be times when I encrypted each disk with a separate password and that’s still a bit more secure than having a single one. However, with multiple passwords comes a great annoyance. These days I only have a single password for all the disks in the same pool. It makes my life MUCH easier.

In theory, somebody cracking one disk will immediately get access to all my data but in practice it makes no big difference. If somebody decrypted one disk, they either: found a gaping hole in Geli and/or underlying encryption and thus the other disks will suffer the same fate and there’s nothing I can do; or they intercepted one of my keys. As I always use all the keys together, chances are that intercepting one is the same effort as intercepting them all. So I trade a bit of security for a major simplification.

Now we get to attach all encrypted drives:

echo "^^DISK-ID-123^^ ^^DISK-ID-456^^ ^^DISK-ID-789^^ ^^DISK-ID-012^^" | \
  tr ' ' '\n' | xargs -I '{}' \
  geli attach '/dev/diskid/{}'

And the final step is creating ZFS pool, using RAIDZ2 and allowing for loss of two disks before data is compromised:

zpool create \
  -o autoexpand=on -m none -O compression=gzip-7 -O atime=off \
  -O utf8only=on -O normalization=formD -O casesensitivity=sensitive \
  ^^Data^^ raidz2 \
  /dev/diskid/^^DISK-ID-123^^.eli  /dev/diskid/^^DISK-ID-456^^.eli /dev/diskid/^^DISK-ID-789^^.eli /dev/diskid/^^DISK-ID-012^^.eli

And that’s it - pool is ready for all the data you can throw at it.

PS: Yes, I am still using Geli - native ZFS encryption didn’t find its way to FreeBSD yet.

PPS: If machine goes down, it is enough to re-attach Geli disks followed by restart of the ZFS daemon:

echo "^^DISK-ID-123^^ ^^DISK-ID-456^^ ^^DISK-ID-789^^ ^^DISK-ID-012^^" | \
  tr ' ' '\n' | xargs -I '{}' \
  geli attach '/dev/diskid/{}'

/etc/rc.d/zfs onestart

Failed to Load SELinux Policy

After a failed yum upgrade (darn low memory) I noticed my CentOS NTP server was not booting anymore. Look at console showed progress bar still loading but pressing Escape showed the real issue: Failed to load SELinux policy, freezing.

The first thing in that situation is to try booting without SELinux and the easiest way I found to accomplish this was pressing e on boot menu and then adding selinux=0 at the end of line starting with linux16. Continuing boot with Ctrl+X will load CentOS but with SELinux disabled.

As I actually don’t run my public-facing servers without SELinux, it was time to fix it. Since I didn’t have package before, I installed selinux-policy-targeted but I would equally use reinstall if package was already present. In any case, running both doesn’t hurt:

sudo yum install -y selinux-policy-targeted
sudo yum reinstall -y selinux-policy-targeted

Finally we need to let system know SELinux should be reapplied. This can be done by creating a special .autorelabel file in the root directory followed by a reboot:

sudo touch /.autorelabel
sudo reboot

During reboot SELinux will reapply all labeling it needs and we can enjoy our server again.