Encrypted ZFS (A Slightly Parallel Edition)

Initial encryption of ZFS pool does require a bit of work - especially when it comes to initial disk randomization. Yes, you could skip it but then encrypted bits are going to stick out. It’s best to randomize it all before even doing anything ZFS related.

The first problem I had with the old setup was the need to start randomizing each disk separately. Since operation takes a while (days!), this usually resulted in me starting all dd commands concurrently thus starving it of resources (mostly CPU for random number generation).

As my CPU can generate enough random data to saturate two disks, it made sense to use parallelize xargs using the serial number (diskid) of each disk as an input. While using /dev/sd* would work, I tend to explicitly specify disks serial number as it’s not destructive if ran on the wrong machine. I consider it a protection against myself. :)

The final command still takes ages but it requires only one window and it will take care to keep only two disks busy at a time:

echo "^^DISK-ID-123^^ ^^DISK-ID-456^^ ^^DISK-ID-789^^ ^^DISK-ID-012^^" | \
  tr ' ' '\n' | xargs -I '{}' -P 2 \
  dd if=/dev/urandom of=/dev/diskid/{} bs=1M

After drives are “cleaned”, I do the encryption (one-by-one this time):

echo "^^DISK-ID-123^^ ^^DISK-ID-456^^ ^^DISK-ID-789^^ ^^DISK-ID-012^^" | \
  tr ' ' '\n' | xargs -I '{}' \
  geli init -e AES-XTS -l 128 -s 4096 '/dev/diskid/{}'

There used to be times when I encrypted each disk with a separate password and that’s still a bit more secure than having a single one. However, with multiple passwords comes a great annoyance. These days I only have a single password for all the disks in the same pool. It makes my life MUCH easier.

In theory, somebody cracking one disk will immediately get access to all my data but in practice it makes no big difference. If somebody decrypted one disk, they either: found a gaping hole in Geli and/or underlying encryption and thus the other disks will suffer the same fate and there’s nothing I can do; or they intercepted one of my keys. As I always use all the keys together, chances are that intercepting one is the same effort as intercepting them all. So I trade a bit of security for a major simplification.

Now we get to attach all encrypted drives:

echo "^^DISK-ID-123^^ ^^DISK-ID-456^^ ^^DISK-ID-789^^ ^^DISK-ID-012^^" | \
  tr ' ' '\n' | xargs -I '{}' \
  geli attach '/dev/diskid/{}'

And the final step is creating ZFS pool, using RAIDZ2 and allowing for loss of two disks before data is compromised:

zpool create \
  -o autoexpand=on -m none -O compression=gzip-7 -O atime=off \
  -O utf8only=on -O normalization=formD -O casesensitivity=sensitive \
  ^^Data^^ raidz2 \
  /dev/diskid/^^DISK-ID-123^^.eli  /dev/diskid/^^DISK-ID-456^^.eli /dev/diskid/^^DISK-ID-789^^.eli /dev/diskid/^^DISK-ID-012^^.eli

And that’s it - pool is ready for all the data you can throw at it.

PS: Yes, I am still using Geli - native ZFS encryption didn’t find its way to FreeBSD yet.

PPS: If machine goes down, it is enough to re-attach Geli disks followed by restart of the ZFS daemon:

echo "^^DISK-ID-123^^ ^^DISK-ID-456^^ ^^DISK-ID-789^^ ^^DISK-ID-012^^" | \
  tr ' ' '\n' | xargs -I '{}' \
  geli attach '/dev/diskid/{}'

/etc/rc.d/zfs onestart