Encrypted ZFS for My Backup Machine

I already wrote about my ZFS setup. However, for my new machine I made a few changes. However, setup is still NAS4Free based.

The very first thing I forgot last time is randomizing the disks upfront. While not increasing security of new data, it does remove any old unencrypted bits you might have laying around. Even if disk is fresh, you don’t want zeros showing where your data is. Dangerous utility called dd comes handy here (once for each disk):

dd if=/dev/urandom of=/dev/ada0 bs=1M
dd if=/dev/urandom of=/dev/ada1 bs=1M

This takes a while but fortunately it is possible to see current progress with Ctrl+T. Do use tmux to keep session alive as this will take long time (with a big disk, more than a day is not unexpected).

Next, instead of using glabel, I decided to use the whole disk. That makes it easier to move disk later to other platform. No, I am not jumping BSD ship but I think having setup that can change environments is really handy for emergency recovery.

While ZFS can handle using device names like ada0 and ada1 and all shenanigans that come with their dynamic order, I decided to rely on serial number of drive. Normally device labels containing serial number are found under /dev/diskid/ directory. However, NAS4Free doesn’t have them on by default.

To turn them on, we go to System, Advanced, and loader.conf tab. There we add kern.geom.label.disk_ident.enable=1 and reboot. After this, we can use /dev/diskid/* for drive identification.

Those drives I then encrypt and attach each drive:

geli init -e AES-XTS -l 128 -s 4096 /dev/diskid/^^DISK-WD-WCC7KXXXXXXX^^
geli init -e AES-XTS -l 128 -s 4096 /dev/diskid/^^DISK-WD-WCC7KYYYYYYY^^

geli attach /dev/diskid/^^DISK-WD-WCC7KXXXXXXX^^
geli attach /dev/diskid/^^DISK-WD-WCC7KYYYYYYY^^

Finally, I can create the pool. Notice that I put quota around 80% of the total pool capacity. Not only this helps performance but it also prevents me from accidentally filling the whole pool. Dealing with CoW file system when it is completely full is something you want to avoid. And also, do not forget .eli suffix.

zpool create -o autoexpand=on -m none -O compression=on -O atime=off -O utf8only=on -O normalization=formD -O casesensitivity=sensitive -O quota=3T Data mirror /dev/diskid/^^DISK-WD-WCC7KXXXXXXX^^.eli /dev/diskid/^^DISK-WD-WCC7KYYYYYYY^^.eli

zdb | grep ashift
            ashift: 12

Once pool was created, I snapshotted each dataset on old machine and sent it over network. Of course, this assumes your pool is named Data, you are working from “old” machine, and new machine is at 192.168.1.2:

zfs snapshot -r ^^Data^^@Migration
zfs send -Rv ^^Data^^@Migration | ssh ^^192.168.1.2^^ zfs receive -Fs ^^Data^^

This step took a while (more than a day) as all datasets had to be recursively sent. Network did die a few times but resumable send saved my ass.

First I would get token named receive_resume_token from the destination:

zfs get receive_resume_token

And resume sending with:

zfs send -v -t ^^<token>^^ | ssh ^^192.168.1.2^^ zfs receive -Fs ^^Data/dataset^^

Unfortunately resume token does not work with recursion so each dataset will have to be separately specified from that moment onward.

Once bulk of migration was done, I shut every single service on old server. After that I took another (much smaller) snapshot and sent it over network:

zfs snapshot -r ^^Data^^@MigrationFinal
zfs send -Ri ^^Data^^@Migration ^^Data^^@MigrationFinal | ssh ^^192.168.1.2^^ zfs receive -F ^^Data^^

And that is it - shutdown the old machine and bring services up on the new one.

PS: If newly created machine goes down, it is enough to re-attach GELI disks followed by restart of ZFS daemon:

geli attach /dev/diskid/^^DISK-WD-WCC7KXXXXXXX^^
geli attach /dev/diskid/^^DISK-WD-WCC7KYYYYYYY^^
/etc/rc.d/zfs onestart

[2018-07-22: NAS4Free has been renamed to XigmaNAS as of July 2018]

Running Script Without Forking

Default way of running scripts in Linux is that shell forks new process based on hashbang (#!) found in the first line and gives rest of content to that process. And this works beautifully most of the time.

But what if we really need something found only in our current shell?

Fortunately, as long you are using bash, it is easy to run script without creating a separate shell. Just prefix it with dot (.):

./myScript

Some restrictions apply of course - the biggest gotcha being that script should be either bash or with only simple commands as content will be executed directly regardless of hash-bang (#!) specified.

PS: Yes, this works with other shells too, I use bash here as it is most common shell by far.

My Backup ZFS Machine

With my current ZFS pool getting close to its 2 TB limit I started to think about expanding it a bit. However, before doing anything with main NAS server, I decided to get the backup server up to speed. Reason is simple: while I am not a ZFS newbie, I don’t move pools on regular basis and moving backup server’s data will give me nice practice opportunity so I have procedure pinned down when I go to deal with the main storage.

My main parameters were to build machine capable of at least ZFS mirror, be reasonably easy to hide as it will be situated in living room, be as quiet as possible, and cost no more than $300 for all the hardware excluding disks. Note that nowhere I have any conditions on its speed - it is a backup machine I hopefully will never touch again.

For the case I went with Supermicro SC504-203B 19" 1U rack mountable chassis. Some of you might be reasonable wondering which drugs I am taking that causes me to think of 19" rack case as unobtrusive and easy to fit in living room. Well, you are missing a detail that my TV stand is piece of solid wood with opening that is about 25" wide and 2" tall. Just enough for me to slide the case below and for it never to be visible again.

As point of interest, you might notice Supermicro offers SC504-203B and SC505-203B rack cases that have seemingly identical specs. It took me a while to figure out the only difference: 504 has all the connectors in the back and 505 has the motherboard connectors at the front. For my case, more common setup of connectors at the back was better, but your mileage might vary.

Other than its perfect size, this case is one of rare in the lower price range to have enough place for two 3.5" drives. As I am really not too happy with my current situation of backup on a single (albeit) ZFS-formatted drive, upgrading backup to a mirrored combo seemed like a long delayed improvement. Other than that, case has an efficient power supply (80+ Gold) alongside a wide selection of compatible boards.

And there is also a bummer - not all mini-ITX boards can fit this case. Better said, they will fit the case but their IO shield will be a tad too high for an 1U format. Yes, you can always go without the shield or by frankensteining the solution but I actually found a reasonably priced Supermicro board.

I decided to go with X10SBA-L board instead of similarly priced but seemingly more powerfull X10SBA. The “L” board doesn’t have additional Marvell chip and thus it has only two SATA 2.0 ports (Marvell brings additional four SATA 3.0 ports), it has one USB 3.0 and three USB 2.0 ports where Marvell offers additional two 2.0 ports, it has m-SATA port (which cannibalizes one of SATA 3.0 ports), and lastly it lacks eDP. For me neither of those were breaking deal as I intended to use only two disks with the single USB 3.0 port carrying Nas4Free installation.

A bit controversial decision is lack of ECC memory support that is not really frowned upon when dealing with ZFS. In reality, I couldn’t find any ECC board that would fit within my budget. And, while memory is important for ZFS, let’s not forget that this is just a backup machine and that memory errors are usually not catastrophic. Plan is to have ECC RAM on my next ZFS server. But my backup server - mah…

Speaking of memory, I essentially selected the cheapest 2x 4GB modules from manufacturer I trust. While I have bought Crucial, I would have taken Corsair, Kingston, or Samsung the same.

For disks I opted to go with WD Red 4TB model for now. As my current data actually fits into 2.5" 2TB drive, space-wise I have quite a buffer for the next year and probably much longer. I was toying with the idea to use Seagate Ironwolf due to its attractive 6TB price, but noise level made me stick with WD Red. To minimize any potential issue affecting the whole batch, I actually bought disks at two different places (NewEgg and B&H).

A few curiosities were observed while pairing motherboard and case. First one is lack of opening for display port on the back. While slightly annoying, I found it bearable since HDMI opening, while grossly oversized, was accessible. If you really need display port you can order separate part number MCP-260-00068-0B but it will cost you $25.

Another one is mismatch between chassis power supply that has only 20-pin ATX connector and motherboard that requires 24-pin connector. As motherboard is really modest as far as power consumption goes, plugging 20-pin connector and leaving 4 leftmost pins empty works fine.

I also connected front-panel connector to PCI bracket in order to bring those ports to the outside. I only did it because I had bracket already available. Likewise, I swapped SATA cables that arrived with motherboard with shorter, round ones. This is just purely for cosmetics.

Without further ado, here is the itemized list:

Watching Sector Count

[2021-03-01: If you have disk larger than 8 TB, a slightly different rules apply.]

Ever since I first dealt with RAID, I was taught and was witness to ugly fact that two disks marked with the same size might not be exactly the same. Even the same disk from the same manufacturer could have slight difference in number of sectors. And some manufacturers were notorious for this (yes, Sun/Oracle, I am looking at you.

It was common wisdom to always make RAID a bit smaller than your full drive size to accommodate for potential “shrinkage” in the future. And I was myself known to warn others. However, things change…

Believe it or not, these days all disks have completely standardized sector count. For 512b sector size, formula is:

97,696,368 + (1,953,504 * (CapacityInGB - 50))

For 4K sectors, formula is:

12,212,046 + (244,188 * (CapacityInGB - 50))

This is courtesy of LBA Count for Disk Drives Standard (LBA1-03) that is seemingly followed by all manufacturers. I couldn’t find a single drive not following this standard and I’ve tried.

What does this mean? It means, if you have drive that is manufacturer in last five years, you can forget about under-sizing your RAID. Any replacement drive you order will have exactly the same sector count as long as sector size is same.

PS: Find below table for disks with 4K sectors:

PPS: While these rules are valid for SSDs too, depending on their configuration (e.g., over-provisioning) exact sector count available to end-user might vary.

[2017-07-16: Added calculator]

WRT Settings 1.20

Since I created WRT settings there weren’t too many versions. It is as simple as it gets - you have list of name value pairs, formats are well defined and that’s it.

However, Iv7777 found an issue in way how encryption works for AsusWRT v2 configuration format. He has a bit longer explanation in addition to his fix. Short version is that some random values are not working properly with Asus’ configuration file encryption. Leaving aside discussion on how useful this encryption is in the first place, I give you new version with this correction.

In addition to this major issue, there is a bugfix for handling empty DD-WRT files and program has been upgraded to use .NET Framework 4.

As always, new version is available either from application itself or from project’s page.