Mixing HDD and SSD in a ZFS Mirror

One of my test bad computers had a ZFS mirror between its internal 2.5" HDD (ST20000LM003) and external My Passport 2.5" USB 3.0 HDD (WDC WD20NMVW-11W68S0). And yes, having a mirror between SATA and USB is not the most ideal solution to start with, but it does work. In any case, setup happily chugged along until recently when the internal drive started having faults. Replacement was in order.

But replacing an old 2 TB drive proved not to be so easy. When it comes to 2 TB 2.5" models, all laptop drives manufactured these days are SMR. While you can use them in ZFS pool, performance is abysmal during resilvering. Normal use might be ok, depending on load, but it wouldn't be as good as CMR. The only way to get equivalent drive to the one I had was to get a refurbished years old drive. Not ideal.

But then my eyes went toward cheap 2 TB SSDs. For just a $10 more, I could get a (somewhat) faster drive. However, searching on Internet, I noticed that idea of mixing HDD and SSD in the same pool seems to be frowned upon.

And yes, I knew that you won't get full benefits of either HDD or SSD when using them together in the same pool but it seemed like an arbitrary limitation especially when price in 2 TB range is essentially equivalent. Why wouldn't you use SSD when drive needs replacing?

So I ordered myself a cheap SSD and tried to see if there are any downsides to mixed HDD/SSD setup.

The first test I did was an FIO sequential read/write (fio-seq-RW.fio). With two HDDs in mirror, I was at 148/99 MB/s for read/write, respectively. After changing the internal drive to SSD, speeds went to almost identical 147/98 MB/s. Adding a single SSD brought no practical difference in this scenario. Based on this test alone, I would have said that while SSD doesn't bring a performance improvement, it doesn't drag it down too much. Having an SMR drive in this setup would bring performance down more than this low-price SSD ever could.

The second test I tried was random read/write (fio-rand-RW.fio). Here speed with two HDD was 480/320 KB/s while combination of HDD and SSD brought speed all the way to 4980/3330 KB/s. Essentially ten-fold increase in performance. If you have virtual machines running on top of ZFS you will feel the difference.

The third test was just to verify if previous two tests looked sensible (ssd-test.fio). While numbers did differ slightly, overall data looked the same. No improvement when it comes to sequential access (even a slight performance decrease) but a huge improvements for random data access.

My conclusion is that, while replacing HDD with SSD might not be the most cost effective approach when it comes to larger pools, there is nothing bad about it as such and, depending on your workload, you might see a healthy improvement. It's not an appropriate solution when it comes to larger drives, but for pools having up to 2 TB drives, go for it!

PS: For curious, here is raw testing data.

PPS: No, I don't want to talk about who hurt me that much that I'm willing to use an external USB as part of a mirrored pool.