There is a newer version of this guide for Ubuntu 21.10.
Technically, I already have a guide for encrypted ZFS setup on Ubuntu 20.04. However, that guide used Geli and, as correctly one reader noted in comments (thanks Alex!), there was no reason not to use ZFS' native encryption. So, here is adjusted variant of my setup.
First of all, Ubuntu 20.04 has a ZFS setup option as of 19.10. You should use it instead of the manual installation procedure unless you need something special. Namely, manual installation allows for encryption, in addition to the custom pool layout and naming. You should also check the great Root on ZFS installation guide that's part of ZFS-on-Linux project for a full picture. I find its final ZFS layout a bit too complicated for my taste but there is a lot of interesting tidbits on that page. Here is my somewhat simplified version of the same, intended for a singe disk installation.
After booting into Ubuntu desktop installation we want to get a root prompt. All further commands are going to need root credentials anyhow.
Terminalsudo -i
The very first step should be setting up a few variables - disk, pool, host name, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.
TerminalDISK=/dev/disk/by-id/ata_disk
POOL=ubuntu
HOST=desktop
USER=user
General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are planning to have multiple kernels, increasing boot partition size might be a good idea.
Terminalblkdiscard $DISK
sgdisk --zap-all $DISK
sgdisk -n1:1M:+127M -t1:EF00 -c1:EFI $DISK
sgdisk -n2:0:+512M -t2:8300 -c2:Boot $DISK
sgdisk -n3:0:0 -t3:8309 -c3:Ubuntu $DISK
sgdisk --print $DISK
Finally we're ready to create system ZFS pool. Note that you need to encrypt it at the moment it's created.
Terminalzpool create -o ashift=12 -o autotrim=on \
-O compression=lz4 -O normalization=formD \
-O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
-O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \
-O canmount=off -O mountpoint=none -R /mnt/install $POOL $DISK-part3
On top of this encrypted pool, we can create our root dataset.
Terminalzfs create -o canmount=noauto -o mountpoint=/ $POOL/root
zfs mount $POOL/root
Assuming UEFI boot, two additional partitions are needed. One for EFI and one for booting. Unlike what you get with the official guide, here I don't have ZFS pool for boot partition but a plain old ext4. I find potential fixup works better that way and there is a better boot compatibility. If you are thinking about mirroring, making it bigger and ZFS might be a good idea. For a single disk, ext4 will do.
Terminalyes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/
mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi
To start the fun we need debootstrap package.
Terminalapt install --yes debootstrap
Bootstrapping Ubuntu on the newly created pool is next. This will take a while.
Terminaldebootstrap focal /mnt/install/
zfs set devices=off $POOL
Our newly copied system is lacking a few files and we should make sure they exist before proceeding.
Terminalecho $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/
If you are installing via WiFi, you might as well copy your wireless credentials. Don't worry if this returns errors - that just means you are not using wireless.
Terminalmkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/
Finally we're ready to "chroot" into our new system.
Terminalmount --rbind /dev /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys /mnt/install/sys
chroot /mnt/install \
/usr/bin/env DISK=$DISK POOL=$POOL USER=$USER \
bash --login
Let's not forget to setup locale and time zone.
Terminallocale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales
dpkg-reconfigure tzdata
Now we're ready to onboard the latest Linux image.
Terminalapt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic
Followed by boot environment packages.
Terminalapt install --yes zfs-initramfs grub-efi-amd64-signed shim-signed tasksel
To mount boot and EFI partition, we need to do some fstab setup.
Terminalecho "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
/boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
/boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab
Now we get grub started and update our boot environment.
TerminalKERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL
Grub update is what makes EFI tick.
Terminalupdate-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
--recheck --no-floppy
Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it'll take a considerable amount of time.
Terminaltasksel install ubuntu-desktop-minimal
Short package upgrade will not hurt.
Terminalapt dist-upgrade --yes
We can omit creation of the swap dataset but I personally find a small one handy.
Terminalzfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
-o sync=always -o primarycache=metadata -o secondarycache=none $POOL/swap
mkswap -f /dev/zvol/$POOL/swap
echo "/dev/zvol/$POOL/swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume
If one is so inclined, /home directory can get a separate dataset too.
Terminalrmdir /home
zfs create -o mountpoint=/home $POOL/home
The only remaining task before restart is to create the user, assign a few extra groups to it, and make sure its home has correct owner.
Terminaladduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
passwd $USER
As install is ready, we can exit our chroot environment.
Terminalexit
And cleanup our mount points.
Terminalumount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a
After the reboot you should be able to enjoy your installation.
Terminalreboot
PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 22.04 and 20.04
PPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, 19.04, and 18.10.
[2020-06-27: Added blkdiscard and autotrim.]
This looks great and i will use native zfs encryption for my new setup on a laptop, where i intend to use hibernate as well, therefore i need a swap partition i might have otherwise omitted. I read on the original root on zfs article that zfs deadlocks might occur if you put swap on a zfs dataset. Is this something you took in mind when writing this article? Did you ever experience issues with it?
I was never able to setup properly encrypted swap partition with hibernation. :(
Hibernation does work if you just create a separate unencrypted swap partition but that does come at the cost of security.
If you want to use hibernation on your laptop, you will probably need to increase the default swap partition created by the Ubiquity installer. Since this swap partition is not a ZFS partition, you won’t face the deadlock bug.
In order to increase the swap partition (which max 2GiB during the Ubuntu installation), you will need to modify the zsys-setup script, as explained in this discourse thread: https://discourse.ubuntu.com/t/zfs-focus-on-ubuntu-20-04-lts-blog-posts/16355/71. It is easy to do but you will need to access the terminal during the Ubuntu installation, just before the partitioning page.
How is the performance on zfs? Do you feel the speed drawbacks in everyday usage?
“Your storage drive’s max throughput will reduce. For example when I tested a Samsung 970 EVO formatted to ext4, I’d get ~260MB/s when doing 4KiB, Queue depth 1 writes. That went down to ~60MB/s with ZFS!
from https://medium.com/@ruvi.d/zfs-on-ubuntu-20-04-lts-53728f3bc9e9
I actually haven’t found it that bad. On rotational drive I do not even see the difference.
However, on NVMe drive it’s noticeable. My experience is that speed is in 100-150 MB/s range most of the time. I still find disk access snappy are it remains low latency but I do see difference whenever I get to copy big files around.
For me that’s acceptable as I don’t copy big files a lot of times. It all depends on your use case…
PS: For synthetic test, you can check here.
Hi Josip,
I can confirm that switching on Corsair MP600 works too and flawless as on Corsair MP510.
What I have done:
“`
sudo su
nvme –list
“`
output:
“`
Node Model Format
/dev/nvme0n1 Samsung SSD 970 PRO 512GB 512 B + 0 B
/dev/nvme1n1 Force MP600 512 B + 0 B
“`
asked for LBA options with:
“`
nvme id-ns /dev/nvme1n1 -H | grep LBA
“`
output:
“`
LBA Format 0 : Metadata Size: 0 bytes – Data Size: 512 bytes – Relative Performance: 0x2 Good (in use)
LBA Format 1 : Metadata Size: 0 bytes – Data Size: 4096 bytes – Relative Performance: 0x1 Better
“`
format it with:
“`
nvme format /dev/nvme1n1 -l 1
“`
output:
“`
Success formatting namespace:1
““
check LBA with:
“`
nvme id-ns /dev/nvme1n1 -H | grep LBA
“`
output:
“`
LBA Format 0 : Metadata Size: 0 bytes – Data Size: 512 bytes – Relative Performance: 0x2 Good
LBA Format 1 : Metadata Size: 0 bytes – Data Size: 4096 bytes – Relative Performance: 0x1 Better (in use)
“`
and recheck with:
“`
nvme –list
“`
output:
“`
Node Model Format
/dev/nvme0n1 Samsung SSD 970 PRO 512GB 512 B + 0 B
/dev/nvme1n1 Force MP600 4 KiB + 0 B
“`
Now, if I discover how to modify the LBA on WD4002FFWX… I will be the happiest man
I have a Samsung 970 PRO, originally is configured “physical: 512” (P) and “logical: 4096” (L), they call it “512e”.
You can detect (as root) with: `nvme id-ns /dev/nvme0n1 -H | grep LBA` , if you reset the “drive” you get P: 512 and L: 512 and run faster because of no “overhead”.
I hear/read from Corsair MP510 showing format-level “0, 1, 2, 3,” respectively:
“512 , no Metadata” = Level 0.
“512, with Metadata” = Level 1.
“4096, no metadata” = Level 2 (best, not only for ZFS).
“4096, with metadata” = Level 3.
That’s the response of `nvme-cli` after formatting
“LBA Format 2 : Metadata Size: 0 bytes – Data Size: 4096 bytes – Relative Performance: 0 Best (in use)”
and this the command for formatting:
`nvme format /dev/nvme1n1 -l 2`
I just bought one “Corsair MP600” and will test it
>If you are thinking about mirroring, making it bigger and ZFS might be a good idea
How would one translate the steps to use ZFS intead?
>yes | mkfs.ext4 $DISK-part2
>mkdir /mnt/install/boot
>mount $DISK-part2 /mnt/install/boot/
>mkfs.msdos -F 32 -n EFI $DISK-part1
>mkdir /mnt/install/boot/efi
>mount $DISK-part1 /mnt/install/boot/efi
Tried following the steps in this guide, but having an EFI boot issue. The machine used to have FreeBSD. No matter what I pick from the EFI bootlist, it always seems to try and install Freebsd. It is even a new set of disks from what I had before.
Any suggestions?
Hi Francisco,
I know, between FreeBSD and “FreeBSOD’ is no much difference (joke).
Even under old BIOS had I some difficulties with FreeBSD and or FreeNAS by turning back to Linux.
If you have UEFI you should first access motherboard-EFI and delete (if you don’t need anymore) the BDS-entry, if you cannot do it… move to latest line down.
In Linux is a program called “Efibootmanager” and you can install it with:
sudo apt-get install efibootmgr
This Wiki https://wiki.ubuntuusers.de/efibootmgr/ is in German but the listed commands are in English.
Use `sudo efibootmgr` or `sudo efibootmgr –verbose` to see what’s inside UEFI , after them use:
sudo efibootmgr -A xxxx #for xxxx set the bootnumber of BSD
set a new bootorder with:
sudo efibootmgr -o 0005,0003
you can test a bootfile/entry with:
sudo efibootmgr –test FILENAMEHERE
I hope it’s helpful
Great job! first, than I miss the detection of “/dev/disk/by-id/ata_disk” through `ls -la /dev/disk/by-id` and the choose of:
– take the output beginning with: `nvme-eui.xxxxxxxxxxxxxxxx` for “NVM-e”
– my output (e.g.): `nvme-eui.0123456789abcdef`
– take the output beginning with: `wwn-0xxxxxxxxxxxxxxxxx` for “SATA”
– my output (e.g.): `wwn-0x0123456789abcdef`
Contemplate your instruction this (https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS.html#boot-grub-not-mounted)?
What does mean `sed “s/ubuntu/$HOST/” /etc/hosts > /mnt/install/etc/hosts` exactly? I want call my `HOST=Zub-20-04`, remain this line like you wrote?
Can I add `–no-install-reccomends` to `tasksel install ubuntu-desktop`? or should i follow your method (ubuntu-desktop-minimal) and install it later with `apt`?
Can I add `kde-plasma-desktop` and `kde-full` with `tasksel` ? or I should add it later on?
Can I change the “–bootloader-id” here
`grub-install –target=x86_64-efi –efi-directory=/boot/efi –bootloader-id=Ubuntu \
–recheck –no-floppy`
with `–bootloader-id=Zubuntu`?
Thanks for your help.
If you used variables above, e.g. you have set
HOST=Zub-20-04You can use that line verbatim – it will replace host itself.
That should works just fine. Be careful with
–no-install-recommendsthough, as it sometimes wrecks havoc with GUI stuff.There is a `install.py` or `python.py` in which is an option:
“`
install recommends=true
“`
edit with normal editor and set it to `false`
Can you tell me or post a thread for installing Kubuntu on encrypted zfs-root?
Which program have I to install before start installing Kubuntu?
Must I or can I insert `echo MYPASSWORD | zpool create -f \` and than the remain part?
What’s better `-O relatime=on \` or `-O atime=off \`?
Not sure about Kubuntu but I assume they should be close if not equal to Ubuntu.
or depends on your use case. I personally ignore access time (thus atime=off) altogether.
What’s about
“`
-O sync=disabled \ *
-O recordsize=1M \
“`
by zpool create? are these options let consuming to much memory (20 GB)?
or how can I reduce memory consumption?
is default so there is no need to specify it.
depends on your setup. For me default of 128K worked ok.
Hi, I followed everything exactly, but when I tried to run `update-grub`, I got this:
root@ubuntu:/# update-grub
Sourcing file `/etc/default/grub’
Sourcing file `/etc/default/grub.d/init-select.cfg’
Generating grub configuration file …
cannot open ‘bpool/BOOT/root’: dataset does not exist
Found linux image: vmlinuz-5.4.0-58-generic in ubuntu/root
Found initrd image: initrd.img-5.4.0-58-generic in ubuntu/root
device-mapper: reload ioctl on osprober-linux-sda3 failed: Device or resource busy
Command failed.
Adding boot menu entry for UEFI Firmware Settings
done
Any ideas?
Could it be that your pool is named
bpool/BOOT? becausePOOLvariable is intended to be just a root pool name, i.e.POOL=bpool.Hello Josip,
thank you very much for your article.
Do you know any further progress with hibernate?
I found this. Maybe you could give this a try and extend your manual (https://lab.neuronfarm.net/nomad/wiki-system/wiki/right-fit-swap-for-laptop-hibernation-Ubuntu-20.04-full-ZFS-install).
IHPH
IHPH;SELECT/**/PG_SLEEP(15)–