Medo's Home Page

UEFI Install for Root ZFS Ubuntu 18.10

There is a newer version of this guide for Ubuntu 19.04.

Booting ZFS Ubuntu of MBR is story I already told. But what if we want an encrypted UEFI ZFS setup?

Well, it’s quite simple to previous steps and again just a derivation on ZFS-on-Linux project.

As before, we first need to get into root prompt:

sudo -i

Followed by getting a few basic packages ready:

apt-add-repository universe
apt update
apt install --yes debootstrap gdisk zfs-initramfs

Disk setup is quite simple with only two partitions:

sgdisk --zap-all             /dev/disk/by-id/^^ata_disk^^

sgdisk -n2:1M:+511M -t2:EF00 /dev/disk/by-id/^^ata_disk^^
sgdisk -n1:0:0      -t1:8300 /dev/disk/by-id/^^ata_disk^^

sgdisk --print               /dev/disk/by-id/^^ata_disk^^
 Number  Start (sector)    End (sector)  Size       Code  Name
    1         1050624        67108830   31.5 GiB    8300
    2            2048         1050623   512.0 MiB   8300

I believe full disk encryption should be a no-brainer so of course we set up LUKS:

cryptsetup luksFormat -qc aes-xts-plain64 -s 512 -h sha256 /dev/disk/by-id/^^ata_disk^^-part1
cryptsetup luksOpen /dev/disk/by-id/^^ata_disk^^-part1 luks1

Creating ZFS stays the same as before:

zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 -O normalization=formD \
    -O xattr=sa -O mountpoint=none rpool /dev/mapper/luks1
zfs create -o canmount=noauto -o mountpoint=/mnt/rpool/ rpool/system
zfs mount rpool/system

Getting basic installation on our disks follows next:

debootstrap cosmic /mnt/rpool/
zfs set devices=off rpool
zfs list

And then we setup EFI boot partition:

mkdosfs -F 32 -n EFI /dev/disk/by-id/^^ata_disk^^-part2
mount /dev/disk/by-id/^^ata_disk^^-part2 /mnt/rpool/boot/

We need to ensure boot partition auto-mounts:

echo PARTUUID=$(blkid -s PARTUUID -o value /dev/disk/by-id/^^ata_disk^^-part2) /boot vfat noatime,nofail,x-systemd.device-timeout=5s 0 1 >> /mnt/rpool/etc/fstab
cat /mnt/rpool/etc/fstab

Before we start using anything, we should prepare a few necessary files:

cp /etc/hostname /mnt/rpool/etc/hostname
cp /etc/hosts /mnt/rpool/etc/hosts
cp /etc/netplan/*.yaml /mnt/rpool/etc/netplan/
sed '/cdrom/d' /etc/apt/sources.list > /mnt/rpool/etc/apt/sources.list

If you are dual-booting system with Windows, do consider turning off UTC BIOS time:

echo UTC=no >> /mnt/rpool/etc/default/rc5

With chroot we can get the first taste of our new system:

mount --rbind /dev  /mnt/rpool/dev
mount --rbind /proc /mnt/rpool/proc
mount --rbind /sys  /mnt/rpool/sys
chroot /mnt/rpool/ /bin/bash --login

Now we can update our software:

apt update

Immediately followed with locale and time zone setup:

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we install Linux image and basic ZFS boot packages:

apt install --yes --no-install-recommends linux-image-generic
apt install --yes zfs-initramfs

Since we’re dealing with encrypted data, our cryptsetup should be also auto mounted:

apt install --yes cryptsetup

echo "luks1 UUID=$(blkid -s UUID -o value /dev/disk/by-id/^^ata_disk^^-part1) none luks,discard,initramfs" >> /etc/crypttab
cat /etc/crypttab

Now we get grub started:

apt install --yes grub-efi-amd64

And update our boot environment again (seeing errors is nothing unusual):

update-initramfs -u -k all

And then we finalize our grup setup:

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=ubuntu --recheck --no-floppy

Finally we get the rest of desktop system:

apt-get install --yes ubuntu-desktop samba linux-headers-generic
apt dist-upgrade --yes

We can omit creation of the swap dataset but I always find it handy:

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput -o sync=always \
    -o primarycache=metadata -o secondarycache=none rpool/swap
mkswap -f /dev/zvol/rpool/swap
echo "/dev/zvol/rpool/swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

If one is so inclined, /home directory can get a separate dataset too:

rmdir /home
zfs create -o mountpoint=/home rpool/data

Only remaining thing before restart is to create user:

adduser ^^user^^
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sambashare,sudo ^^user^^
chown -R ^^user^^:^^user^^ /home/^^user^^

As install is ready, we can exit our chroot environment and reboot:

exit
reboot

You will get stuck after the password prompt as our mountpoint for system dataset is wrong. That’s easy to correct:

zfs set mountpoint=/ rpool/system
exit
reboot

Assuming nothing went wrong, your UEFI system is now ready.

PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 21.10 and 20.04

PPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, and 19.04.

Exporting JSON via Awk

I wanted to process Unicode CSV file to extract the first two columns into JSON. With awk it seemed easy enough:

awk '
    BEGIN {
        FS=";"
        print "["
    }
    {
        print "  { \"code\": \"" $1 "\", \"description\": \"" $2 "\" },"
    }
    END {
        print "]"
    }
    ' UnicodeData.txt | less

This will give you ALMOST parsable output. One thing that will spoil it is the last “hanging” comma making the whole JSON invalid (albeit some parsers will still load it). And no, there is no way to tell awk to do something special with the last line as processing of the lines is done one-by-one and thus there is no telling which line is last at any give moment.

What we can do is tell awk to process lines with a single line delay:

awk '
    BEGIN {
        FS=";"
        print "["
    }
    NR>1 {
        print "    { \"code\": \"" code "\", \"description\": \"" description "\" },"
    }
    {
        code = $1
        description = $2
    }
    END {
        print "    { \"code\": \"" code "\", \"description\": \"" description "\" }"
        print "]"
    }
    ' UnicodeData.txt | less

This prints content starting from the second line (NR>1) and what we do in the main loop is just storing fields into our variables that’ll be read in the next iteration. Essentially what we have is a single line delay mechanism. To catch up with the last line we just print it out without trailing comma in END portion of our program.

Valid JSON at last.

IniEd

You can do wonders with sed and awk when it comes to editing config files. However, if format is not line but section based, editing it becomes exercise in writing full blown programs and even a simple reading has potential to turn into a mess of regex nobody will understand in a year. It was on a day such as that I decided to make myself a command line INI file editor.

First, the most common action, reading value is as simple as giving section and key name:

inied --section mysqld --key key_buffer --print  examples/my.cnf

Modifying a value is as easy:

inied --section mysqld --key key_buffer --edit 200M  examples/my.cnf

From there on you can go crazy with pretty-printing, filtering, or even just piping it into any other tool.

You can install utility from Debian package or grab the source code on GitHub and compile it using make yourself.

Common Code for Files and Pipes in Rust

To read a file in Rust, the following code is easy enough to figure:

fn open(file_name: &str) -> Result<(), Error> {
    let input = File::open(file_name)?;
    let mut reader = io::BufReader::new(input);
    …

What if you want to read from standard input when file is not given? In many other languages you might solve this using interface. However, in Rust we haven’t got interface support. What we do have are traits.

fn parse(file_name: Option<&str>) -> Result<(), Error> {
    let input = match file_name {
        Some(file_name) => Box::new(File::open(file_name)?) as Box<Read>,
        None => Box::new(io::stdin()) as Box<Read>,
    };
    let mut reader = io::BufReader::new(input);
    …

As you can see, it reads almost exactly like an interface would, the only curiosity being explicit boxing you have to do.

To see the code in context, you can check IniEd and it’s implementation.

UEFI Install for Root ZFS Ubuntu 18.10

Exporting JSON via Awk

IniEd

Common Code for Files and Pipes in Rust

Post-Quantum Cryptography – Round Two