Ubuntu Server 20.04 on UEFI ZFS Without Encryption

Illustration

With Ubuntu 20.04 Desktop there is a (still experimental) ZFS setup option in the addition to long time manual ZFS installation option. For Ubuntu Server we’re still dependent on the manual steps.

Steps here follow my 19.10 server guide but without the encryption steps. While I normally love having encryption enabled, there are situations where it gets in the way. Most notable example is a machine which you cannot access remotely to enter encryption key.

To start with installation we need to get to the root prompt. Just find Enter Shell behind Help menu item (Shift+Tab comes in handy) and you’re there.

The very first step is setting up a few variables - disk, pool, host name, and user name. This way we can use them going forward and avoid accidental mistakes. Make sure to replace these values with the ones appropriate for your system. It’s a good idea to use something unique for the pool name (e.g., host name). I personally also like having pool name start with uppercase but there is no real rule here.

DISK=/dev/disk/by-id/^^ata_disk^^
POOL=^^Ubuntu^^
HOST=^^server^^
USER=^^user^^

To start the fun we need debootstrap and zfsutils-linux package. Unlike desktop installation, ZFS package is not installed by default.

apt install --yes debootstrap zfsutils-linux

General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are planning to have multiple kernels, increasing boot partition size might be a good idea. Major change as compared to my previous guide is partition numbering. While having partition layout different than partition order had its advantages, a lot of partition editing tools would simply “correct” the partition order to match layout and thus cause issues down the road.

blkdiscard $DISK

sgdisk --zap-all                        $DISK

sgdisk -n1:1M:+127M -t1:EF00 -c1:EFI    $DISK
sgdisk -n2:0:+512M  -t2:8300 -c2:Boot   $DISK
sgdisk -n3:0:0      -t3:8309 -c3:Ubuntu $DISK

sgdisk --print                          $DISK

Now we’re ready to create system ZFS pool.

zpool create -o ashift=12 -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O canmount=off -O mountpoint=none -R /mnt/install $POOL $DISK-part3
zfs create -o canmount=noauto -o mountpoint=/ $POOL/root
zfs mount $POOL/root

Assuming UEFI boot, two additional partitions are needed. One for EFI and one for booting. Unlike what you get with the official guide, here I don’t have ZFS pool for boot partition but a plain old ext4. I find potential fixup works better that way and there is a better boot compatibility. If you are thinking about mirroring, making it bigger and ZFS might be a good idea. For a single disk, ext4 will do.

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

Bootstrapping Ubuntu on the newly created pool is next. As we’re dealing with server you can consider using --variant=minbase rather than the full Debian system. I personally don’t see much value in that as other packages get installed as dependencies anyhow. In any case, this will take a while.

debootstrap focal /mnt/install/

zfs set devices=off $POOL

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

echo $HOST > /mnt/install/etc/hostname
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install /usr/bin/env DISK=$DISK POOL=$POOL USER=$USER bash --login

Let’s not forget to setup locale and time zone. If you opted for minbase you can either skip this step or manually install locales and tzdata packages.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes zfs-initramfs grub-efi-amd64-signed shim-signed

To mount EFI and boot partitions, we need to do some fstab setup too:

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=1 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=1 0 1" >> /etc/fstab
cat /etc/fstab

Now we get grub started and update our boot environment. Due to Ubuntu 19.10 having some kernel version kerfuffle, we need to manually create initramfs image. As before, boot cryptsetup discovery errors during mkinitramfs and update-initramfs as OK.

KERNEL=`ls /usr/lib/modules/ | cut -d/ -f1 | sed 's/linux-image-//'`
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Since we’re dealing with computer that will most probably be used without screen, it makes sense to install OpenSSH Server.

apt install --yes openssh-server

I also prefer to allow remote root login. Yes, you can create a sudo user and have root unreachable but that’s just swapping one security issue for another. Root user secured with key is plenty safe.

sed -i '/^#PermitRootLogin/s/^.//' /etc/ssh/sshd_config
mkdir /root/.ssh
echo "^^<mykey>^^" >> /root/.ssh/authorized_keys
chmod 644 /root/.ssh/authorized_keys

If you’re willing to deal with passwords, you can allow them too by changing both PasswordAuthentication and PermitRootLogin parameter. I personally don’t do this.

sed -i '/^#PasswordAuthentication yes/s/^.//' /etc/ssh/sshd_config
sed -i '/^#PermitRootLogin/s/^.//' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
passwd

Short package upgrade will not hurt.

apt dist-upgrade --yes

We can omit creation of the swap dataset but I personally find its good to have it just in case.

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
    -o sync=always -o primarycache=metadata -o secondarycache=none $POOL/swap
mkswap -f /dev/zvol/$POOL/swap
echo "/dev/zvol/$POOL/swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

If one is so inclined, /home directory can get a separate dataset too.

rmdir /home
zfs create -o mountpoint=/home $POOL/home

And now we create the user and assign a few extra groups to it.

adduser --disabled-password --gecos '' $USER
asermod -a -G adm,cdrom,dip,plugdev,sudo $USER
chown -R $USER:$USER /home/$USER
passwd $USER

Consider enabling firewall. While you can go wild with firewall rules, I like to keep them simple to start with. All outgoing traffic is allowed while incoming traffic is limited to new SSH connections and responses to the already established ones.

apt install --yes man iptables iptables-persistent

iptables -F
iptables -X
iptables -Z
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -p ipv6-icmp -j ACCEPT

netfilter-persistent save

As install is ready, we can exit our chroot environment.

exit

And cleanup our mount points.

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

After the reboot you should be able to enjoy your installation.

reboot

XigmaNAS as a SysLog Server

As it is the machine with the most redundancy, there are many tasks that fall onto my XigmaNAS server. One of those tasks is serving as a syslog destination for other devices.

However, you cannot get that behavior by default as its syslog server will start in secure mode ([-ss](https://www.freebsd.org/cgi/man.cgi?query=syslogd&sektion=8)) without opening any socket. And there is no easy GUI way to make it accept syslog messages.

But here comes the power of postinit actions. Just add the following in SystemAdvancedCommand scripts as the new PostInit entry:

sed -i -e 's^ -ss^ -a 192.168.0.0/16^' /etc/rc.d/syslogd ; /etc/rc.d/syslogd restart

Your XigmaNAS server will now accept all messages coming from your local network.

PS: To troubleshoot settings, you can do the following on the command line:

$ kenv rc.debug=1
$ /etc/rc.d/syslogd restart

Using Mikrotik's Router to Detect Power Outage

Before I had CyberCard, I still had a need to monitor if my system was running off the UPS power. If my server could detect power out and shut down other devices, my battery life would keep server up for longer.

If you have Mikrotik’s router with two power supplies and an SSH connection to the same there is a trick you can use - Mikrotik can show you each power supply state. If you take care to plug one power supply into the UPS and the other one into the non-UPS outlet, you suddenly have a detector.

ssh -i ~/.ssh/id_rsa admin@router.home "/system health print"
            voltage: 23.6V
            current: 426mA
        temperature: 50C
  power-consumption: 10W
       psu1-voltage: 24.4V
       psu2-voltage: 0V

Even better, the voltage doesn’t go immediately to 0 V as soon as power is out so there is a delay built-in. So, script is as easy as detecting 0V on the output. Something like this.

ssh -i ~/.ssh/id_rsa admin@^^router.home^^ "/system health print" \
  | egrep 'psu[12]-voltage' | grep -q '0V' && echo "Do Something!"

PS: If you’re interested in the whole script around this, you can download it here.

Sendmail via GMail on Ubuntu Server

I finally decided to migrate my Wordpress onto Ubuntu 20.04 only to discover e-mail I configured via Smtpmail stopped working. That meant my WordPress and various PHP and command line tools couldn’t use Google’s e-mail relay to deliver e-mails to my Inbox. It was time to setup my server to use Google’s SMTP again. And this time I decided to go with postfix.

Installing postfix usually involves a GUI asking a few questions. However, you can use debconf-set-selections to preload the answers. Make sure to be the root used (sudo su -).

debconf-set-selections &lt;&lt;&lt; "postfix postfix/main_mailer_type string 'Internet Site'"
debconf-set-selections &lt;&lt;&lt; "postfix postfix/mailname string ''"
apt-get install --assume-yes postfix libsasl2-modules

Once installed, we need to provide credentials in /etc/postfix/sasl/sasl_passwd.

unset HISTFILE
echo "[smtp.gmail.com]:587 ^^relay@gmail.com^^:^^password^^" > /etc/postfix/sasl/sasl_passwd
postmap /etc/postfix/sasl/sasl_passwd
chmod 0600 /etc/postfix/sasl/sasl_passwd /etc/postfix/sasl/sasl_passwd.db

Finally we need to update /etc/postfix/main.cf for authentication options.

sed -i 's/relayhost = /relayhost = [smtp.gmail.com]:587/' /etc/postfix/main.cf
cat <<EOF >> /etc/postfix/main.cf
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd
smtp_sasl_security_options = noanonymous
EOF

And that’s pretty much it. The only remaining thing is to restart postfix:

systemctl restart postfix

To test if it works, just use sendmail.

echo "Subject: Test via sendmail" | sendmail -v ^^youremail@example.com^^

[2022-06-16: As of 2022-06-01, it’s not possible to use your Google email and password directly. However, you can still follow this guide and use App Password instead.]

The Cost of CyberCard

After publishing text about the CyberCard project I got the question from a friend. Wasn’t it cheaper to buy Jeff Mayes’ interface driver then to build my own?

Answer is yes - at $30 that board is cheap. But that’s not all. Even the original RMCARD205 at $150 is cheaper than what I spent.

First of all, there were 4 revisions. The first revision was a bit too large. Manually filing PCB did the trick for the troubleshooting but I wanted to have revision B with the correct width. While width was now correct, I accidentally shortened it a bit. And yes, this brought me to the third revision. For that revision I also changed MCP2221A to SOIC package. It wasn’t strictly necessary but I figured having all three ICs in SOIC looked nicer than having different package styles on the same board. The last revision D was just a bit more fiddling with design without any major change. Yes, there were some other changes but this was a gist of it.

Considering each revision was around $25 in PCB cost (OSHPark) and I spent about $50 in parts for them, project was more expensive than official RMCARD205 even without accounting for my time. Since the first version was actually working, you can view all the time and money spent afterward as wasted.

But I disagree. From the moment I started working on it I knew it would end more expensive than the original part. Even for the first board I spent more money in PCB and parts than what Jeff’s adapter would cost with shipping. I found this board to be the perfect project: it would result in something useful, it was simple enough that I could work with it whenever I had some spare time, cheap enough that it wouldn’t break the bank, and an excellent chance to setup PIC16F1454 as an USB device.

I was eyeing PIC16F1454 for a few years now (I still have sample from Microchip from when it was originally announced) but I never got around to. When I first started with the board design I noticed MCP2221A USB-to-serial bridge was compatible with 16F1454’s footprint. If I was a betting man, I would have said that MCP2221A was nothing other than PIC16F1454 with the custom code. This project gave me a reason to get into this interesting PIC and do some USB programming.

I actually paid not for the final board - no matter how well it works. I paid a good money to keep me entertained and to fill my free time. And it was worth every penny.