Moving Blog from CentOS 7 to Ubuntu 19.04

With WordPress 5.2 I decided to leave my CentOS 7 installation as it required PHP 5.6.20 or higher and my CentOS only had 5.4.16. Yes, I could have manually upgraded PHP from alternative CentOS repositories but I decided not to. While CentOS has been stable for me and I had pretty much no issues with it, that stability came at the cost of slow or non-existent new features. Biggest example being stuck in pre-HTTP2 times.

It took me a while to decide which distribution to use as a replacement, the final candidates being Debian and Ubuntu. While I was really tempted to go with Debian due to its reputation, I decided to go with Ubuntu at the end. It was due to frequency of updates more than anything else. After sticking with CentOS for so long, I wanted something more dynamic.

Most of my setup scripts needed just a simple replacement of yum with apt-get, the only major hurdle presented switching from CentOS Apache (httpd) to Ubuntu’s Apache (apache2). Despite both being 2.4, differences were significant enough that I couldn’t just move my config files. So I went with true and tested way of gradually modifying my config file until I could actually start Apache.

The first hurdle was how to install it. My CentOS installation used httpd mod_ssl php php-mysql, and php-gd packages. The equivalent Ubuntu setup consisted of apache2 php php-mysql php-gd. Essentially only the SSL package was missing from list and only because Apache2 already contained it.

The first error appeared with User and Group setting. While CentOS uses apache, the Ubuntu needs www-data. Remember that you need to change this not only in config file but that a chown or two might be needed.

The biggest issue were includes. On CentOS I used the following collections:

Include conf.modules.d/00-base.conf
Include conf.modules.d/00-mpm.conf
Include conf.modules.d/00-ssl.conf
Include conf.modules.d/00-systemd.conf
Include conf.modules.d/10-php.conf

On Ubuntu there is no equivalent. One has to either use wildcards to load all or list each include separately. Since list of enabled modules is rather conservative, I decided to use wildcards:

Include mods-enabled/*.load
Include mods-enabled/*.conf
Include conf-enabled/*.conf

Due to aforementioned conservative list of enabled modules, I had to enable a few more:

a2enmod autoindex
a2enmod expires
a2enmod headers
a2enmod rewrite
a2enmod ssl
a2enmod socache_shmcb
a2enmod php7.2

And yes, this is not strictly speaking equivalent of CentOS configuration but it’s close enough for my WordPress installation. Depending on your exact setup, you might need to enable a few more (or choose to skip some).

The last change I had to do was for firewall. As Ubuntu uses ufw, the following commands were needed to allow traffic through:

ufw allow 'Apache'
ufw allow 'Apache Secure'
sudo ufw reload

And now my WordPress runs on Ubuntu. :)

Remembering the Last Grub OS

If you are dual booting Linux and Windows, you might want to have Grub automatically select the last booted OS instead of just going with the default. The whole trick rests on adding just two lines to /etc/default/grub:

GRUB\_DEFAULT=saved GRUB\_SAVEDEFAULT=true

Of course that has to be followed by update-grub:

sudo update-grub

Now Grub will track which OS you booted last and preselect that menu entry.

PS: If you have custom menu entries, do not forget that you need to have savedefault option present.

Adding UEFI Windows 10 Menu Entry to Grub

If you install Ubuntu first and Windows later, you’ll notice that it’s not possible to boot into Linux anymore. As Windows boot loader doesn’t really handle Linux, you’ll need to tell Windows to use Grub.

Once you’re in command prompt with administrative privileges, you can execute:

BCDEDIT /set {bootmgr} path \EFI\Ubuntu\grubx64.efi

After reboot Grub will show it’s ugly face and you’ll have another problem - there are no Windows entries.

To get them into Grub menu, one can simply update grub:

sudo update-grub

On most Linux distributions this will trigger OS Prober and Windows Boot Manager entry will magically appear. However, if you have OS Prober disabled or you want to disable it in future for some reason, you can add manual entry too:

cat << EOF | sudo tee /etc/grub.d/25_windows
#!/bin/sh
exec tail -n +3 \$0
menuentry 'Windows 10' {
  savedefault
  search --no-floppy --set=root --file /EFI/Microsoft/Boot/bootmgfw.efi
  chainloader (\${root})/EFI/Microsoft/Boot/bootmgfw.efi
}
EOF

sudo chmod +x /etc/grub.d/25_windows

sudo update-grub

In either case, boot menu should now offer you option to get into Windows.

Symbolic Links on VirtualBox Share

If you are running VirtualBox with Linux guest under a Linux host, you will definitely need symbolic links at one point in time. And, while you can do links toward shared folders, you cannot create a link on shared folder itself. You will get something like:

ln -s /something /somethingelse
 ln: failed to create symbolic link '/somethingElse': Read-only file system

Well, fortunately it’s easy to enable connection on per share basis. Just run the following command on your host (replacing GuestName and ShareName with correct information):

VBoxManage setextradata '^^GuestName^^' VBoxInternal2/SharedFoldersEnableSymlinksCreate/^^ShareName^^ 1

If your machine is running, you will need to fully shut it down (no, restart doesn’t help) and you get to enjoy the full symbolic link glory.

Installing UEFI ZFS Root on Ubuntu 19.04

There is a newer version of this guide for Ubuntu 19.10.

As rumors of Ubuntu 19.04 including ZFS installer proved to be a bit premature, I guess it’s time for a slight adjustment to my previous ZFS instructions.

Again, all this is just a derivation on ZFS-on-Linux project’s instruction for older version.

As before, we first need to get into root prompt:

sudo -i

Followed by getting a few basic packages ready:

apt-add-repository universe
apt update
apt install --yes debootstrap gdisk zfs-initramfs

Disk setup is quite simple with only two partitions:

sgdisk --zap-all                      /dev/disk/by-id/^^ata_disk^^

sgdisk -n3:1M:+511M -t3:8300 -c3:Boot /dev/disk/by-id/^^ata_disk^^
sgdisk -n2:0:+128M  -t2:EF00 -c2:EFI  /dev/disk/by-id/^^ata_disk^^
sgdisk -n1:0:0      -t1:8300 -c1:Data /dev/disk/by-id/^^ata_disk^^

sgdisk --print                        /dev/disk/by-id/^^ata_disk^^

I believe full disk encryption should be a no-brainer so of course we set up LUKS:

cryptsetup luksFormat -q --cipher aes-xts-plain64 --key-size 512 \
    --pbkdf pbkdf2 --hash sha256 /dev/disk/by-id/^^ata_disk^^-part1
cryptsetup luksOpen /dev/disk/by-id/^^ata_disk^^-part1 system

Creating ZFS stays the same as before:

zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 \
      -O normalization=formD -O xattr=sa -O mountpoint=none system /dev/mapper/system
zfs create -o canmount=noauto -o mountpoint=/mnt/system/ system/root
zfs mount system/root

Getting basic installation on our disks follows next:

debootstrap disco /mnt/system/
zfs set devices=off system
zfs list

And then we setup EFI boot partition:

yes | mkfs.ext4 /dev/disk/by-id/^^ata_disk^^-part3
mount /dev/disk/by-id/^^ata_disk^^-part3 /mnt/system/boot/

mkdir /mnt/system/boot/efi
mkfs.msdos -F 32 -n EFI /dev/disk/by-id/^^ata_disk^^-part2
mount /dev/disk/by-id/^^ata_disk^^-part2 /mnt/system/boot/efi

We need to ensure boot partition auto-mounts:

echo PARTUUID=$(blkid -s PARTUUID -o value /dev/disk/by-id/^^ata_disk^^-part3) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1 >> /mnt/system/etc/fstab
echo PARTUUID=$(blkid -s PARTUUID -o value /dev/disk/by-id/^^ata_disk^^-part2) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1 >> /mnt/system/etc/fstab
cat /mnt/system/etc/fstab

Before we start using anything, we should prepare a few necessary files:

echo "^^hostname^^" > /mnt/system/etc/hostname
sed 's/ubuntu/^^hostname^^/' /etc/hosts > /mnt/system/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/system/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/system/etc/netplan/

If you are installing via WiFi, you might as well copy your credentials:

mkdir -p /mnt/system/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/system/etc/NetworkManager/system-connections/

With chroot we can get the first taste of our new system:

mount --rbind --make-rslave /dev  /mnt/system/dev
mount --rbind --make-rslave /proc /mnt/system/proc
mount --rbind --make-rslave /sys  /mnt/system/sys
chroot /mnt/system/ /bin/bash --login

Now we can update our software:

apt update

Immediately followed with locale and time zone setup:

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we install Linux image and basic ZFS boot packages:

apt install --yes --no-install-recommends linux-image-generic
apt install --yes zfs-initramfs

Since we’re dealing with encrypted data, our cryptsetup should be also auto mounted:

apt install --yes cryptsetup keyutils

echo "system UUID=$(blkid -s UUID -o value /dev/disk/by-id/^^ata_disk^^-part1) \
    none luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab

cat /etc/crypttab

Now we get grub started:

apt install --yes grub-efi-amd64

And update our boot environment again (seeing errors is nothing unusual):

update-initramfs -u -k all

And then we finalize our grup setup:

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
    --bootloader-id=Ubuntu --recheck --no-floppy

Finally we get the rest of desktop system:

apt-get install --yes ubuntu-desktop samba linux-headers-generic
apt dist-upgrade --yes

We can omit creation of the swap dataset but I always find it handy:

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
    -o sync=always -o primarycache=metadata -o secondarycache=none system/swap
mkswap -f /dev/zvol/system/swap
echo "/dev/zvol/system/swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

If one is so inclined, /home directory can get a separate dataset too:

rmdir /home
zfs create -o mountpoint=/home system/home

Only remaining thing before restart is to create user:

adduser ^^user^^
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sambashare,sudo ^^user^^
chown -R ^^user^^:^^user^^ /home/^^user^^

As install is ready, we can exit our chroot environment and reboot unmount our new environment. If unmount fails, just repeat it until it doesn’t. :)

exit
umount -R /mnt/system

Finally we can correct root’s mount point and reboot:

zfs set mountpoint=/ system/root
reboot

Assuming nothing went wrong, your UEFI system is now ready.

[2019-10-27: Added --make-rslave]

PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 21.10 and 20.04

PPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, and 18.10.

Appending to Certificate Bundle

When one needs to add an extra certificate to a certificate bundle, the first idea might be something like this:

cat example.pem >> /etc/ssl/certs/ca-bundle.crt

And that will work - if you are a root user. However, if you are just a sudoer, or God-forbid a cloud-user, you might find yourself up the creek without a paddle.

You see, sudo won’t simply do as it operates on the source and not the destination:

sudo cat example.pem >> /etc/ssl/certs/ca-bundle.crt
 -bash: /etc/ssl/certs/ca-bundle.crt: Permission denied

You might want to play with tee, but depending on your exact permissions that might fail also.

cat example.pem | sudo tee -a /etc/ssl/certs/ca-bundle.crt

However, one command that never fails is vi as it has quite a lot of code to work out writing to read-only files. And conveniently, we can script it too:

sudo ex +"r /opt/install/ca.pem" -scwq! /etc/ssl/certs/ca-bundle.crt

Yep, we essentially start vi, read the extra content into it, and then force-save it all.

Configuring Thumb Button on M720 Under Ubuntu

The most usable function I found for my mouse thumb button is actually Minimize window. When you combine it with Alt+Tab, it does wonders for fast window switching. Under Windows it’s trivial to configure this within Logitech Options. While not as easy in Linux, it’s actually not that hard either.

Since the Thumb button on most Logitech’s devices is not a mouse button but a key combination, we can use Ubuntu’s built-in Keyboard utility instead of remapping buttons. Or we can do it from command line equally easy:

gsettings set org.gnome.desktop.wm.keybindings minimize "['<Primary><Alt>Tab']"

If you really want to have quick switching, you might want to also disable animations:

gsettings set org.gnome.desktop.interface enable-animations false

With those two changes you have your thumb button configured.

PS: And yes, this works for MX Master too.

Remap M720 Mouse Buttons Under Ubuntu

M720 mouse has quite a few buttons. But remapping them under Linux is not necessarily the easiest thing in the world. For example, if one wants Forward and Backward buttons remapped to window left/right snap, there are a few things that need to be done manually.

First we need a few packages:

sudo apt install -y xbindkeys xautomation

Then we need to figure out which code our forward and backward keys are:

xev | grep -A 2 Button

In my case, these were 8 and 9 respectively.

Then we need to write mappings in ~/.xbindkeysrc:

"xte 'keydown Super_L' 'key Left' 'keyup Super_L'"
b:9

"xte 'keydown Super_L' 'key Right' 'keyup Super_L'"
b:8

And lastly, we need to restart xbindkeys:

killall xbindkeys 2>/dev/null
xbindkeys

If all went well, your buttons now know a new trick.

UEFI Install for Root ZFS Ubuntu 18.10

There is a newer version of this guide for Ubuntu 19.04.

Booting ZFS Ubuntu of MBR is story I already told. But what if we want an encrypted UEFI ZFS setup?

Well, it’s quite simple to previous steps and again just a derivation on ZFS-on-Linux project.

As before, we first need to get into root prompt:

sudo -i

Followed by getting a few basic packages ready:

apt-add-repository universe
apt update
apt install --yes debootstrap gdisk zfs-initramfs

Disk setup is quite simple with only two partitions:

sgdisk --zap-all             /dev/disk/by-id/^^ata_disk^^

sgdisk -n2:1M:+511M -t2:EF00 /dev/disk/by-id/^^ata_disk^^
sgdisk -n1:0:0      -t1:8300 /dev/disk/by-id/^^ata_disk^^

sgdisk --print               /dev/disk/by-id/^^ata_disk^^
 Number  Start (sector)    End (sector)  Size       Code  Name
    1         1050624        67108830   31.5 GiB    8300
    2            2048         1050623   512.0 MiB   8300

I believe full disk encryption should be a no-brainer so of course we set up LUKS:

cryptsetup luksFormat -qc aes-xts-plain64 -s 512 -h sha256 /dev/disk/by-id/^^ata_disk^^-part1
cryptsetup luksOpen /dev/disk/by-id/^^ata_disk^^-part1 luks1

Creating ZFS stays the same as before:

zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 -O normalization=formD \
    -O xattr=sa -O mountpoint=none rpool /dev/mapper/luks1
zfs create -o canmount=noauto -o mountpoint=/mnt/rpool/ rpool/system
zfs mount rpool/system

Getting basic installation on our disks follows next:

debootstrap cosmic /mnt/rpool/
zfs set devices=off rpool
zfs list

And then we setup EFI boot partition:

mkdosfs -F 32 -n EFI /dev/disk/by-id/^^ata_disk^^-part2
mount /dev/disk/by-id/^^ata_disk^^-part2 /mnt/rpool/boot/

We need to ensure boot partition auto-mounts:

echo PARTUUID=$(blkid -s PARTUUID -o value /dev/disk/by-id/^^ata_disk^^-part2) /boot vfat noatime,nofail,x-systemd.device-timeout=5s 0 1 >> /mnt/rpool/etc/fstab
cat /mnt/rpool/etc/fstab

Before we start using anything, we should prepare a few necessary files:

cp /etc/hostname /mnt/rpool/etc/hostname
cp /etc/hosts /mnt/rpool/etc/hosts
cp /etc/netplan/*.yaml /mnt/rpool/etc/netplan/
sed '/cdrom/d' /etc/apt/sources.list > /mnt/rpool/etc/apt/sources.list

If you are dual-booting system with Windows, do consider turning off UTC BIOS time:

echo UTC=no >> /mnt/rpool/etc/default/rc5

With chroot we can get the first taste of our new system:

mount --rbind /dev  /mnt/rpool/dev
mount --rbind /proc /mnt/rpool/proc
mount --rbind /sys  /mnt/rpool/sys
chroot /mnt/rpool/ /bin/bash --login

Now we can update our software:

apt update

Immediately followed with locale and time zone setup:

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we install Linux image and basic ZFS boot packages:

apt install --yes --no-install-recommends linux-image-generic
apt install --yes zfs-initramfs

Since we’re dealing with encrypted data, our cryptsetup should be also auto mounted:

apt install --yes cryptsetup

echo "luks1 UUID=$(blkid -s UUID -o value /dev/disk/by-id/^^ata_disk^^-part1) none luks,discard,initramfs" >> /etc/crypttab
cat /etc/crypttab

Now we get grub started:

apt install --yes grub-efi-amd64

And update our boot environment again (seeing errors is nothing unusual):

update-initramfs -u -k all

And then we finalize our grup setup:

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=ubuntu --recheck --no-floppy

Finally we get the rest of desktop system:

apt-get install --yes ubuntu-desktop samba linux-headers-generic
apt dist-upgrade --yes

We can omit creation of the swap dataset but I always find it handy:

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput -o sync=always \
    -o primarycache=metadata -o secondarycache=none rpool/swap
mkswap -f /dev/zvol/rpool/swap
echo "/dev/zvol/rpool/swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

If one is so inclined, /home directory can get a separate dataset too:

rmdir /home
zfs create -o mountpoint=/home rpool/data

Only remaining thing before restart is to create user:

adduser ^^user^^
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sambashare,sudo ^^user^^
chown -R ^^user^^:^^user^^ /home/^^user^^

As install is ready, we can exit our chroot environment and reboot:

exit
reboot

You will get stuck after the password prompt as our mountpoint for system dataset is wrong. That’s easy to correct:

zfs set mountpoint=/ rpool/system
exit
reboot

Assuming nothing went wrong, your UEFI system is now ready.

PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 21.10 and 20.04

PPS: For LUKS-based ZFS setup, check the following posts: 20.04, 19.10, and 19.04.

Exporting JSON via Awk

I wanted to process Unicode CSV file to extract the first two columns into JSON. With awk it seemed easy enough:

awk '
    BEGIN {
        FS=";"
        print "["
    }
    {
        print "  { \"code\": \"" $1 "\", \"description\": \"" $2 "\" },"
    }
    END {
        print "]"
    }
    ' UnicodeData.txt | less

This will give you ALMOST parsable output. One thing that will spoil it is the last “hanging” comma making the whole JSON invalid (albeit some parsers will still load it). And no, there is no way to tell awk to do something special with the last line as processing of the lines is done one-by-one and thus there is no telling which line is last at any give moment.

What we can do is tell awk to process lines with a single line delay:

awk '
    BEGIN {
        FS=";"
        print "["
    }
    NR>1 {
        print "    { \"code\": \"" code "\", \"description\": \"" description "\" },"
    }
    {
        code = $1
        description = $2
    }
    END {
        print "    { \"code\": \"" code "\", \"description\": \"" description "\" }"
        print "]"
    }
    ' UnicodeData.txt | less

This prints content starting from the second line (NR>1) and what we do in the main loop is just storing fields into our variables that’ll be read in the next iteration. Essentially what we have is a single line delay mechanism. To catch up with the last line we just print it out without trailing comma in END portion of our program.

Valid JSON at last.