Serial Number In Intel Hex - Bash Edition

When creating USB devices having an unique serial number for each is quite important. Yes, stuff will work with duplicate (or missing) serial number but you might notice Windows will get confused. With the serial devices the most noticeable effect will be a different port assignment on each plug. Not a breaking deal but a bit annoying.

The easiest way to deal with this is having a dummy serial number in code and then using post-build action to change it in the final Intel hex file. For TmpUsb project I even created a script to do exactly this. However, that script was written in PowerShell which until recently (March 2020) didn’t work on Linux and it definitely doesn’t come installed there by default. This and a few other reasons were why I decided to rewrite this in Bash for my CyberCard project.

The first step is setting up the serial number. The most crucial step here is __at macro that will ensure serial number is always at a predictable location. For my example, I selected location close to the end of addressable space (0x1F00). One could use special characters here but I would highly advise to have a valid USB serial number just in case script fails. That means at least 12 alphanumeric characters but I always go for numbers-only as it makes script a bit easier.

//Serial string descriptor
const struct{uint8_t bLength;uint8_t bDscType;uint16_t string[13];}sd003 ^^__at(0x1F00)^^={
    sizeof(sd003),USB_DESCRIPTOR_STRING, {
        '1','9','7','9','0','1','2','8','1','8','1','5','0'
    }};

The final Intel hex file location will end up multiplied by 2 and thus at offset 0x3E00 further shifted by 4 hexadecimal characters. Script then needs to replace every 4th hexadecimal byte up to the count of 13 in my case (as my serial number is 13 digits long).

The only task remaining is calling it once project is done building. Previously I would do this using project properties dialog (Project Properties, Conf, Building, Execute this line after build) but that proved to be an issue if you want to compile on both Windows and Linux. While Linux setup was quite straightforward, on Windows it would get confused if you have any Linux-based toolset installed (e.g. Git Bash) and throw an error:

"User defined post-build step: [update-serial.sh]"
nbproject/Makefile-default.mk:105: recipe for target '.build-conf' failed
make[1]: Leaving directory 'Q:/Projects/Electronics/CyberCard/Firmware/Source'
nbproject/Makefile-impl.mk:39: recipe for target '.build-impl' failed
process_begin: CreateProcess(NULL, bash Q:\Projects\Electronics\CyberCard\Firmware\Source\update-serial.sh, ...) failed.
make (e=2): The system cannot find the file specified.

One workaround was to simply wrap call to this script in .cmd (or .bat) file but that would cause issues when compiling under Linux. Only if there was a way to execute one command for Windows and another one when running on Linux… Cue the Makefile.

While you cannot differentiate OS from GUI, you can go and edit MPLAB project Makefile directly. There we can point toward .cmd on Windows and toward .sh when running on Linux.

.build-post: .build-impl
# Add your post 'build' code here...
ifeq ($(OS),Windows_NT)
        update-serial.cmd
else
        ./update-serial.sh
endif

All that is left is actually placing the script(s) in the same directory the Makefile is situated (download here) and you’re good assuming you defined serial number as I did. If you didn’t, just adjust LOCATIONS variable inside.


PS: This assumes you have Git (or some other bash-capable environment) installed in Windows.

PPS: Don’t forget to use <Tab> in Makefile for indentation. Spaces won’t do here.

Moving Mercurial Repository from BitBucket to GitHub

I already wrote once about moving from Mercurial to Git. That was way back in 2015 when I moved most of my repositories. However, I didn’t move all. Quite a few old electronics projects remained on BitBucket, my Mercurial home of choice. With BitBucket removing Mercurial starting June 1st, it was time to move remainder.

First you will need Ubuntu Linux with a few tools.

sudo apt install --yes git mercurial wget

mkdir ~/bin
wget https://raw.github.com/felipec/git-remote-hg/master/git-remote-hg -O ~/bin/git-remote-hg
chmod +x ~/bin/git-remote-hg

Here the git-remote will do most of the heavy lifting as it will enable cloning of Mercurial repository (if it fails, try adding --config format.sparse-revlog=0 option):

git clone hg::ssh://hg@bitbucket.org/^^user^^/^^project^^

With repository now in Git format, we can treat it as any other repository and push it to our server. In the case of GitHub, that would look something like this:

git remote rm origin
git remote add origin git@github.com:^^user^^/^^project^^.git
git push --all 

That’s all. Rinse and repeat until there is no Mercurial repo left.

PS: Considering how great Mercurial is for beginners, I am sad to see it’s primary online home gone. There are other places allowing hosting of Mercurial repositories but BitBucket was among first and it actually worked remarkably well.

PPS: Yes, this works for local repositories too.

Unique Machine ID

Every once in a while I need to uniquely identify a machine. I don’t case about its name or properties. I just want something unique to key off and ideally I want its format to be the same on both Windows and Linux.

On Linux you can use content of /etc/machine-id. On pretty much all distributions I’ve tried this file will be generated on install and will contain UUID. For Windows, one must look in Registry. Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography\MachineGuid Windows will keep their unique identifier also in UUID format (albeit with dashes this time).


PS: Note these values can be manually changed by user so you cannot really depend on them for security purposes (e.g. generating licenses). But for machine identification they’ll do.

Installing UEFI ZFS Root on Ubuntu 20.04

There is a newer version of this guide for Ubuntu 20.04, but with native ZFS encryption.


Ubuntu 20.04 has a ZFS setup option as of 19.10. And frankly, you should use it instead of the manual installation procedure. However, manual installation does offer it’s advantages - especially when it comes to pool layout and naming. If manual installation is needed, there is a great Root on ZFS installation guide that’s part of ZFS-on-Linux project but its final ZFS layout is a bit too complicated for my taste. Here is my somewhat simplified version of the same, intended for a singe disk installations.

After booting into Ubuntu desktop installation we want to get a root prompt. All further commands are going to need root credentials anyhow.

sudo -i

The very first step should be setting up a few variables - disk, pool, host name, and user name. This way we can use them going forward and avoid accidental mistakes. Just make sure to replace these values with ones appropriate for your system.

DISK=/dev/disk/by-id/^^ata_disk^^
POOL=^^Ubuntu^^
HOST=^^desktop^^
USER=^^user^^

General idea of my disk setup is to maximize amount of space available for pool with the minimum of supporting partitions. If you are planning to have multiple kernels, increasing boot partition size might be a good idea.

blkdiscard $DISK

sgdisk --zap-all                        $DISK

sgdisk -n1:1M:+127M -t1:EF00 -c1:EFI    $DISK
sgdisk -n2:0:+512M  -t2:8300 -c2:Boot   $DISK
sgdisk -n3:0:0      -t3:8309 -c3:Ubuntu $DISK

sgdisk --print                          $DISK

Unless there is a major reason otherwise, I like to use disk encryption.

cryptsetup luksFormat -q --cipher aes-xts-plain64 --key-size 512 \
    --pbkdf pbkdf2 --hash sha256 $DISK-part3

I like to use disk name as the name of mapped (encrypted) luks device when I open it, but really anything goes.

LUKSNAME=<code>basename $DISK</code>
cryptsetup luksOpen $DISK-part3 $LUKSNAME

Finally we’re ready to create system ZFS pool.

zpool create -o ashift=12 -o autotrim=on \
    -O compression=lz4 -O normalization=formD \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto -O atime=off \
    -O canmount=off -O mountpoint=none -R /mnt/install $POOL /dev/mapper/$LUKSNAME
zfs create -o canmount=noauto -o mountpoint=/ $POOL/root
zfs mount $POOL/root

Assuming UEFI boot, two additional partitions are needed. One for EFI and one for booting. Unlike what you get with the official guide, here I don’t have ZFS pool for boot partition but a plain old ext4. I find potential fixup works better that way and there is a better boot compatibility. If you are thinking about mirroring, making it bigger and ZFS might be a good idea. For a single disk, ext4 will do.

yes | mkfs.ext4 $DISK-part2
mkdir /mnt/install/boot
mount $DISK-part2 /mnt/install/boot/

mkfs.msdos -F 32 -n EFI $DISK-part1
mkdir /mnt/install/boot/efi
mount $DISK-part1 /mnt/install/boot/efi

To start the fun we need debootstrap package.

apt install --yes debootstrap

Bootstrapping Ubuntu on the newly created pool is next. This will take a while.

debootstrap focal /mnt/install/

zfs set devices=off $POOL

Our newly copied system is lacking a few files and we should make sure they exist before proceeding.

echo $HOST > /mnt/install/etc/hostname
sed "s/ubuntu/$HOST/" /etc/hosts > /mnt/install/etc/hosts
sed '/cdrom/d' /etc/apt/sources.list > /mnt/install/etc/apt/sources.list
cp /etc/netplan/*.yaml /mnt/install/etc/netplan/

If you are installing via WiFi, you might as well copy your wireless credentials:

mkdir -p /mnt/install/etc/NetworkManager/system-connections/
cp /etc/NetworkManager/system-connections/* /mnt/install/etc/NetworkManager/system-connections/

Finally we’re ready to “chroot” into our new system.

mount --rbind /dev  /mnt/install/dev
mount --rbind /proc /mnt/install/proc
mount --rbind /sys  /mnt/install/sys
chroot /mnt/install \
    /usr/bin/env DISK=$DISK POOL=$POOL USER=$USER LUKSNAME=$LUKSNAME \
    bash --login

Let’s not forget to setup locale and time zone.

locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

dpkg-reconfigure tzdata

Now we’re ready to onboard the latest Linux image.

apt update
apt install --yes --no-install-recommends linux-image-generic linux-headers-generic

Followed by boot environment packages.

apt install --yes zfs-initramfs cryptsetup keyutils grub-efi-amd64-signed shim-signed tasksel

Since we’re dealing with encrypted data, we should auto mount it via crypttab. If there are multiple encrypted drives or partitions, keyscript really comes in handy to open them all with the same password. As it doesn’t have negative consequences, I just add it even for a single disk setup.

echo "$LUKSNAME UUID=$(blkid -s UUID -o value $DISK-part3) none \
    luks,discard,initramfs,keyscript=decrypt_keyctl" >> /etc/crypttab
cat /etc/crypttab

To mount boot and EFI partition, we need to do some fstab setup too:

echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part2) \
    /boot ext4 noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
echo "PARTUUID=$(blkid -s PARTUUID -o value $DISK-part1) \
    /boot/efi vfat noatime,nofail,x-systemd.device-timeout=5s 0 1" >> /etc/fstab
cat /etc/fstab

Now we get grub started and update our boot environment.

KERNEL=<code>ls /usr/lib/modules/ | cut -d/ -f1 | sed &#039;s/linux-image-//&#039;</code>
update-initramfs -u -k $KERNEL

Grub update is what makes EFI tick.

update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Ubuntu \
    --recheck --no-floppy

Finally we install out GUI environment. I personally like ubuntu-desktop-minimal but you can opt for ubuntu-desktop. In any case, it’ll take a considerable amount of time.

tasksel install ubuntu-desktop-minimal

Short package upgrade will not hurt.

apt dist-upgrade --yes

We can omit creation of the swap dataset but I personally find a small one handy.

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off -o logbias=throughput \
    -o sync=always -o primarycache=metadata -o secondarycache=none $POOL/swap
mkswap -f /dev/zvol/$POOL/swap
echo "/dev/zvol/$POOL/swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

If one is so inclined, /home directory can get a separate dataset too.

rmdir /home
zfs create -o mountpoint=/home $POOL/home

The only remaining task before restart is to create the user, assign a few extra groups to it, and make sure its home has correct owner.

sudo adduser --disabled-password --gecos '' $USER
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sudo $USER
passwd $USER

As install is ready, we can exit our chroot environment.

exit

And cleanup our mount points.

umount /mnt/install/boot/efi
umount /mnt/install/boot
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

After the reboot you should be able to enjoy your installation.

reboot

PS: There are versions of this guide using the native ZFS encryption for other Ubuntu versions: 21.10 and 20.04

PPS: For LUKS-based ZFS setup, check the following posts: 22.10, 19.10, 19.04, and 18.10.

[2020-05-18: Changed boot partition size to 512M (was 384M). Reason is ever increasing size of kernel making it difficult to do future upgrades without going through cleanups if you use multiple kernels.]

[2020-06-27: Added blkdiscard and autotrim.]

The Cost of A+

Illustration

As someone maintaining my own web server, I often use various tools to determine if things are good. As web servers are not my daily job, I found that is the only way to save both sanity and time.

One of the most helpful tools comes courtesy of SSL Labs. Their SSL/TLS test suite is both simple to use and full of really good data. While getting a good score doesn’t guarantee everything is secure, it shows you are doing at least some things right.

As of Jan 31st 2020, SSL Labs decided to cap grade to B for lower TLS (1.0 and 1.1) protocols. That means even if your server was a class star until then, starting February it got relegated to a B league. Totally unacceptable!

Fortunately, if you are using Apache, change is easy:

SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:-MEDIUM:!LOW:!aNULL:!MD5
SSLHonorCipherOrder on

With this adjustment your server can enjoy A+ again.


PS: Cost? Say goodbye to Android 4.3, Windows Phone 8, Java 7, Safari 6, and Internet Explorer on Windows 7. For me personally all things I can live without.

PPS: If you want to disable some algorithms manually, a table mapping between OpenSSL and IANA names will be most useful.

PPPS: For curious, here are my TLS definitions for Apache:

SSLProtocol "TLSv1.3" "+TLSv1.2"
SSLCipherSuite "TLSv1.3" "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256"
SSLCipherSuite "HIGH -MEDIUM !LOW !aNULL !MD5 !3DES !AES128 !ARIA128 !CAMELLIA !RSA"
SSLHonorCipherOrder on
SSLOpenSSLConfCmd Curves secp521r1:secp384r1:prime256v1
SSLOpenSSLConfCmd ECDHParameters secp384r1
SSLOptions +StrictRequire
SSLCompression off
SSLStrictSNIVHostCheck off

SSLSessionCache "shmcb:/run/httpd/sslcache(512000)"
SSLSessionCacheTimeout 300

SSLStaplingCache "shmcb:/tmp/stapling_cache(128000)"
SSLUseStapling on