Parsing GZip Stream Without Looking Back

Some files can exist in two equivalent forms - compressed and uncompressed. One excellent example is .pcap. You can get it as standard .pcap we all know and love but it also comes compressed as .pcap.gz. To open a compressed file in C#, you could pass it to GZipStream - it works flawlessly. However, before doing that you might want to check if you’re dealing with compressed or uncompressed form.

Check itself is easy. Just read first 2 bytes and, if they’re 0x1F8B, you’re dealing with a compressed stream. However, you just consumed 2 bytes and simply handing over file stream to GZipStream will no longer work. If you are dealing with file on a disk, just seek backward and you’re good. But what if you are dealing with streaming data and seeking is not possible?

For .pcap and many more transparently compressed formats, you can simply decide to skip into bread-and-butter of encryption - deflate algorithm. You see, GZip is just a thin wrapper over deflate stream. And quite often it only has a fixed size header. If you move just additional 8 bytes (thus skipping a total of 10), you can use DeflateStream and forget about “rewinding.”

Wanna see example? Check constructor of PcapReader class.

SignTool and Error -2146869243/0x80096005

As I was trying out my new certificate, I got the following error:

SignTool Error: An unexpected internal error has occurred.
Error information: "Error: SignerSign() failed." (-2146869243/0x80096005)

Last time I had this error, I simply gave up and used other timeserver. This time I had a bit more time and wanted to understand from where the error was coming. After a bit of checking, I think I got it now. It’s the digest algorithm.

SignTool still uses SHA-1 as default. Some servers (e.g. timestamp.digicert.com) are ok with that. However, some servers (e.g. timestamp.comodoca.com and timestamp.sectigo.com) are not that generous. They simply refuse to use weak SHA-1 for their signature.

Solution is simple - just add /td sha256 to the list of codesign arguments.

SDR on Ubuntu x86

[This is a post 3 in two-part series :), for hardware setup go here]

While running SDR radio on Raspberry was fine, I kinda wanted to move this to one of my x86 servers. Due to this, I had to revisit my old guide.

When device is plugged in, trace should be seen in dmesg. If everything is fine, you should see some activity.

dmesg | tail
 …
 [4306437.661393] usbcore: registered new interface driver dvb_usb_rtl28xxu

To get SDR running, there is some work involved with its compilation. Note the DETACH_KERNEL_DRIVER=ON flag enabling SDR application to access device without disabling its driver. Rest is really similar to official instructions.

sudo apt-get install -y git build-essential cmake libusb-1.0-0-dev libglib2.0-dev
cd ~
git clone git://git.osmocom.org/rtl-sdr.git
cd rtl-sdr/
mkdir build
cd build
cmake ../ -DINSTALL_UDEV_RULES=ON -DDETACH_KERNEL_DRIVER=ON
make
sudo make install
sudo ldconfig

This is an ideal time to test it. As I have the iptables active, I manually enable port on external interface. Other than that I will not restrict application to a single IP but allow it to listen on all interfaces.

iptables -A INPUT -i ^^eth0^^ -p tcp --dport 1234 -j ACCEPT
/usr/local/bin/rtl_tcp -a 0.0.0.0

The last step is to enable running it as a service. We need to create a separate user, enable service, and finalize it all with reboot.

sudo adduser --disabled-password --gecos "" sdr
sudo usermod -a -G plugdev sdr

sudo cat > /lib/systemd/system/rtl_tcp.service <<- EOF
[Unit]
After=network.target

[Service]
Type=exec
ExecStart=/usr/local/bin/rtl_tcp -a 0.0.0.0
KillMode=process
Restart=on-failure
RestartSec=10
User=sdr

[Install]
WantedBy=multi-user.target
Alias=rtl_tcp.service
EOF

sudo systemctl enable /lib/systemd/system/rtl_tcp.service
sudo reboot

And that’s it. Now you can run SDR TCP server on your Ubuntu server.

Merging Two Git Repositories

As I went onto rewriting QText, I did so in the completely new repository. It just made more sense that way. In time, this new code became what the next QText version will be. And now there’s a question - should I still keep it in a separate repository?

After some thinking, I decided to bring the new repository (QTextEx) as a branch in the old repository (QText). That way I have a common history while still being able to load the old C# version if needed.

All operations below are to executed in the destination repository (QText).

The first step is to create a fresh new branch without any common history. This will ensure Git doesn’t try to do some “smart” stuff when we already know these repositories are unrelated.

git switch --discard-changes --orphan ^^new^^

This will leave quite a few files behind. You probably want to clean those before proceeding.

The next step is to add one repository into the other. This can be done by adding remote into destination, pointing toward the source. Remote can be anywhere but I find it easiest if I use it directly from my file system. After fetching the repository, a simple merge is all it takes to get the commits. Once that’s done, you can remove the remote.

git remote add ^^QTextEx^^ ^^../QTextEx^^
git fetch --tags ^^QTextEx^^
git merge --ff --allow-unrelated-histories ^^QTextEx^^/main
git remote remove ^^QTextEx^^

After a push, you’ll see that all commits from the source repository are now present in destination too.

PS: Do make backups and triple verify all before pushing it upstream.

PPS: This is a permanent merge of histories. Consider using subtree if you want to keep them separate.

PPPS: Things get a bit more complicated if you want to transfer multiple branches from the other repository.

Testing Native ZFS Encryption Speed (20.10)

[2020-11-02: There is a newer version of this post]

Back in the days of Ubuntu 20.04, I did some ZFS native encryption testing. Results were not promising to say the least but they were done using ZFS 0.8.3 on Ubuntu 20.04. There was a hope that Ubuntu 20.10 bringing 0.8.4 would have a lot of performance improvements. So I repeated my testing.

First I tested CCM and saw that results were 10-15% lower than in 20.04. However, this was probably not due to ZFS changes as both Luks and no-encryption numbers dropped too. As my testing was done on a virtual machine, it might not be anything related to Ubuntu at all. For all practical purposes, you can view those results as unchanged.

However, when I tested GCM encryption speed, I had to repeat test multiple times because I couldn’t believe the results I was seeing. ZFS native encryption using GCM was only about 25% slower than no encryption at all and handily beating Luks numbers. Compared to the last year’s times, GCM encryption got a fivefold improvement. That’s what I call optimization.

Last year I suggested going with native ZFS encryption only when you are really interested in ZFS having direct physical access to drives or if you were interested in encrypted send/receive. For performance critical scenarios, Luks was the way to go.

Now I can honestly recommend going with the native ZFS encryption (provided you use GCM). It’s as fast as Luks, allows ZFS to handle physical drives directly, and simplifies the setup. The only scenario where Luks still matters is if you want to completely hide your disk content as native encryption does leak some metadata (e.g., dataset properties). And no, you don’t need to upgrade to 20.10 for the speed as some performance improvements have been backported to 20.04 too.

I have migrated my own main file server to ZFS native encryption some time ago mostly to give ZFS direct disk access and without much care for array speed. Now there is no reason not to use it on desktop either.

PS: You can take a peek at the raw data if you’re so inclined.

PPS: Test procedure is in the previous post so I didn’t bother repeating it here.