Let's Encrypt and Handshake Failure

After my web server worked for a few days, I wanted to add additional domain. Easy enough, done it before - I though. However, I got hit by an error as I tried to get Let’s Encrypt certificate in place: The server experienced a TLS error during domain verification :: remote error: tls: handshake failure.

The most interesting thing was that I haven’t changed anything on the web server. So I tried a few old commands (history is handy here) with the Let’s Encrypt’s staging server. Although I was sure they worked when I was originally setting up the server, I was suddenly presented with the same error for all of them. And nothing changed!

After looking at the error message a bit more, I suddenly remembered that one thing did change - I moved my domain records to CloudFlare. Certbot was trying to authenticate my web server but ended up contacting CloudFlare and fetching their TLS certificate for my web site instead. Solution was simple - just temporarily disable caching on CloudFlare and certbot will successfully issue the certificate.

While solution was simple, it wasn’t ideal as this also meant I would need to repeat the same disabling every 90 days in order to do certificate renewal. I wanted solution that would allow automating renewal without the need for any manual action. This came by slightly altering how certbot performs verification.

certbot renew --cert-name ^^example.com^^ --webroot --webroot-path ^^/var/www/html/^^ --post-hook "apachectl graceful"

Alternative approach above stores certificate into .well-known directory under the root web site path and thus works nicely with the CloudFlare caching.

MAC Address Anatomy

While most people understand the structure behind IPv4 and even IPv6 address, MAC address is usually just assumed as something unique enough not to cause routing issues. But did you ever think whether there is any non-random meaning behind those 48 bits usually expressed as 12 hexadecimal digits?

First bit that gets transmitted is determining if given MAC is for individual device (0) or for a group (1). Most of the time we just care about individual type with group support mostly being used for multicast albeit historically a few more usages were known.

Next transmitted bit controls whether MAC is intended to be globally unique (0) or its intended to be unique only on local level (1). If properly manufactured, all physical network devices will have that bit set to 0 and you can count on it being unique. That being said, I have once personally experienced duplicate MAC on cheap network (ISA) card. However, that is unlikely scenario and hopefully one will never need to troubleshoot it these days.

While one would think locally administered addresses would be an awesome match for virtualization platforms, most don’t take advantage of it and have their addresses instead (incorrectly) allocated from globally unique pool. One major exception is KVM which by default does take care to set bit correctly. Realistically there is no difference in behavior for either choice but I find it nice that someone is actually paying attention.

Further 22 bits are identifier for the organization. Together with aforementioned 2 bits this makes total of 24 bits, also known as three bytes (or octets if you are a stickler). These identifiers are assigned by IEEE and it is easy to check which entity assigned your MAC address. If you despise manual search, you can also check one of the many OUI lookup web sites.

And finally the last 24 bits (the other 3 bytes) are assigned either sequentially or randomly - doesn’t really matter - by manufacturer within their OUI number. The only real requirement is to keep it unique so some manufacturers might have multiple assignments (e.g. almost 500 for Intel Corporations) while others might have just one.

While this covers all 48 bits, there is one piece of puzzle still missing - order of bits. Notice how I said that Individual/Group bit is the one first transmitted. Well, its not the first written. Bits on network get transmitted in least-significant first order.

If we take some imaginary MAC address 9C-DA-3E-E2-34-EB we can express this in binary as 10011100 11011010 00111110 11100010 00110100 11101011. When looking at binary form the Local/Group bit is actually at position 8 while at position 7 you can see the Unique/Local bit.

Those playing with blade servers can now also understand why their network adapters seem to have MAC addresses spaced by 4 - with the lowest two bits set to 0 that decision is forced by math and not some evil plan to waste already small MAC address space.

Professional C++ (Fourth Edition!)

If you are curious about C++ and the news it brings (yes, development is still much alive) you are in luck. Written by Marc Grégoire and dealing with the 17th edition of C++, you are sure to find something interesting.

This C++ release includes a filesystem API, template argument deduction for constructors, optional values, the variant type, the any type, parallel algorithms, string conversion primitives, nested namespaces, and more. Considering the wide net this edition has casted, you are sure to find something useful for your development.

While C++ is not the easiest language to learn or perfect I found that a lot of examples is extremely helpful to make this medicine go down. And this book does deliver as examples are available for both Windows and Linux. Even better, you can check examples without downloading book. A bit cheeky but excellent way to determine how interested in the book you might be.

Book is published by Wiley/Wrox and available at Amazon.

Limiting Web Page Access To a Certificate

While my site is publicly accessible (as proved by the fact you’re reading this), I also have a few more private domains. Whether for stats or just testing, some areas are simply not meant for general public. My usual approach has been to simply turn on basic password authentication and call it a day.

But, as I serve as my own certificate authority, I started to wonder whether I can skip password altogether and rely on client-side TLS certificate.

The easiest approach is simply modifying virtual host section in httpd.conf to verify client certificate:

<VirtualHost *:443>
  …
  SSLVerifyClient require
  SSLVerifyDepth 1
  SSLCACertificateFile /srv/www/ca.crt
</VirtualHost>

However, this approach has downside of blocking every single file. If you are using Let’s Encrypt, congratulations, renewals are successfully blocked too. Since that is not really desired, a bit more complicated setup is needed.

Virtual host section remains almost the same, albeit with slight difference on where SSLVerifyClient is handled:

<VirtualHost *:443>
  …
  <Location "/">
    SSLVerifyClient require
  </Location>
  <Location "/.well-known/">
    SSLVerifyClient none
  </Location>
  SSLVerifyDepth 1
  SSLCACertificateFile /srv/www/ca.crt
</VirtualHost>

With these directives, your website is as private as you want it to be while still allowing Let’s Encrypt to work.

PS: To create a certificate, you can use my micro CA script. It supports both using centralized CA:

./microca.sh -s "CN=Something" -u client -q -x ^^MyCertificate^^

Or self-signed certificate:

./microca.sh -s "CN=Something" -u client -q -x -p ^^MyCertificate^^

PPS: Do not forget to load certificate into browser of your choice (or Windows Certificate Store).

Chrony NTP Server on CentOS 7.3

I already wrote about setting pool NTP server using good old ntpd daemon. However, CentOS comes with another daemon installed by default - Chrony. Let me guide you through the setup of Chrony NTP server on CentOS 7.3 for the purpose of joining the pool.

While NTP server needs a good (single-core) CPU and fast network, it cares nothing about the RAM. That makes it ideal for even the smallest cloud instances. Both $5 Linode (1024 MB RAM) and $2.50 Vultr (512 MB RAM) will work wonderfully. Just don’t set your pool speed over 100 Mbps to avoid any extra bandwidth charges.

Immediately after CentOS has been installed, it’s best to update system to the latest and greatest:

yum -y upgrade

As Chrony is installed by default on CentOS, there are no new packages to install. However, the firewall still must allow for external requests:

firewall-cmd --permanent --add-service ntp
firewall-cmd --reload

Just installing NTP is no good if we don’t have any servers to synchronize with. To setup those, editing /etc/chrony.conf is needed. Depending which data center you select for the server, you will want to have 4 to 7 servers from the stratum one list physically close to your location while obeying any restrictions noted (especially if server is open access or not). For the virtual machine located in Miami, the following servers would work (do not forget to remove servers already in file):

server ntp-s1.cise.ufl.edu iburst
server time-a.bbnx.net iburst
server time-b.bbnx.net iburst
server time-a.timefreq.bldrdoc.gov iburst
server time-c.timefreq.bldrdoc.gov iburst
…

Of course, a server won’t be the server until we add the following line to /etc/chrony.conf so that external connections are accepted while control is limited to local machine only:

…
allow all
bindcmdaddress 127.0.0.1
bindcmdaddress ::1

Now finally, with all configured, we can restart our server:

systemctl restart chronyd

To verify the server is working, the first step is to see all sources the server synchronizes with. As soon as one of them has asterisk next to its name, all is good.

chronyc sources

A bit more realistic check will be actually requesting the time from remote computer:

ntpdate -q ^^<ip>^^

Once server works for a while, it can be added to the NTP.org pool. Since the server has both IPv4 and IPv6 address (I surely hope you added IPv6 ;)), it will be necessary to add it twice. At the start it will be only monitored but, as its score increases, soon you will start seeing traffic from all over the world.

PS: It would be wise to keep 1 Mpbs as defined speed until you see how much traffic you’re actually getting. Once the server has been handling requests at basic speed for a while you can think about increasing the number.