Custom Directory for Apache Logs

On my web server I wanted to use a separate directory for my logs. All I needed was to configure ErrorLog and CustomLog directives and that’s it. Well, I did that only to have following error: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

And no, there weren’t any details worth mentioning in systemctl status httpd.service nor journalctl -xe.

To cut long story short, after a bit of investigation I narrowed the problem to SELinux that is enabled by default on CentOS. Armed with that knowledge, I simply transferred security from default log directory to my desired location:

chcon -R --reference=/etc/httpd/logs/ ^^/var/www/logs/^^

With that simple adjustment, my httpd daemon started and my logs lived happily ever after.

Bimil 2.30

This version is all about enabling you to quickly see which passwords suck. For this purpose a centralized weak password search has been implemented. Yes, you could see which password is weak even before but it required opening every single account - and that can take a while. This way you get the same information but after a single click.

Additionally, there is an option to check all accounts for breaches at Have I been pwned? site. While password for these accounts might not be compromised themselves, risk is quite increased and changing them is not necessarily the worst idea - let’s not even think about the sites using trivial hashes (like MD5) or no hashing at all. Search is smart enough to verify when exactly you changed your password last time to avoid false positives.

Moreover, if you go into Options, you can enable more thorough search. If selected, you can verify all your passwords (hashed and sent over TLS 1.2) against all exposed passwords. This is not enabled by default (even hidden a bit) because it requires quite a big leap of faith toward Troy and his website. I personally do trust him, but your mileage might vary.

As always, new version is available from Bimil’s page or you can update it through application.

Linode NTP

One of the features I added to Bimil was NTP client support for time-based two-factor authentication. For this I needed NTP server so I turned to ntp.org pool and requested vendor zone. Once zone got approved I suddenly had infinite* amount of NTP servers at my disposal.

So, when I decided to give Linode’s $5 virtual server a try, I didn’t want just to create dummy machine. I also wanted to do something for community. As NTP pool service is one of invisible pillars of Internet-connected devices and I was really happy such service was provided for free to myself, it was easy to decide. I am going to build NTP server.

Creating account on linode was a breeze as it was creating the machine. It was literally, click-next, click-next process. Once I finally logged on to it, the first action was to update system to the latest packages. Surprisingly, on Linode there was literally nothing to do - all was already up to date. Awesome!

yum update -y
 …
 No packages marked for update

By default, Linode’s CentOS installation has firewall disabled. As this server will be open to public, enabling firewall is not the worst idea ever:

systemctl start firewalld

systemctl enable firewalld

firewall-cmd --state
 running

And, while dealing with firewall, you might as well allow NTP through and check if configuration is correct:

firewall-cmd --permanent --add-service ntp
 success

firewall-cmd --reload
 success

firewall-cmd --list-all
 public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: ssh dhcpv6-client ^^ntp^^
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

With firewall configuration completed, you can finally install NTP:

yum install -y ntp

And this brings you to the most involved part of the process. You need to go over available stratum 1 time servers and select between four and seven of them for your devious synchronization purposes. Which servers should you select? As long as they are reasonably close (in the terms of network distance) you will be fine.

Using your favorite editor, you need to adjust /etc/ntp.conf file. Following ntp.org recommendations always worked for me but with a slight adjustment in the form of a separate log file and forcing IPv4 resolving for servers. Quite a few IPv6 capable servers only serve clients over IPv6 and don’t like other servers via the same. I personally use the following configuration (don’t forget to adjust servers names):

driftfile /var/lib/ntp/drift

restrict -4 default kod limited nomodify notrap nopeer noquery
restrict -6 default kod limited nomodify notrap nopeer noquery

restrict -4 127.0.0.1
restrict -6 ::1

server -4 ^^clock.fmt.he.net^^ iburst
server -4 ^^clock.sjc.he.net^^ iburst
server -4 ^^usno.hpl.hp.com^^ iburst
server -4 ^^clepsydra.dec.com^^ iburst
server -4 ^^tick.ucla.edu^^ iburst
server -4 ^^time-a.timefreq.bldrdoc.gov^^ iburst
server -4 ^^time-c.timefreq.bldrdoc.gov^^ iburst

logfile /var/log/ntp.log

With configuration ready, it is the moment of truth - start the NTP daemon and configure its automatic startup upon boot. Don’t forget to disable chrony too:

systemctl start ntpd
systemctl enable ntpd
systemctl disable chronyd

With all up, wait for couple minutes while checking state with ntpstat or ntpq. Forgetting it for hour or two will save you lot of angst :) I consider sync good enough whenever pooling interval goes to 1024s.

watch "ntpq -np ; echo ; ntpstat"
      remote           refid      st t when poll reach   delay   offset  jitter                                                            ``==============================================================================``
 *66.220.9.122    .CDMA.           1 u   41  512  377    2.022    6.680   6.798
 +216.218.254.202 .CDMA.           1 u   77 1024  377    2.127    5.663   6.180
 +204.123.2.72    .GPS.            1 u  257  512  377    4.908    2.753   5.031
 +204.123.2.5     .GPS.            1 u   40  512  377    5.232    5.278   6.052
 +164.67.62.194   .GPS.            1 u  532  512  377    9.978   -0.637   3.795
 +132.163.4.101   .NIST.           1 u  362 1024  377   35.226    5.489   7.610
 +132.163.4.103   .NIST.           1 u  430  512  377   35.148    5.353   7.607
 synchronised to NTP server (66.220.9.122) at stratum 2
   time correct to within 19 ms
   polling server every 1024 s

It will take some time for other servers to “discipline” yours so do be patient. If servers are showing INIT refid for a while, this might indicate a permanent issue (e.g. server might be down) or just something temporary (e.g. server might be overloaded). If server is not reachable for a while, toss it out and select another one from stratum 1 list (followed by systemctl restart ntpd).

I personally gave server an hour or two to get into the shape before proceeding with the final step - adding it to pool. This can be done at ntp.org management pages and it is as easy as simply adding server using either host name or IP address.

After monitoring server for some time and assuming its time is stable, your score will raise and you get to be the part of the collective NTP pool.

* some restrictions apply

Speeding-Up GELI on NAS4Free

If you are using GELI to encrypt ZFS disk as I do, there is a slight issue with default NAS4Free configuration. Disk access is slow and everything points toward encryption as culprit.

To test encryption speed, one approach can be to load zero geom and encrypt a few bytes.

kldload geom_zero
geli onetime -e aes-xts -l 128 -s 4096 /dev/gzero
dd if=/dev/gzero.eli of=/dev/null bs=1M count=256

My result was around 40 MB/s. Definitely not what I expected nor it comes even close to what’s needed to support full disk speed.

Further we can see that hardware crypto is supported and that GELI is not using it:

dmesg | grep AESNI
 Features2=0x43d8e3bf<SSE3,PCLMULQDQ,…,TSCDLT,^^AESNI^^,RDRAND>``

geli list | grep Crypto
 Crypto: software
 Crypto: software
 Crypto: software

By default NAS4Free doesn’t load AES encryption module. And AES without hardware support is not as impressive as one would expect.

Fortunately, it is simple to load it. Going to System, Advanced, loader.conf, and adding aesni_load=YES is all that is needed. After reboot, we can check if module is properly loaded:

kldstat | grep aesni.ko
 7    1 0xffffffff8ab3d000 bd50     aesni.ko

geli list | grep Crypto
 Crypto: hardware
 Crypto: hardware
 Crypto: hardware

If one repeats previous test, now speed is more reasonable 350 MB/s. Manually enabling hardware encryption module definitely pays off!

Here are results I saw on my ZFS machines:

PS: Yes, this works for FreeNAS too.

PPS: You can also load module for temporary testing using kldload aesni.

[2018-07-22: NAS4Free has been renamed to XigmaNAS as of July 2018]

Replacing Glass on Casio Edifice EF-308D

My beloved Casio Edifice has been on my wrist for years. And due to “gentleness” on my part, its mineral glass got scratched to the point of impacting readability. It was time for either a new watch or change of the glass. Guess which one I have chosen?

First task was the hardest - I had to find appropriately-sized replacement glass. Diameter was easy as you can measure it with calipers. Thickness was a bit more difficult. Someone doing it properly would disassemble the watch, remove crystal, and measure it. I just eyeballed it and decided that beveled 2.5 mm should fit just fine. And bevel did save me as its “unbeveled” height of 1.6 mm fits nicely in place of Casio’s 2 mm flat mineral glass.

For the new glass, I didn’t just want anything. I wanted to have sapphire as to minimize scratching in the future. And on that front I had choice between generic sapphire glass or a bit fancier AR coated ones. I went with blue AR coating as I felt it would bring a bit of pizzazz.

Once I received my glass, I was ready to begin with process. Instructions for it, of course, I found on YouTube. Disassembly went according to plan as soon as I took the bracelet off to ease handling. I used case opener to get the back off and then spent 10 minutes searching for a small hole to release the crown. With crown out the watch mechanism itself was trivial to remove.

All this was done while wearing finger cots (I find them much more comfortable than full gloves) and with occasional blow from dust blower. With quartz mechanism as one in this Casio, this might not be too important but I read too many horror stories about dust getting into gears so playing it safe seemed reasonable. To protect mechanism and avoid losing small parts I used a small plastic container.

Watch press was the most fidgety part of process because I had to find appropriately sized dies. Pains were worth it because glass did come out in a single piece despite the loud cracking noise.

As I have a good experience with silicon grease from my fountain pen hobby, I decided to get both case and the side of glass generously coated. Not sure whether my glass was better sized than one in video or due to silicone grease but glass was fitted on the first try. I found it so unbelievable that I had to check multiple times it’s really in and fitted straight. While grease probably did help with inserting it also meant I had to spend a few minutes cleaning the excess. Well worth if you ask me.

With glass mounted it was time to reassemble the watch. As expected, the same steps followed in reverse direction were all it took. In hindsight, I should have tested waterproofness before returning mechanism in but I simply forgot. In any case, my son has been wearing it while swimming in pool a few times already thus proving it’s reasonably waterproof. I wouldn’t bet on full 100 m rating though.

With sapphire replacement watch got a new lease on life and completely new look. Polishing the case would probably bring me a step further but I’ll save that for some later time. :)

Here is the list of all parts used in the course of swap. Note I didn’t actually buy all these things specifically for this project (e.g. Calipers, silicon grease, and figer cots I had from before) but I listed them anyhow.

PS: Albeit pronoun I is used all throughout this text, most of work was actually done by my son while I was providing paternal support in form of instructions and warnings. :)