Private Internet Access Client On Encrypted Linux Mint

Upon getting Linux Mint installed, I went ahead with installing Private Internet Access VPN client. All the same motions as usually albeit now with slightly different result - it wouldn’t connect.

Looking at logs ($HOME/.pia_manager/log/openvpn.log) just gave cryptic operation not permitted and no such device errors:

 SIOCSIFADDR: Operation not permitted
 : ERROR while getting interface flags: No such device
 SIOCSIFDSTADDR: Operation not permitted

Quick search on internet brought me to Linux Mint forum where exactly the same problem was described. And familiarity didn’t stop there; author had one other similarity - encrypted home folder - the root cause of the whole problem. Sounded like a perfect fit so I killed PIA client and went with his procedure:

sudo mkdir /home/pia
sudo chown -R $USER:$USER /home/pia
mv ~/.pia_manager /home/pia/.pia_manager
ln -s /home/pia/.pia_manager ~/.pia_manager

However, this didn’t help. Still the same issue in my log files.

So I decided to go with nuclear option. First I killed PIA client (again) and removed PIA completely together with all my modifications:

rm ~/.pia_manager
rm -R /home/pia
sudo rm ~/.local/share/applications/pia_manager.desktop

With all perfectly clean, I decided to start with fresh directory structure, essentially the same as in the original solution:

sudo mkdir -p /home/pia/.pia_manager
sudo chown -R $USER:$USER /home/pia
ln -s /home/pia/.pia_manager ~/.pia_manager

Than I repeated installation of PIA client:

cd ~/Downloads
tar -xzf pia-v72-installer-linux.tar.gz
./pia-v72-installer-linux.sh

And it worked! :)

Installing Wordpress on Linode CentOS

For the purpose of testing new stuff, it is always handy to have Wordpress installation ready. And probably one of the cheapest ways to do so is to use one of virtual server providers - in my case it is Linode.

I won’t be going into specifics of creating server on Linode as it is trivial. Instead, this guide starts at moment your CentOS is installed are you are logged in.

First of all, Linode’s CentOS installation has firewall disabled. As this server will be open to public, enabling firewall is not the worst idea ever:

systemctl start firewalld

systemctl enable firewalld

firewall-cmd --state
 running``

Next you need to install database:

yum install -y mariadb-server

To have database running as a separate user, instead of root, you need to add user=mysql in /etc/my.cnf. You can do that either manually or use the following command to the same effect:

sed -i "/\[mysqld\]/auser=mysql" /etc/my.cnf

Now you can start MariaDB and ensure it starts automatically upon reboot.

systemctl start mariadb

systemctl enable mariadb
  Created symlink from …

I always highly advise securing database a bit. Luckily, there is a script for that. Going with defaults will ensure quite a secure setup.

mysql_secure_installation

A good test for MariaDB setup is creating WordPress database:

mysql -e "CREATE DATABASE ^^wordpress^^;"

mysql -e "GRANT ALL PRIVILEGES ON ^^wordpress^^.* TO ^^'username'^^@'localhost' IDENTIFIED BY '^^password^^';"

mysql -e "FLUSH PRIVILEGES;"

With database sorted out, you can move onto installation of PHP:

yum install -y httpd mod_ssl php php-mysql php-gd

We can start Apache at this time and allow it to start automatically upon reboot:

systemctl start httpd

systemctl enable httpd
 Created symlink from …

With all else installed and assuming you have firewall running, it is time to poke some holes through it:

firewall-cmd --add-service http --permanent
 success

firewall-cmd --add-service https --permanent
 success

firewall-cmd --reload
 success

If all went well, you can now see welcome page when you point your favorite browser to server IP address.

Now finally you get to install WordPress:

yum install -y wget

wget http://wordpress.org/latest.tar.gz -O /var/tmp/wordpress.tgz

tar -xzvf /var/tmp/wordpress.tgz -C /var/www/html/ --strip 1

chown -R apache:apache /var/www/html/

Of course, you will need to create initial file - sample is a good beginning:

cp /var/www/html/wp-config-sample.php /var/www/html/wp-config.php

sed -i "s/database_name_here/^^wordpress^^/" /var/www/html/wp-config.php

sed -i "s/username_here/^^username^^/" /var/www/html/wp-config.php

sed -i "s/password_here/^^password^^/" /var/www/html/wp-config.php

while $(grep -q "put your unique phrase here" /var/www/html/wp-config.php); do
  sed -i "0,/put your unique phrase here/s//$(uuidgen -r)/" /var/www/html/wp-config.php;
done

With wp-config.php fields fully filled, you can go to server’s IP address and follow remaining WordPress installation steps (e.g. site title and similar details).

PS: While this is guide for Linode and CentOS, it should also work with other Linux flavors provided you swap httpd for apache.

Interface Stats

Sometime you just wanna check how many packets and bytes are transferred via network interface. For my Linode NTP server I solved that need using the following script:

#!/bin/bash

INTERFACE=eth0

LINE_COUNT=`tput lines`
LINE=-1

while true
do
    if (( LINE % (LINE_COUNT-1) == 0 ))
    then
        echo "INTERFACE   RX bytes packets     TX bytes packets"
    fi
    LINE=$(( LINE+1 ))

    RX1_BYTES=$RX2_BYTES
    TX1_BYTES=$TX2_BYTES
    RX1_PACKETS=$RX2_PACKETS
    TX1_PACKETS=$TX2_PACKETS
    sleep 1
    RX2_BYTES=`cat /sys/class/net/$INTERFACE/statistics/rx_bytes`
    TX2_BYTES=`cat /sys/class/net/$INTERFACE/statistics/tx_bytes`
    RX2_PACKETS=`cat /sys/class/net/$INTERFACE/statistics/rx_packets`
    TX2_PACKETS=`cat /sys/class/net/$INTERFACE/statistics/tx_packets`

    if [[ "$RX1_BYTES" != "" ]]
    then
        RX_BYTES=$(( RX2_BYTES - RX1_BYTES ))
        TX_BYTES=$(( TX2_BYTES - TX1_BYTES ))
        RX_PACKETS=$(( RX2_PACKETS - RX1_PACKETS ))
        TX_PACKETS=$(( TX2_PACKETS - TX1_PACKETS ))

        printf "%-7s  %'11d %'7d  %'11d %'7d\n" $INTERFACE $RX_BYTES $RX_PACKETS $TX_BYTES $TX_PACKETS
    fi
done

Custom Directory for Apache Logs

On my web server I wanted to use a separate directory for my logs. All I needed was to configure ErrorLog and CustomLog directives and that’s it. Well, I did that only to have following error: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

And no, there weren’t any details worth mentioning in systemctl status httpd.service nor journalctl -xe.

To cut long story short, after a bit of investigation I narrowed the problem to SELinux that is enabled by default on CentOS. Armed with that knowledge, I simply transferred security from default log directory to my desired location:

chcon -R --reference=/etc/httpd/logs/ ^^/var/www/logs/^^

With that simple adjustment, my httpd daemon started and my logs lived happily ever after.

Linode NTP

One of the features I added to Bimil was NTP client support for time-based two-factor authentication. For this I needed NTP server so I turned to ntp.org pool and requested vendor zone. Once zone got approved I suddenly had infinite* amount of NTP servers at my disposal.

So, when I decided to give Linode’s $5 virtual server a try, I didn’t want just to create dummy machine. I also wanted to do something for community. As NTP pool service is one of invisible pillars of Internet-connected devices and I was really happy such service was provided for free to myself, it was easy to decide. I am going to build NTP server.

Creating account on linode was a breeze as it was creating the machine. It was literally, click-next, click-next process. Once I finally logged on to it, the first action was to update system to the latest packages. Surprisingly, on Linode there was literally nothing to do - all was already up to date. Awesome!

yum update -y
 …
 No packages marked for update

By default, Linode’s CentOS installation has firewall disabled. As this server will be open to public, enabling firewall is not the worst idea ever:

systemctl start firewalld

systemctl enable firewalld

firewall-cmd --state
 running

And, while dealing with firewall, you might as well allow NTP through and check if configuration is correct:

firewall-cmd --permanent --add-service ntp
 success

firewall-cmd --reload
 success

firewall-cmd --list-all
 public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: ssh dhcpv6-client ^^ntp^^
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

With firewall configuration completed, you can finally install NTP:

yum install -y ntp

And this brings you to the most involved part of the process. You need to go over available stratum 1 time servers and select between four and seven of them for your devious synchronization purposes. Which servers should you select? As long as they are reasonably close (in the terms of network distance) you will be fine.

Using your favorite editor, you need to adjust /etc/ntp.conf file. Following ntp.org recommendations always worked for me but with a slight adjustment in the form of a separate log file and forcing IPv4 resolving for servers. Quite a few IPv6 capable servers only serve clients over IPv6 and don’t like other servers via the same. I personally use the following configuration (don’t forget to adjust servers names):

driftfile /var/lib/ntp/drift

restrict -4 default kod limited nomodify notrap nopeer noquery
restrict -6 default kod limited nomodify notrap nopeer noquery

restrict -4 127.0.0.1
restrict -6 ::1

server -4 ^^clock.fmt.he.net^^ iburst
server -4 ^^clock.sjc.he.net^^ iburst
server -4 ^^usno.hpl.hp.com^^ iburst
server -4 ^^clepsydra.dec.com^^ iburst
server -4 ^^tick.ucla.edu^^ iburst
server -4 ^^time-a.timefreq.bldrdoc.gov^^ iburst
server -4 ^^time-c.timefreq.bldrdoc.gov^^ iburst

logfile /var/log/ntp.log

With configuration ready, it is the moment of truth - start the NTP daemon and configure its automatic startup upon boot. Don’t forget to disable chrony too:

systemctl start ntpd
systemctl enable ntpd
systemctl disable chronyd

With all up, wait for couple minutes while checking state with ntpstat or ntpq. Forgetting it for hour or two will save you lot of angst :) I consider sync good enough whenever pooling interval goes to 1024s.

watch "ntpq -np ; echo ; ntpstat"
      remote           refid      st t when poll reach   delay   offset  jitter                                                            ``==============================================================================``
 *66.220.9.122    .CDMA.           1 u   41  512  377    2.022    6.680   6.798
 +216.218.254.202 .CDMA.           1 u   77 1024  377    2.127    5.663   6.180
 +204.123.2.72    .GPS.            1 u  257  512  377    4.908    2.753   5.031
 +204.123.2.5     .GPS.            1 u   40  512  377    5.232    5.278   6.052
 +164.67.62.194   .GPS.            1 u  532  512  377    9.978   -0.637   3.795
 +132.163.4.101   .NIST.           1 u  362 1024  377   35.226    5.489   7.610
 +132.163.4.103   .NIST.           1 u  430  512  377   35.148    5.353   7.607
 synchronised to NTP server (66.220.9.122) at stratum 2
   time correct to within 19 ms
   polling server every 1024 s

It will take some time for other servers to “discipline” yours so do be patient. If servers are showing INIT refid for a while, this might indicate a permanent issue (e.g. server might be down) or just something temporary (e.g. server might be overloaded). If server is not reachable for a while, toss it out and select another one from stratum 1 list (followed by systemctl restart ntpd).

I personally gave server an hour or two to get into the shape before proceeding with the final step - adding it to pool. This can be done at ntp.org management pages and it is as easy as simply adding server using either host name or IP address.

After monitoring server for some time and assuming its time is stable, your score will raise and you get to be the part of the collective NTP pool.

* some restrictions apply

Custom Samba Sizing

After reorganizing my ZFS datasets a bit, I suddenly noted I couldn’t copy any file larger than a few MB. A bit of investigation later and I figured why it was so.

My ZFS data sets were as follows:

zfs list
 NAME                            USED  AVAIL  REFER  MOUNTPOINT
 Data                           2.06T   965G    96K  none
 Data/Users                      181G   965G    96K  none
 Data/Users/User1               44.3G  19.7G  2.23G  /Data/Users/User1
 Data/Users/User2               14.7G  49.3G   264K  /Data/Users/User2
 Data/Users/User3                224K  64.0G    96K  /Data/Users/User3

And my Samba share was pointing to /Data/Users/.

Guess what? Path /Data/Users was not pointing to any dataset as my parent dataset for Data/Users was not mounted. Instead it pointed to memory disk md0 which had just a few MB free. Samba doesn’t check full path for disk size but only its root share.

The easiest way to workaround this would be to simply mount parent dataset. But why go for easy?

A bit more complicated solution is getting Samba to use custom script to determine free space. We can then use this script to return available disk space for our parent dataset instead of built-in samba calculation.

To do this, we first create script /myScripts/sambaDiskFree:

#!/bin/sh
DATASET=`pwd | cut -c2-`
zfs list -H -p -o available,used $DATASET | awk '{print $1+$2 " " $1}'

This script will check current directory, map its name to dataset (in my case it is as easy as stripping first slash character) and return two numbers. First is total disk space, followed by available diskspace - both in bytes.

Once script is saved and marked as executable (chmod +x), we just need to reference it in Services > CIFS/SMB > Settings under Additional parameters:

dfree command = /myScripts/sambaDiskFree

This will tell Samba to use our script for disk space determinations.

My Backup ZFS Machine (Correction)

Before going on vacation I finished setting up my new ZFS backup machine, initialized first replication, and happily gone to see the big hole.

When I remotely connected to my main machine a few days later, I’ve found my sync command has failed before finishing. Also I couldn’t connect to my backup server. Well, that was unfortunate but I had enough foresight to get it connected via smart plug so I did power-off/power-on dance. My system booted and I restarted replication. I checked on it a few days later, to find it stuck again. Rinse-repeat. And the next day too. And the one after that…

Why? I have not idea as I was connected only remotely and I literally came home on the last day when I could return it to Amazon. Since I did raise a case with Supermicro in regards to video card error (5 beeps) which seemed hardware related my suspicions were definitely pointing only in direction of motherboard issue. I know memory was fine as I tested it thoroughly in another machine and power supply is happily working even now.

For my peace of mind I needed something that would allow me not only to reboot machine but to also access its screen and keyboard directly without any OS involvement. Variants are known under different names and slightly different execution. Whether it is KVM, iLO, AMT, or IPMI.

So I decided to upgrade to more manageable Supermicro A1SRi-2558F. With its C2558 processor (4 cores) and quad LAN it was definitely an overkill for my purpose but it was the cheapest IPMI-capable board I could find at $225 (compared to $150 for X10SBA-L). Unfortunately for my budget its ECC requirement meant adding another $35 for ECC RAM. And of course, different layout made my 6" right-angle SATA cable useless so now they decorate my drawer.

Board itself is really full of stuff with total of six USB ports (four are USB 3.0), one of which was even soldered on motherboard for internal USB needs. Having four gigabit ports is probably useless as Atom is unlikely to be able to drive them all at full speed but I guess it does allow for more relaxed network configuration. Moreover two SATA3 and four SATA2 just scream NAS. And rear bracket on my 1U case fits rear IO perfectly. Frankly, the only thing missing is HDMI albeit IPMI greatly reduces chance of ever needing it.

Total difference in system cost was $100 and it gave me a rock-solid experience (hasn’t crashed a single time in more than a month). Here is updated shopping list:

Setting Up Private Internet Access on Mint, 2018 Edition

I have already written about getting Private Internet Access running on Linux Mint back in 2016. Main reason is that with Linux Mint 18, not all DNS changes are properly propagated.

As OpenVPN client is installed by default these days, we only need to download PIA’s OpenVPN configuration files. More careful ones will notice these files are slightly different than recommended default. These have VPN server IP instead of DNS name. While this might cause long term issues if that IP ever changes, it does help a lot with firewall setup as we won’t need to poke a hole for DNS over our eth0 adapter.

From downloaded archive select .ovpn file with desired destination (usually going with one closest to you gives the best results) and also get both .crt and .pem file. Copy them all to your desktop and we’ll use them later for setup. Yes, you can use any other directory too - this is just one I prefer.

With this done we can go into configuring VPN from Terminal window (replacing username and password with actual values):

sudo mv ~/Desktop/*.crt /etc/openvpn/
sudo mv ~/Desktop/*.pem /etc/openvpn/
sudo mv ~/Desktop/*.ovpn /etc/openvpn/client.conf

sudo sed -i "s*ca *ca /etc/openvpn/*" /etc/openvpn/client.conf
sudo sed -i "s*crl-verify *crl-verify /etc/openvpn/*" /etc/openvpn/client.conf

sudo echo "auth-user-pass /etc/openvpn/client.login" >> /etc/openvpn/client.conf
sudo echo "mssfix 1400" >> /etc/openvpn/client.conf
sudo echo "dhcp-option DNS 209.222.18.218" >> /etc/openvpn/client.conf
sudo echo "dhcp-option DNS 209.222.18.222" >> /etc/openvpn/client.conf
sudo echo "script-security 2" >> /etc/openvpn/client.conf
sudo echo "up /etc/openvpn/update-resolv-conf" >> /etc/openvpn/client.conf
sudo echo "down /etc/openvpn/update-resolv-conf" >> /etc/openvpn/client.conf

unset HISTFILE
echo '^^username^^' | sudo tee -a /etc/openvpn/client.login
echo '^^password^^' | sudo tee -a /etc/openvpn/client.login
sudo chmod 500 /etc/openvpn/client.login

Now we can test our VPN connection:

sudo openvpn --config /etc/openvpn/client.conf

Assuming that this last step ended with Initialization Sequence Completed, we just need to verify whether this connection is actually used and I’ve found whatismyipaddress.com quite helpful here. Just check if IP detected there is different then IP you usually get without VPN.

Stop the test connection using Ctrl+C so we can configure automatic startup and test it.

echo "AUTOSTART=all" | sudo tee -a /etc/default/openvpn
sudo reboot

Once computer has booted and you are satisfied with VPN configuration, you can think about firewall and disabling default interface when VPN is not active. This means allowing traffic only on tun0 interface (VPN) and allowing only port 1198.

sudo ufw reset
sudo ufw default deny incoming
sudo ufw default deny outgoing
sudo ufw allow out on tun0
sudo ufw allow out on `route | grep '^default' | grep -v "tun0$" | grep -o '[^ ]*$'` proto udp to `cat /etc/openvpn/client.conf | grep "^remote " | grep -o ' [^ ]* '` port 1198
sudo ufw enable

Assuming all went well, VPN should be happily running.

Adding Mirrored Disk to Existing ZFS Pool

Great thing about ZFS is that even with a single disk you get some benefits - data integrity being the most important. And all ZFS commands work perfectly well, for example status:

zpool status
   pool: Data.Tertiary``
  state: ONLINE``
 config:``
         NAME                   STATE     READ WRITE CKSUM``
         Data.Tertiary          ONLINE       0     0     0``
           diskid/DISK-XXX.eli  ONLINE       0     0     0``

However, what if one disk is not sufficient any more? It is clear zpool add can be used to create striped pool for higher speeds. And it is clear we can add another device to make a three way mirror. But what if we want to convert solo disk to mirror configuration?

Well, in that case we can get creative with attach command giving it both disks as an argument:

zpool attach Data.Tertiary ^^diskid/DISK-XXX.eli^^ ^^diskid/DISK-YYY.eli^^

After a few seconds, our mirror is created with all our data intact:

zpool status
   pool: Data.Tertiary
  state: ONLINE
 status: One or more devices is currently being resilvered.  The pool will
         continue to function, possibly in a degraded state.
 action: Wait for the resilver to complete.
 config:
         NAME                     STATE     READ WRITE CKSUM
         Data.Tertiary            ONLINE       0     0     0
           mirror-0               ONLINE       0     0     0
             diskid/DISK-XXX.eli  ONLINE       0     0     0
             diskid/DISK-YYY.eli  ONLINE       0     0     0  (resilvering)

PS: Yes, I use encrypted disks from /dev/diskid/ as I used them in previous ZFS examples. If you want plain devices, just use ada0 and companions instead.

Stuck at System Initializing

After a routine SATA cable change, my Supermicro A1SRi-2558F motherboard simply wouldn’t boot. From its fortunate IPMI interface I saw it was hanging at “System initializing…” with code 19 being prominently in bottom right corner.

As only thing I did was to replace SATA cable, I first returned the old one to be greeted with the same issue. It took retracing my steps to I notice I replaced memory module into the wrong slot (had to remove it to more easily reach SATA connector latch). Since this motherboard does require DIMMs to be fitted in certain order, error was clearly mine but two things confuse me.

First one is why I haven’t got beep notification that something is wrong with memory. This board does beep at you if there is no memory (5 short, 1 long) but there is not a beep if memory is incorrectly installed. Why?

Secondly, why the heck IPMI doesn’t include more details about system status - dare I say useful error log? If memory is wrongly installed, I should be able to see an error message in log. With this I am scared how ECC experience is going to look like - will it just simply fail without a message?

In any case, reinstalling the memory module at the correct spot did the trick and board happily worked ever since. :)