Configuring Google's SMTP Via Smtpmail on Linode CentOS

When you install Wordpress on Linode, one thing that’s not supported out of box is mail - php requires sendmail installed and running.

Configurations might differ depending on the exact use case, but for my Wordpress I wanted to use Google’s SNMP server. While guides are plentiful, most of information is a bit obsolete - whether it is in regards to package name or exact configuration. So I went my own way…

To get us going, we need to install a few packages related to sendmail functionality and allow the same in SELinux (if enforced):

yum install -y sendmail sendmail-cf cyrus-sasl-plain cyrus-sasl-md5
setsebool -P httpd_can_sendmail on

First thing to prepare is file containing our authentication details for Google’s server. I will assume here that login you usually use is relay@gmail.com and password is password. Of course, these must be substituted for correct values:

mkdir -p -m 700 /etc/mail/authinfo
echo 'AuthInfo: "U:root" "I:^^relay@gmail.com^^" "P:^^password^^"' > /etc/mail/authinfo/gmail
makemap hash /etc/mail/authinfo/gmail < /etc/mail/authinfo/gmail

To /etc/mail/sendmail.mc we need to add the following lines just ABOVE the first MAILER line.

…
define(`SMART_HOST',`[smtp.gmail.com]')dnl
define(`RELAY_MAILER_ARGS', `TCP $h 587')dnl
define(`ESMTP_MAILER_ARGS', `TCP $h 587')dnl
define(`confAUTH_OPTIONS', `A p')dnl
define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confCACERT', `/etc/pki/tls/certs/ca-bundle.trust.crt')dnl
TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
FEATURE(`authinfo',`hash -o /etc/mail/authinfo/gmail.db')dnl
``…``

Once configuration has been updated, we can proceed with “compiling” that new configuration and starting the daemon for the first time.

make -C /etc/mail
systemctl start sendmail

The easiest test is sending an e-mail via command line:

echo "Subject: Test via sendmail" | sendmail -v ^^youremail@example.com^^

If there are issues, you can always check journal for startup errors:

journalctl -xe

The most common error is “available mechanisms do not fulfill requirements” and that signals Cyrus SASL plugins are not installed for MD5 and PLAIN methods. Make sure cyrus-sasl-plain and cyrus-sasl-md5 packages are installed.

Lastly, if sendmail does work but it is very slow, add your hostname (output of hostname command) to the end of localhost (IPv4 and IPv6) entries in /etc/hosts file.

Sorting \"Dot\" Files

As I got Cent OS 7.4 running, a bit strange thing happened. When I ran usual ll (alias to ls -lA), I got a slightly unexpected result:

ll
 drwxrwx--- 4 apache apache  4096 Dec 24 06:50 download
 -rw-rw---- 1 apache apache  5430 Dec 23 08:06 favicon.ico
 -rw-rw---- 1 apache apache 12300 Dec 26 02:25 .htaccess
 -rw-rw---- 1 apache apache   460 Dec 23 08:06 index.php
 -rw-rw---- 1 apache apache   117 Dec 23 20:39 robots.txt
 drwxrwx--- 2 apache apache  4096 Dec 26 01:44 .well-known
 drwxrwx--- 5 apache apache  4096 Dec 23 17:32 wordpress

Can you spot the issue?

Yep, Cent OS got a bit (too) smart so sorting ignores the starting dot and gets those files too in the alphabetic order. Those used to dot files on the top - though luck.

Well, it’s possible to “correct” this behavior using the slightly different alias in .bashrc:

alias ll='LC_COLLATE=C ls -lA'

This gives a (properly) sorted output:

ll
 -rw-rw---- 1 apache apache 12300 Dec 26 02:25 .htaccess
 drwxrwx--- 2 apache apache  4096 Dec 26 01:44 .well-known
 drwxrwx--- 4 apache apache  4096 Dec 24 06:50 download
 -rw-rw---- 1 apache apache  5430 Dec 23 08:06 favicon.ico
 -rw-rw---- 1 apache apache   460 Dec 23 08:06 index.php
 -rw-rw---- 1 apache apache   117 Dec 23 20:39 robots.txt
 drwxrwx--- 5 apache apache  4096 Dec 23 17:32 wordpress

Requiring Authentication For All But One File

As I planned move of my site to Linode, first I needed a place to test. It was easy enough to create test domain and fill it with migrated data but I didn’t want Google (or any other bot) to index it. The easiest way to do so was to require authentication. In Apache configuration that can be done using Directory directive:

<Directory "/var/www/html">
    AuthType Basic
    AuthUserFile "/var/www/.htpasswd"
    Require valid-user
</Directory>

However, this also means that my robots.txt with disallow statements was also forbidden. What I really wanted was to allow only access to robots.txt while forbidding everything else.

A bit of modification later, this is what I came up with:

<Directory "/var/www/html">
    AuthType Basic
    AuthUserFile "/var/www/.htpasswd"
    Require valid-user
    <Files "robots.txt">
        Allow from all
        Satisfy Any
    </Files>
</Directory>

Tailing Two Files

As I got my web server running, it came to me to track Apache logs for potential issues. My idea was to have a base script that would, on a single screen, show both access and error logs in green/yellow/red pattern depending on HTTP status and error severity. And I didn’t want to see the whole log - I wanted to keep information at minimum - just enough to determine if things are going good or bad. If I see something suspicious, I can always check full logs.

Error log is easy enough but parsing access log in the common log format (aka NCSA) is annoyingly difficult due to its “interesting” choice of delimiters.

Just looks at this example line:

108.162.245.230 - - [26/Dec/2017:01:16:45 +0000] "GET /download/bimil231.exe HTTP/1.1" 200 1024176 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"

First three entries are space separated - easy enough. Then comes date in probably the craziest format one could fine and enclosed in square brackets. Then we have request line in quotes, followed by a bit more space-separated values. And we finish with a few quoted values again. Command-line parsing was definitely not in mind of whoever “designed” this.

With Apache you can of course customize format for logging - but guess what? While you can make something that works better with command-line tools, you will lose a plethora of tools that already work with NCSA format - most notably Webalizer. It might be a bad choice for command line, but it’s the standard regardless.

And extreme flexibility of Linux tools also means you can do trickery to parse fields even when you deal with something as mangled as NCSA.

After a bit of trial and error, my final product was the script looking a bit like this:

#!/bin/bash

LOG_DIRECTORY="/var/www"

trap 'kill $(jobs -p)' EXIT

tail -Fn0 $LOG_DIRECTORY/apache_access.log | gawk '
  BEGIN { FPAT="([^ ]+)|(\"[^\"]+\")|(\\[[^\\]]+\\])" }
  {
    code=$6
    request=$5

    ansi="0"
    if (code==200 || code==206 || code==303 || code==304) {
      ansi="32;1"
    } else if (code==301 || code==302 || code==307) {
      ansi="33;1"
    } else if (code==400 || code==401 || code==403 || code==404 || code==500) {
      ansi="31;1"
    }
    printf "%c[%sm%s%c[0m\n", 27, ansi, code " " request, 27
  }
' &

tail -Fn0 $LOG_DIRECTORY/apache_error.log | gawk '
  BEGIN { FPAT="([^ ]+)|(\"[^\"]+\")|(\\[[^\\]]+\\])" }
  {
    level=$2
    text=$5 " " $6 " " $7 " " $8 " " $9 " " $10 " " $11 " " $12 " " $13 " " $14 " " $15 " " $16

    ansi="0"
    if (level~/info/) {
      ansi="32"
    } else if (level~/warn/ || level~/notice/) {
      ansi="33"
    } else if (level~/emerg/ || level~/alert/ || level~/crit/ || level~/error/) {
      ansi="31"
    }
    printf "%c[%sm%s%c[0m\n", 27, ansi, level " " text, 27
  }
' &

wait

Script tails both error and access logs, waiting for Ctrl+C. Upon exit, it will kill spawned jobs via trap.

For access log, gawk script will check status code and color entries accordingly. Green color is for 200 OK, 206 Partial Content, 303 See Other, and 304 Not Modified; yellow for 301 Moved Permanently, 302 Found, and 307 Temporary Redirect; red for 400 Bad Request, 401 Unauthorized, 403 Forbidden, and 404 Not Found. All other codes will remain default/gray. Only code and first request line will be printed.

For error log, gawk script will check only error level. Green color will be used for Info; yellow color is for Warn and Notice; red is for Emerg, Alert, Crit, and Error. All other (essentially debug and trace) will remain default/gray. Printout will consist just of error level and first 12 words.

This script will not only shorten quite long error and access log lines to their most essential parts, but coloring will enable one to see the most important issues at a glance - even when lines are flying around. Additionally, having them interleaved lends itself nicely to a single screen monitoring station.

[2018-02-09: If you are running this via SSH on remote server, don’t forget to use -t for proper cleanup after SSH connection fails.]

Creating ISO From the Command Line

Creating read-only archives is often beneficial. This is especially so when we are dealing with something standard across many system. And rarely you will find anything more standard than CD/DVD .iso files. You can mount it on both Windows 10 and Linux without any issues.

There are quite a few programs that will allow you to create .iso files but they are often overflowing with ads. Fortunately every Linux distribution comes with a small tool capable of the same without any extra annoyances. That tool is called [mkisofs](https://linux.die.net/man/8/mkisofs).

Basic syntax is easy:

mkisofs -input-charset -utf8 -udf -V "My Label" -o MyDVD.iso ~/MyData/

Setting input charset is essentially only needed to suppress warning. UTF-8 is default anyhow and in 99% cases exactly what you want.

Using UDF as output format enables a bit more flexible file and directory naming rules. Standard ISO 9660 format (even when using level 3) is so full of restrictions making it annoying at best- most notable being support for only uppercase file names. UDF allows Unicode file names up to 255 characters in length and has no limit to directory depth.

Lastly, DVD label is always a nice thing to have.

Solving \"Failed to Mount Windows Share\"

Most of the time I access my home NAS via samba shares. For increased security and performance I force it to use SMB v3 protocol. And therein lies the issue.

Whenever I tried to access my NAS from Linux Mint machine using Caja browser, I would get the same error: “Failed to mount Windows share: Connection timed out.” And it wasn’t connectivity issues as everything would work if I dropped my NAS to SMB v2. And it wasn’t unsupported feature either as Linux supports SMB3 for a while now.

It was just a case of a bit unfortunate default configuration. Albeit man pages tell client max protocol is SMB3, something simply doesn’t click. However, if one manually specifies only SMB3 is to be used, everything starts magically working.

Configuring it is easy; in /etc/samba/smb.conf, within [global], one needs to add

client min protocol = SMB3
client max protocol = SMB3

Alternatively, this can also be done with the following one-liner:

sudo sed -i "/\\[global\\]/a client min protocol = SMB3\nclient max protocol = SMB3" /etc/samba/smb.conf

Once these settings are in, share is accessible.

Private Internet Access Client On Encrypted Linux Mint

Upon getting Linux Mint installed, I went ahead with installing Private Internet Access VPN client. All the same motions as usually albeit now with slightly different result - it wouldn’t connect.

Looking at logs ($HOME/.pia_manager/log/openvpn.log) just gave cryptic operation not permitted and no such device errors:

 SIOCSIFADDR: Operation not permitted
 : ERROR while getting interface flags: No such device
 SIOCSIFDSTADDR: Operation not permitted

Quick search on internet brought me to Linux Mint forum where exactly the same problem was described. And familiarity didn’t stop there; author had one other similarity - encrypted home folder - the root cause of the whole problem. Sounded like a perfect fit so I killed PIA client and went with his procedure:

sudo mkdir /home/pia
sudo chown -R $USER:$USER /home/pia
mv ~/.pia_manager /home/pia/.pia_manager
ln -s /home/pia/.pia_manager ~/.pia_manager

However, this didn’t help. Still the same issue in my log files.

So I decided to go with nuclear option. First I killed PIA client (again) and removed PIA completely together with all my modifications:

rm ~/.pia_manager
rm -R /home/pia
sudo rm ~/.local/share/applications/pia_manager.desktop

With all perfectly clean, I decided to start with fresh directory structure, essentially the same as in the original solution:

sudo mkdir -p /home/pia/.pia_manager
sudo chown -R $USER:$USER /home/pia
ln -s /home/pia/.pia_manager ~/.pia_manager

Than I repeated installation of PIA client:

cd ~/Downloads
tar -xzf pia-v72-installer-linux.tar.gz
./pia-v72-installer-linux.sh

And it worked! :)

Installing Wordpress on Linode CentOS

For the purpose of testing new stuff, it is always handy to have Wordpress installation ready. And probably one of the cheapest ways to do so is to use one of virtual server providers - in my case it is Linode.

I won’t be going into specifics of creating server on Linode as it is trivial. Instead, this guide starts at moment your CentOS is installed are you are logged in.

First of all, Linode’s CentOS installation has firewall disabled. As this server will be open to public, enabling firewall is not the worst idea ever:

systemctl start firewalld

systemctl enable firewalld

firewall-cmd --state
 running``

Next you need to install database:

yum install -y mariadb-server

To have database running as a separate user, instead of root, you need to add user=mysql in /etc/my.cnf. You can do that either manually or use the following command to the same effect:

sed -i "/\[mysqld\]/auser=mysql" /etc/my.cnf

Now you can start MariaDB and ensure it starts automatically upon reboot.

systemctl start mariadb

systemctl enable mariadb
  Created symlink from …

I always highly advise securing database a bit. Luckily, there is a script for that. Going with defaults will ensure quite a secure setup.

mysql_secure_installation

A good test for MariaDB setup is creating WordPress database:

mysql -e "CREATE DATABASE ^^wordpress^^;"

mysql -e "GRANT ALL PRIVILEGES ON ^^wordpress^^.* TO ^^'username'^^@'localhost' IDENTIFIED BY '^^password^^';"

mysql -e "FLUSH PRIVILEGES;"

With database sorted out, you can move onto installation of PHP:

yum install -y httpd mod_ssl php php-mysql php-gd

We can start Apache at this time and allow it to start automatically upon reboot:

systemctl start httpd

systemctl enable httpd
 Created symlink from …

With all else installed and assuming you have firewall running, it is time to poke some holes through it:

firewall-cmd --add-service http --permanent
 success

firewall-cmd --add-service https --permanent
 success

firewall-cmd --reload
 success

If all went well, you can now see welcome page when you point your favorite browser to server IP address.

Now finally you get to install WordPress:

yum install -y wget

wget http://wordpress.org/latest.tar.gz -O /var/tmp/wordpress.tgz

tar -xzvf /var/tmp/wordpress.tgz -C /var/www/html/ --strip 1

chown -R apache:apache /var/www/html/

Of course, you will need to create initial file - sample is a good beginning:

cp /var/www/html/wp-config-sample.php /var/www/html/wp-config.php

sed -i "s/database_name_here/^^wordpress^^/" /var/www/html/wp-config.php

sed -i "s/username_here/^^username^^/" /var/www/html/wp-config.php

sed -i "s/password_here/^^password^^/" /var/www/html/wp-config.php

while $(grep -q "put your unique phrase here" /var/www/html/wp-config.php); do
  sed -i "0,/put your unique phrase here/s//$(uuidgen -r)/" /var/www/html/wp-config.php;
done

With wp-config.php fields fully filled, you can go to server’s IP address and follow remaining WordPress installation steps (e.g. site title and similar details).

PS: While this is guide for Linode and CentOS, it should also work with other Linux flavors provided you swap httpd for apache.

Interface Stats

Sometime you just wanna check how many packets and bytes are transferred via network interface. For my Linode NTP server I solved that need using the following script:

#!/bin/bash

INTERFACE=eth0

LINE_COUNT=`tput lines`
LINE=-1

while true
do
    if (( LINE % (LINE_COUNT-1) == 0 ))
    then
        echo "INTERFACE   RX bytes packets     TX bytes packets"
    fi
    LINE=$(( LINE+1 ))

    RX1_BYTES=$RX2_BYTES
    TX1_BYTES=$TX2_BYTES
    RX1_PACKETS=$RX2_PACKETS
    TX1_PACKETS=$TX2_PACKETS
    sleep 1
    RX2_BYTES=`cat /sys/class/net/$INTERFACE/statistics/rx_bytes`
    TX2_BYTES=`cat /sys/class/net/$INTERFACE/statistics/tx_bytes`
    RX2_PACKETS=`cat /sys/class/net/$INTERFACE/statistics/rx_packets`
    TX2_PACKETS=`cat /sys/class/net/$INTERFACE/statistics/tx_packets`

    if [[ "$RX1_BYTES" != "" ]]
    then
        RX_BYTES=$(( RX2_BYTES - RX1_BYTES ))
        TX_BYTES=$(( TX2_BYTES - TX1_BYTES ))
        RX_PACKETS=$(( RX2_PACKETS - RX1_PACKETS ))
        TX_PACKETS=$(( TX2_PACKETS - TX1_PACKETS ))

        printf "%-7s  %'11d %'7d  %'11d %'7d\n" $INTERFACE $RX_BYTES $RX_PACKETS $TX_BYTES $TX_PACKETS
    fi
done

Custom Directory for Apache Logs

On my web server I wanted to use a separate directory for my logs. All I needed was to configure ErrorLog and CustomLog directives and that’s it. Well, I did that only to have following error: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

And no, there weren’t any details worth mentioning in systemctl status httpd.service nor journalctl -xe.

To cut long story short, after a bit of investigation I narrowed the problem to SELinux that is enabled by default on CentOS. Armed with that knowledge, I simply transferred security from default log directory to my desired location:

chcon -R --reference=/etc/httpd/logs/ ^^/var/www/logs/^^

With that simple adjustment, my httpd daemon started and my logs lived happily ever after.