Dev Ups

Published 2022-02-14 in ci-cd

Securing a LAMP VPS

Whilst I refer to a VPS throughout, I developed this procedure first on a local VM. The biggest difference was in obtaining SSL certificates.

Access to configure

To get a shell on the server, we copy our public key manually to the target node's ~/.ssh/authorized_keys file.

Bad actors have tried my SSH for nearly a week. I can learn how far they got using:

sudo grep 'login keyring' /var/log/auth.log
# Or better yet:

The key enables access without a password. The trouble is we are still root. We should make a new user for use by Ansible. We can script creating the user but want to avoid having to enter and record a password. We do so by echoing our public key to ~/.ssh/authorized_keys:

useradd -m -d /home/$username -s /bin/bash $username
mkdir /home/$username/.ssh
chmod 700 /home/$username/.ssh
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC37pjDQLthKgTJnysv7uMf/canNTUqUbbA7EepgyXrE ansible control VM 2021-12-12" >\
chmod 600 /home/$username/.ssh/authorized_keys
chown -R $username:$username /home/$username/.ssh
sed -i "s/^# *PubkeyAuthentication.*$/PubkeyAuthentication yes/" /etc/ssh/sshd_config

cp /etc/sudoers /root/sudoers.bak
echo "$username ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/$username

#userdel -r -f $username  # when we're done with experimental users.

I created the key using:

ssh-keygen -t ed25519 -C "ansible control VM `date +%F`"

Once I was satisfied I could log in and sudo as the new user I disabled root logins:

sed -i "s/^#* *PermitRootLogin.*$/PermitRootLogin no/" /etc/ssh/sshd_config
/etc/init.d/ssh restart

Potential problems

Copy and pasting from Windows meant I needed to insert a line break at the end of the private key.

POSIX permissions mean nothing in Windows. The following is from Git Bash for Windows:

$ ll PycharmProjects/vagrant_controller/.ssh -a
total 10
drwxr-xr-x 1 silverb 197121   0 Dec 13 07:00 ./
drwxr-xr-x 1 silverb 197121   0 Dec 13 07:15 ../
-rw-r--r-- 1 silverb 197121 419 Dec 13 07:00 id_ed25519
-rw-r--r-- 1 silverb 197121 110 Dec 13 00:39


If your VPS is using OpenVZ, this won't work, and may be skipped.

Let's set up the UFW firewall:

sudo ufw allow ssh
#sudo ufw allow OpenSSH
# Rule by app (sudo ufw app list) name not proto/port.
# Default should already be these (cat /etc/default/ufw):
sudo ufw default allow outgoing
sudo ufw default deny incoming
sudo ufw show added

Diagnostic commands:

sudo ufw app list
sudo ufw enable
sudo ufw status numbered
sudo ufw status verbose
sudo systemctl status ufw.service

Let's look at encryption in transit. We must install the server.

LAMP server

Here are the commands to install the WordPress-ready server, followed by a few WordPress specific commands:

sudo apt-get update
# The following packages took 59.9MB of downloading and 369MB of disk space:
sudo apt-get install -y \
      apache2 \
      ghostscript \
      libapache2-mod-php \
      mysql-server \
      php \
      php-bcmath \
      php-curl \
      php-imagick \
      php-intl \
      php-json \
      php-mbstring \
      php-mysql \
      php-xml \
sudo mkdir -p /srv/www
sudo chown www-data: /srv/www
# curl | sudo -u www-data tar zx -C /srv/www
curl | sudo -u www-data tar zx -C /srv/www was the latest, British English, WordPress at the time of writing. Other versions are available and will work. The final command pipes the download into a privileged untar process which assumes the identity of www-data, in keeping with our folder hierarchy permissions created immediately before.

This is creating the DocumentRoot we shall be referring to extensively during configuration of WordPress.

SSL/TLS certificates

Now we have Apache, we can make it use a certificate. In Vagrant this will need to be self-signed. Certification depends on chains of trust. With self-signing we must be in close enough touch with our end-users to securely deliver them our public certificate, that they can trust it. It's almost useless on the public Internet.

Brian Boucheron helped to introduce me to openssl:

argument Description
req sub command indicating X-509 CSR management. Has own man page
-x-509 added to sub cmd to create self-signed cert instead of CSR
-nodes no password needed
-days how long certificate is valid for (default=30)
-newkey includes key generation (rsa:4096) as part of openssl
-keyout specifies where the key will be saved
-out where the certificate will be saved
-subj this is key to automating, we pass our answers here
-new this is redundant (?) as it's being implied by -x509
sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 \
 -keyout /etc/ssl/private/apache-selfsigned.key \
 -out /etc/ssl/certs/apache-selfsigned.crt

This instigates interactive prompts. According to Brian Boucheron:

The most important line is the one that requests the Common Name (e.g. server FQDN or YOUR name). You need to enter the domain name associated with your server or, more likely, your server’s public IP address.

This can be automated using the subject (subj) parameter, eg:

-subj "/C=GB/ST=Test/L=Test/O=NonInc./OU=IT/"

In which the fields mean:

Abbrev Meaning
ST Test State
L Test Locality
O Organisation Name
OU Organisation Unit Name
CN Common Name (FQDN)

While we're exploring adding parameters to openssl, modern browsers, and libraries, require a subject alt name, which openssl provides, via -addext:

-addext ",,IP:"

For testing purposes, without assigning domain names, this can be simplified to -addext "subjectAltName=IP:". We don't need all the DNS fields.

Modern elliptic curve encryption offers advantages over RSA and is widely supported. To specify one type of curve (public (i.e., asymmetric) key) use:

-pkeyopt ec_paramgen_curve:secp384r1

In context:

openssl req -new -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -days 365 -nodes -x509 \
    -subj "/CN=localhost" \
    -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt

Investigating existing certificates

Given the iterations and examples leading up to the previous openssl invocation it is desirable to be able to verify the certificate from the client. Determining the FQDN (common name) of the cert is a start.

The end-to-end test would involve viewing the certificate in a web browser. At https://localhost:8541/, or whatever port number mapped to 443. MS Edge has a red exclamation mark left of the URL which can be used to view the certificate:

MS Edge detailing why

This was commonly referred to as "clicking the padlock" on other browsers. That's the end-to-end test. The server will contain hundreds of other certificates. The number of private keys is far fewer; Apache uses ssl-cert-snakeoil.pem by default. A fitting name.

/etc/apache2/sites-enabled/ links to /etc/apache2/sites-available/ for any (virtual host) files that are active (enabled), thus narrowing our search for active certificates.

Understanding of the virtual host configuration file is useful to trace the certificate "end-to-end".

browsing to is handled by apach vhosts

SSLCertificateFile      /etc/ssl/certs/apache-selfsigned.crt
SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key

We will add these to our vhosts.conf later. Our vhosts file will be in /etc/apache2/sites-available/.

Server side tests

Here are the commands to examine and test the certificate and key on the server:

cd /etc/ssl
# View certificate:
openssl x509 -text -in certs/apache-selfsigned.crt -noout

# Compare public keys (use of hashes is optional for compact keys):
sudo openssl pkey -pubout -in private/apache-selfsigned.key | openssl sha256
openssl x509 -pubkey -in certs/apache-selfsigned.crt -noout | openssl sha256

sudo apachectl configtest
# sudo apachectl -t  # the shorthand of configtest

The configtest command has some common outcomes:

  1. syntax ok, this alone is optimal, anything else feels like a complaint.
  2. Without syntax ok we may expect a specific syntax error, file, and line number.
  3. AH00558: apache2: Could not reliably determine the server... Not great but just a warning. It still works end to end.

SSL config

/etc/apache2/conf-available/ssl-params.conf is the configuration snippet we provide during provisioning.

Updated ssl-params.conf may be obtained from, courtesy of Remy van Elst. He has since archived this code, and invited others to host it.

The latest given Apache2 requirement, 2.4.36 dates from 2018. I couldn't trace it to a current maintainer, but I will repost to do my bit:

# Requires Apache 2.4.36 & OpenSSL 1.1.1
SSLProtocol -all +TLSv1.3 +TLSv1.2
SSLOpenSSLConfCmd Curves X25519:secp521r1:secp384r1:prime256v1
# Older versions
# SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder On
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
Header always set X-Frame-Options DENY
Header always set X-Content-Type-Options nosniff
# Requires Apache >= 2.4
SSLCompression off
SSLUseStapling on
SSLStaplingCache "shmcb:logs/stapling-cache(150000)"
# Requires Apache >= 2.4.11
SSLSessionTickets Off

Mozilla may prove more future proof. I've had success with it after removing the virtual host blocks that I have already created in sites-available.

Next steps

This was preparatory work for installing a web server.

Eventually I'll post about using Nginx instead of Apache. The VPS I'm using here came with Apache preinstalled. The next article in this series is about installing WordPress.