security-onion-letsencrypt.md

Automate LetsEncrypt certificate deployment in SecurityOnion

Prerequisites

• A domain name
• A DNS provider that works with Certbot. I'm using Cloudflare.
• An API token for the DNS provider that is scoped to allow TXT record creation.
• SecurityOnion 2.4 with sudo privileges and SSH access.

Turn on user certs in Security Onion (do this only once)

1. Administration -> Configuration
2. Options -> Show all configurable settings, including advanced settings
3. Filter "cert"
4. nginx -> ssl -> Replace Default Cert (ssl/tls in SO <= 2.4.80)
5. Replace deafult cert: set to True
6. Don't replace the key files yet - we're going to automate this!

Setup LetsEncrypt

Note: some of these commands may seem unnecessary, e.g. the symlinks. Don't skip them. SecurityOnion 2.4 is based on Oracle Linux, which has some quirks.

1. Log in as your admin user and su to root.

   sudo su

2. Install snapd.

   yum install epel-release
   yum install -y snapd
   ln -s /var/lib/snapd/snap /snap

   Check to ensure that the snapd service is running.

   sudo systemctl status snapd

   If it is not running, start the service.

   sudo systemctl start snapd

3. Install certbot.

   snap install --classic certbot
   ln -s /snap/bin/certbot /usr/bin/certbot

4. Install certbot Cloudflare DNS plugin.

   snap set certbot trust-plugin-with-root=ok
   snap install certbot-dns-cloudflare

5. Initialize the /etc/letsencrypt directory by running certbot without any arguments. Ignore the errors.

   certbot

6. Add Cloudflare DNS API token.

   TOKEN={enter your Cloudflare API token here}
   echo dns_cloudflare_api_token=$TOKEN > /etc/letsencrypt/cloudflare.ini
   chmod 400 /etc/letsencrypt/cloudflare.ini

7. Request the certificate. Replace your.fqdn.xyz with the FQDN of your SecurityOnion server.

   FQDN=your.fqdn.xyz
   certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini -d $FQDN

8. Copy the cert and key to SecurityOnion's nginx salt config.

   cp -f /etc/letsencrypt/live/$FQDN/privkey.pem /opt/so/saltstack/local/salt/nginx/ssl/ssl.key
   chmod 644 /opt/so/saltstack/local/salt/nginx/ssl/ssl.key
   cp -f /etc/letsencrypt/live/$FQDN/fullchain.pem /opt/so/saltstack/local/salt/nginx/ssl/ssl.crt
   chmod 640 /opt/so/saltstack/local/salt/nginx/ssl/ssl.crt

9. Restart nginx.

   so-nginx-restart

   Clear your browser cache (or use an incognito window) and visit your SO admin page to ensure that the new certificate is presented.

10. Create a post-hook to perform this each time certs are renewed.

    cat <<EOF > /etc/letsencrypt/copy_to_nginx.sh
    cp -f /etc/letsencrypt/live/$FQDN/privkey.pem /opt/so/saltstack/local/salt/nginx/ssl/ssl.key
    chmod 644 /opt/so/saltstack/local/salt/nginx/ssl/ssl.key
    cp -f /etc/letsencrypt/live/$FQDN/fullchain.pem /opt/so/saltstack/local/salt/nginx/ssl/ssl.crt
    chmod 640 /opt/so/saltstack/local/salt/nginx/ssl/ssl.crt
    so-nginx-restart
    EOF
    
    chmod +x /etc/letsencrypt/copy_to_nginx.sh

11. Test certbot renew, and create a cron job for it.

    certbot renew --dry-run --post-hook /etc/letsencrypt/copy_to_nginx.sh
    crontab -e

12. Place the following at the TOP of the file and save:

    # check certs once a week
    10 4 * * 0 certbot renew --post-hook /etc/letsencrypt/copy_to_nginx.sh

Maintenance

If the steps above succeeded, there is no special maintenance required. The cron job will renew the certs automatically before they expire.

Use SecurityOnion's built-in soup utility to maintain your SecurityOnion stack.

Everything worked out perfect. I had to remember to use sudo in front of most commands. The entries where it created files (ini & sh) I had to create them manually set chmod to 777 then add the text then reset the permissions. Most likely cause was my lack of Linux experience. Only other thing that caught me out was exiting VIM and saving. Again, Thanks, great write up.

I've been unable to get Elastic to offer the new cert to clients.
If you go to http://FQDN:8220 and look at the SSL you can see it seems to know about the FQDN as it is listed as a SAN, but it still offers the older hostname in my case.
We could paste the new cert into the GUI, but don't want to do this every 3 months.
Probably my lack of expertise and not confident enough to play with it.
Anyone have success with this?

@adamphetamine The cert used by Elastic Filebeat on port 8220 is generated separately and stored at /etc/pki/filebeat.crt /etc/pki/elasticfleet-server.crt. This one might be trickier since I think the Filebeat agent packages are shipped with this cert. You could hypothetically copy your LetsEncrypt cert here with the post-hook but I don't know the full implications of it. Worth a backup & try!

Confession time- I wrote this for SO v2.3- but very glad you've done the work this time...
https://github.com/adamphetamine/letsencrypt-security-onion

That's great information, thank you!
I'm using macOS, and the Elastic Agent downloads from the Server aren't packaged properly- so I've been using the install instructions from Elastic.
If the Server offers a cert that is valid, we don't have to use the --insecure flag so this is definitely something I will test.
Thanks again

@adamphetamine Nice! One of these days I'll get the energy to build the docker image for it and put in a PR to SO.

definitely out of my depth here. I replaced the cert at /etc/pki/elasticfleet-server.crt and rebooted the Server. It reverted to the previous, untrusted cert. Not really surprising, but I wonder if we might have better luck replacing the managerssl cert and key like in this older example- looks dangerous though-
https://github.com/adamphetamine/letsencrypt-security-onion/blob/main/move-certs.sh

Anyway I've reached the limit of my knowledge so I'll leave it to smarter people, thanks again for this writeup, much appreciated.

Just to indicate the scale of the issue-

ls /etc/pki
ca.crt                  elasticfleet-agent.p8      elasticfleet-logstash.p8     elasticfleet-server.key  filebeat.key  java            registry.crt  tls
ca.key                  elasticfleet-kafka.crt     elasticfleet-lumberjack.crt  elasticsearch.crt        filebeat.p8   managerssl.crt  registry.key
ca-trust                elasticfleet-kafka.key     elasticfleet-lumberjack.key  elasticsearch.key        influxdb.crt  managerssl.key  rpm-gpg
elasticfleet-agent.crt  elasticfleet-logstash.crt  elasticfleet-lumberjack.p8   elasticsearch.p12        influxdb.key  redis.crt       rsyslog
elasticfleet-agent.key  elasticfleet-logstash.key  elasticfleet-server.crt      filebeat.crt             issued_certs  redis.key       swid

I replaced the cert at /etc/pki/elasticfleet-server.crt and rebooted the Server. It reverted to the previous, untrusted cert.

Yeah, that's kind of what I was afraid of. I don't know which process manages those certs. I did notice that they are mounted by the Elastic Fleet container as read-only, so if they're getting modified, it's happening somewhere else - probably the salt config for Elastic Fleet.

A start for your search:

docker inspect so-elastic-fleet

Follow up on this- I noticed that my cert had expired and found the cause is that the Cloudflare plugin needs to be updated. Here is the error-

The following plugins are using an outdated python version and must be updated to be compatible with Certbot 3.0. Please see https://community.letsencrypt.org/t/certbot-3-0-could-have-potential-third-party-snap-breakages/226940 for more information:
  * certbot-dns-cloudflare

Removing and re installing the Certbot snap doesn't work because that doesn't delete the old plugin.
After a bunch of trial and error I found this could be done with

# snap list | grep certbot-dns
certbot-dns-cloudflare  2.11.0              3451   latest/stable  certbot-eff**  -
# sudo snap remove certbot-dns-cloudflare

Once you've removed the old Cloudflare plugin, you can install a new one with

# snap install certbot-dns-cloudflare

There might be easier ways to simply update it, but this cost me a lot of time so worth documenting...

That's interesting. snapd automatically upgraded the Cloudflare plugin to 3.0.1 for me. 🤔

I looked extensively but couldn't find a method to update the plugin, I probably missed

sudo snap refresh certbot-dns-cloudflare

If anyone can verify this works- then I don't mind looking a bit dumb if it helps others...