Configure TLS Encrypted Tunnel For Remote Logs Using Syslog-ng

You might be a Sysadmin, developer, DBA or whatever, logs are like treasure boxes for anyone working in IT. And the best practice to keep logs in a central location together with local copy. Most of the logging programs have the ability to send logs to a remote logging server (as well as receive logs from remote machines); eg rsyslog, syslog-ng etc.

But, still there is a concern for sending server/application/database logs sending over tcp as plain text; yes indeed. But no need to worry as most of the logging programs will have simple mechanisms to implement TLS Tunnels for sending and receiving logs. In below demo, we will implement TLS tunnel to send logs from one machine (using syslog-ng) and receive the logs on another logging server (syslog-ng).

On Logging Server

Install Syslog-ng

You will most likely need to enable Extra Packages for Enterprise Linux (EPEL)

# cd /etc/yum.repos.d/
# wget https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng321/repo/epel-7/czanik-syslog-ng321-epel-7.repo
# yum install syslog-ng
# systemctl enable syslog-ng
# systemctl start syslog-ng

Note : If you have rsyslog or other logging systems running, you need to stop that first and configure custom items (if any) in syslog-ng manually.

Generate SSL Certificate

Generate Key and CSR

We will store the keys in /etc/syslog-ng/ssl

# cd /etc/syslog-ng
# mkdir ssl
# cd ssl
# openssl genrsa -des3 -out logserver.key 2048
# openssl req -new -key logserver.key -out logserver.csr

Remove the passphrase from the key

cp logserver.key logserver.key.org
openssl rsa -in logserver key.org -out logserver.key

Generate a self-signed certificate

openssl x509 -req -days 365 -in logserver.csr -signkey logserver.key -out logserver.crt 

Syslog-ng configuration on Server side

Configure a Source to receive logs over TLS

Edit /etc/syslog-ng/syslog-ng.conf and add below section.

(You can either directly edit /etc/syslog-ng/syslog-ng.conf or add separate configuration file under conf.d for easy configuration file management. eg: /etc/syslog-ng/conf.d/mylog.conf)

# Step 1 - TLS Source Listen
source source_514_tls {
 tcp(port(514)
 tls(
   # SSL Certificates which we have created in previous steps
   key_file("/etc/syslog-ng/ssl/logserver.key")
   cert_file("/etc/syslog-ng/ssl/logserver.crt")
   peer_verify(optional-untrusted)
   )
 flags(no-multi-line)
 );
 };

Configure a Destination

This can be another logging server (eg: Elastic, Splunk, ArcSight Connectors etc) or a local destination.

# Step 2 - Configure Destinations

# Local Destination on same server
destination dest_514_local { file("/var/log/messages_514"); };

# Destination on another server:1514
destination dest_1514 {
 tcp("10.1.10.100" port (1514)
 };

Configure Logging

# Step 3 - Configure Source to Destination Logging

# Forward Logs from 514 to another server:1514 
log { source(source_514_tls ); destination(dest_1514); };

# Log same logs received on 514 to locally.
log { source(source_514_tls ); destination(dest_514_local); };

On Client Machine

Here client means any machine who send data to connector machine

Configure Certificate

You need to get the certificate from logging server and configure to send data over TLS.

Download the certificate from server

Download logserver.crt and keep it under directory /etc/syslog-ng/ssl/ssl-for-client/ (or any other suitable location).

# ls -l /etc/syslog-ng/ssl/ssl-for-client/
total 4
-rw-r--r-- 1 root root 1257 Jun 27 09:52 logserver.crt

Find hash for your key

# openssl x509 -noout -hash -in /etc/syslog-ng/ssl/ssl-for-client/logserver.crt
acd0d3bb

Create Symlink to Certificate

Create a symbolic link to the certificate that uses the hash returned by the previous command, with an added .0 suffix.

# ln -s /etc/syslog-ng/ssl/ssl-for-client/logserver.crt /etc/syslog-ng/ssl/ssl-for-client/84d92a45.0
# ls -l /etc/syslog-ng/ssl/ssl-for-client/
total 4
lrwxrwxrwx 1 root root   47 Jun 27 09:54 acd0d3bb.0 -> /etc/syslog-ng/ssl/ssl-for-client/logserver.crt
-rw-r--r-- 1 root root 1257 Jun 27 09:52 logserver.crt

Syslog-ng configuration on Client Side

Please note the certificate path mentioned at client side in destination d_mesg_copy.

Define a Destination to Logging Server

# Step 1 - Destination 10.1.10.200:514
destination d_mesg_copy {
 tcp("10.1.10.200" port (514)
 tls(
   ca_dir("/etc/syslog-ng/ssl/ssl-for-client/")
   )
 );
 };

Enable Logging to Remote Target

# Step 2 - Send Copy of var-log-messages to d_mesg_copy
log { source(s_sys); filter(f_default); destination(d_mesg_copy); };

Make sure you have enabled and restarted syslog-ng daemons on both servers after config updates.