Secure Remote Syslog Transport: Best Practices for Encrypted Logging Over Public Networks


1 views

When centralizing logs from distributed servers, we face a critical security dilemma: syslog's traditional UDP transport (port 514) provides no encryption, while TCP syslog (port 6514) still lacks native encryption in most implementations. This leaves log data vulnerable to interception when traversing public networks.

Here are three production-grade approaches I've successfully implemented:

1. Syslog-ng with TLS Encryption

The most elegant solution uses syslog-ng's native TLS support. First, generate certificates:

openssl req -x509 -newkey rsa:2048 -keyout ca_key.pem -out ca_cert.pem -days 3650 -nodes
openssl genrsa -out server_key.pem 2048
openssl req -new -key server_key.pem -out server.csr
openssl x509 -req -in server.csr -CA ca_cert.pem -CAkey ca_key.pem -CAcreateserial -out server_cert.pem -days 3650

Server configuration (/etc/syslog-ng/syslog-ng.conf):

source s_network {
    syslog(ip(0.0.0.0) port(6514) transport("tls")
    tls(key-file("/etc/syslog-ng/server_key.pem")
        cert-file("/etc/syslog-ng/server_cert.pem")
        ca-dir("/etc/syslog-ng/ca.d")))
};

Client configuration:

destination d_syslog_tls {
    syslog("logs.example.com" port(6514) transport("tls")
    tls(peer-verify(required-trusted)
        ca-dir("/etc/syslog-ng/ca.d")))
};

2. RELP Protocol with Stunnel

For legacy systems, RELP (Reliable Event Logging Protocol) through stunnel provides strong security:

# stunnel server config (/etc/stunnel/stunnel.conf)
cert = /etc/stunnel/stunnel.pem
[syslog]
accept = 6514
connect = 127.0.0.1:601

Configure rsyslog to use RELP:

module(load="omrelp")
*.* :omrelp:logs.example.com:6514

3. SSH Port Forwarding with Correct Source Preservation

The OP's SSH tunnel attempt can work with proper configuration. This preserves original hostnames:

ssh -N -R 5514:localhost:514 -p 22 sysloguser@central-log-server

On the central server's rsyslog.conf:

$PreserveFQDN on
$ModLoad imtcp
$InputTCPServerRun 5514

TLS adds about 15-20% overhead compared to plain TCP. For high-volume environments:

  • Use elliptic curve cryptography (ECDSA) certificates
  • Implement log aggregation at edge nodes
  • Consider protocol buffers instead of plain text

Essential metrics to track:

# TLS handshake failures
grep "TLS handshake failed" /var/log/syslog-ng/syslog-ng.log

# Queue statistics
syslog-ng-ctl stats | grep queue

When trying to centralize logs from multiple servers across the public internet, traditional syslog (UDP/514 or plain TCP) becomes a security liability. Packets can be intercepted, modified, or spoofed. Even worse, some syslog implementations still default to unencrypted transmission.

A naive approach like ssh -L 5514:localhost:514 user@loghost seems tempting but has critical flaws:

# Problematic approach (loses source IP information)
ssh -N -L 5514:localhost:514 user@central-log-server.example.com

This makes all logs appear to originate from localhost on the receiving end, destroying valuable host identification metadata.

Using syslog-ng or rsyslog with TLS provides both encryption and preserved metadata. Here's a working configuration:

# /etc/syslog-ng/conf.d/secure-remote.conf (client side)
destination d_central_logs {
  syslog("central.example.com"
    transport("tls")
    port(6514)
    tls(
      peer-verify(required-trusted)
      ca-dir("/etc/ssl/certs")
    )
  );
};

# /etc/rsyslog.d/secure-remote.conf (alternative client config)
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
*.* @@(o)central.example.com:6514

For legacy systems where TLS isn't available, use SSH's native syslog forwarding:

# Preserves original hostnames
ssh -R 515:localhost:514 remoteuser@central-log-server \
  -o RemoteCommand="nc -kl 515 | logger -t \"%n\""

While often considered "overkill," WireGuard provides excellent performance for log forwarding:

# WireGuard config snippet for log aggregation
[Interface]
PrivateKey = CLIENT_PRIVATE_KEY
Address = 10.8.0.2/24

[Peer]
PublicKey = SERVER_PUBLIC_KEY
AllowedIPs = 10.8.0.1/32
Endpoint = central.example.com:51820
PersistentKeepalive = 25

Always validate your secure channel:

# Test TLS connection (requires OpenSSL)
openssl s_client -connect central.example.com:6514 -showcerts

# Verify packet source preservation
logger "Test message"
tail -f /var/log/remote/hostname.log | grep "Test message"