This document describes the setup of a syslog server using syslog-ng that would be useful for monitoring devices an systems on the network. With proper configuration this could replace expensive enterprise device implementations. The reference system used in this example is a Gentoo GNU/Linux system.

Syslog Concepts

Syslog is designed to make an administrators life easier by centralizing the logs from many devices or systems on the network. With all of these logs in one place administration is simplified; monitoring, archival and compliance can be addressed easily from one repository. All logs from all hosts can be monitored for critical (or warning or whatever) messages. Host failures will not lose the logs and they won't be lost if the original host catches fire. If necessary a policy can be put in place to archive the logs for a set time period, possibly for compliance reasons.

There are many syslog packages to choose from, syslog-ng is used here. Syslog-ng will be installed, configured to have internal and external logging and to monitor those logs for errors. Examples are given showing how to send errors to an email address or to log this information into a database.

Syslog-ng Configurations

Syslog-ng is highly configurable through the use of sources, destinations and filters. These three items are connected via log statements allowing for multiple message paths. Sources are where log information comes from, destinations are where it goes. Filters are used to filter messages on their path through the logging system. All of these configuration options are stored in /etc/syslog-ng/syslog-ng.conf. When editing this file white space is not important, a # marks a comment.

The syslog server needs to have enough horse-power and storage to be able to process and store the logs for as long as policy defines. As a real-world example Edoceo put this in place for a client who had 23 hosts logging to this system. One of the devices generated eight megabytes of data per day, the other 22 generated close to four MB per day combined. The inbound messages were monitored for critical phrases and stored as flat files, one per day. The system was a dual P3/500 with 786MB RAM and four 9G SCSI drives configured as RAID5+hotspare using Linux software raid. No logs have been dropped and load average and memory usage look fine after months of flawless operation. The machine did not have enough storage to archive the logs for seven years so the were moved to a SAN on a nightly basis.

Install syslog-ng

This package was chosen because it's highly configurable and very responsive and stable, all good things. Install according to the target system, Gentoo users do the following as root. It might be a good idea to say man syslog-ng after installation. 

emerge syslog-ng
rc-update -a syslog-ng default

Configure syslog-ng

Syslog-ng demonstrates is usefulness right out of the gate with concepts of source, destination, filter and log. In this setup two sources are defined: internal, for the host running syslog-ng; and external, for the other hosts on the network. Many destinations are defined as well, one for each of the sources. Filters will be demonstrated to automatically route certian message types to an email or database.

Some global options for syslog are shown below, read man syslog-ng.conf for more info. 

options
{
  long_hostnames(off);
  stats(43200);
  # Buffers 5 lines before flush
  sync(5);
};

About Syslog Facilities

Syslog provides various facilities, if you're going to fully leverage Syslog it's important to organise your systems with this in mind. At Edoceo we organise our as follows, however it's not critical that anyone follows these rules. 

# local0 = Database: PostgreSQL,MySQL and other database
# local1 = HylaFax, Asterisk, FreeSWITCH and other communications
# local2 = Rsync
# local3 = Windows Hosts from Winlogd
# local4 = Web Services/Servers (Apache, Lighttpd, Nginx, Varnish)
# local5 = Open
# local6 = Open
# local7 = Company/Internal/Organization Applications

Sources

Open /etc/syslog-ng/syslog-ng.conf and edit as below to create these two sources. 

# Internal Source
source s_internal
{
  internal(); # syslog-ng messages
  pipe("/proc/kmsg");
  unix-stream("/dev/log");
};

# External Source
source s_external
{
    # Typical BSD Syslog
    udp(ip(192.168.42.2) port(514));

    # TCP Syslog - may not be necessary
    tcp(ip(192.168.42.2) port(51400));

    # IETF syslog
    syslog(
        flags(no-multi-line)
        ip(10.65.0.5)
        keep-alive(yes)
        keep_hostname(yes)
        transport(udp)
# TLS Options
#        tls()
    );

};

This creates the initial seperation between logs, to keep things clean. Each of these two sources can have different filters and destinations applied.

Check the settings by using netstat

~ # netstat -tanpu|grep syslog
tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN      11020/syslog-ng
udp        0      0 0.0.0.0:514             0.0.0.0:*                           11020/syslog-ng
udp        0      0 10.65.0.5:601           0.0.0.0:*                           11020/syslog-ng

Destinations

Destinations are where log messages go. In theory the syslog server will not be doing much else so it's internal logs can be sent to one file. The external messages are to directories in the form of /host/year/month/day/facility.log. The directories are automatically created and owned rw by root only. 

destination d_int
{
  file("/var/log/localhost/$YEAR/$MONTH/$DAY/messages.log" \
    owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes));
};

destination d_ext
{
  file("/var/log/$HOST/$YEAR/$MONTH/$DAY/$FACILITY.log" \
    owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes));

  # This seperated by severity too.
  # file("/var/log/$HOST/$YEAR/$MONTH/$DAY/$FACILITY-$PRIORITY.log" \
  #   owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes));
};

Syslog-ng provides more $MACRO expansion options, including numeric $TAG and $PRI fields. See the reference manual for more information.

Syslog-Ng Performance

Centralized System Logging

System logging is very important, won't get into that here, and it must be setup properly. In this document samples of syslog-ng configurations are given to assist the reader in finding the proper setup for their environment.

Configure Sources

It's adviseable to setup one source for the individual inputs, for flexibility. If, however, this is not necessary one may combine multiple source drivers into a single source. The use of the s_ on the names is simply a convention used by many and not required. 

source s_internal { internal(); }

# These two are the same, choose one
source s_external { udp(); };  # All interfaces
source s_external { udp(ip(192.168.0.1) port(514) ); };  # Specific

# Using two source drivers in one source
source s_kernel { pipe("/proc/kmsg"); unix-stream("/dev/log"); };

# This source combines everything
source s_everything { internal(); pipe("/proc/kmsg"); unix-stream("/dev/log"); udp(); };

Configure Destinations

Log message need to be stored somewhere to be useful, use destinations to accomplish this. The d_ prefix on the names is convention and not required. Messages can be sent to files, network hosts or programs 

destination d_file { file("/var/log/messages"); };
destination d_fancy_file { file("/var/log/$YEAR/$MONTH/$DAY/$FACILITY.$PRIORITY.log" \
    owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); };
destination d_network { udp( "192.168.0.1" port(514) ); };
destination d_program { program("/opt/edoceo/sql2syslog"); };

These destinations only describe where to put log messages, not which messages to put there. The first puts all the messages in one file, the second creates special files for each message type. The third sends the messages to a different host, which is hopefully listening. Finally a program may be specified to send the log messages as one line to the STDIN of a program allowing for endless possibilities. See the reference for more information.

Configure Filters

Incomplete.

filter f_auth { facility(auth); };
filter f_authpriv { facility(auth, authpriv); };
filter f_syslog { not facility(authpriv, mail) and not match(ppp.*LCP); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_user { facility(user); };
filter f_uucp { facility(cron); };
filter f_ppp { program(ppp); };
filter f_news { facility(news); };
filter f_debug { not facility(auth, authpriv, news, mail) and not match(ppp.*LCP); };
filter f_messages { level(info..warn) and not facility(auth, authpriv, mail, news); };
filter f_emergency { level(emerg); };

filter f_info { level(info); };
filter f_notice { level(notice); };
filter f_warn { level(warn); };
filter f_crit { level(crit); };
filter f_err { level(err); };

Configure Logs

The sources messages must now be connected to a destination, optionally with a filter, by the log statement.

This setting takes all the messages from the s_everything input source to the d_network destination. No filter is applied. 

log { source(s_everything); desitnation(d_network); };

Examples

Logging in the simplist form.

source s_all { internal(); pipe("/proc/kmsg"); unix-stream("/dev/log"); };
source d_file { file("/var/log/messages"); };
log { source(s_all); destination(d_file); }

Using the macros defined by syslog-ng create a unique file for every facility. Use of the other macros can create directories and files based on date. Notice also that a single source is sent to multiple destinations. 

source s_all { internal(); pipe("/proc/kmsg"); unix-stream("/dev/log"); };
destination d_file { file("/var/log/$FACILITY.log"); };
destination d_date { file("/var/log/$YEAR/$MONTH/$DAY/$FACILITY.log" create_dirs(yes) ); };
log { source(s_all); destination(d_file); destination(d_date); };

This example shows how to setup the system logging to send everything to a central host at 192.168.0.1 Hopefully it's listening and has nice rules for processing setup. 

source s_all { internal(); pipe("/proc/kmsg"); unix-stream("/dev/log"); };
source d_net { udp( "192.168.0.1" ); }
log { source(s_all); destination(d_net); }

Conclusion

Syslog-ng is a super powerful and flexable method of logging. The manual pages for syslog-ng are lacking but the Reference Manual (link below) makes up the difference.

See Also

ChangeLog