2011-11-19T01:31:21 frodo kernel: pid 47812 (try), uid 0: exited on signal 10 (core dumped)

Source of the program is Perl’s Configure script, which compiles its C library test cases into programs named try. And apparently amd64 systems need to use the function va_copy() because the test case without this function leads to a segfault.

Depending on the number of events the step has to be adjusted. The arguments to cut determine the used time intervals; the above aussumes a traditional BSD Syslog timestamp and will yield daily counts. For hourly counts one can use cut -d : -f 1 (then the print command has to be changed as well to printf $2, $3, $4, bar;) — for Syslog-Protocol timestamps one can use cut -d T -f 1 or cut -d : -f 1. Note that only intervals with matches are shown, there is no easy way to add empty lines.

Example:

[mschuett@mail] /var/log> fgrep -h 'idle for too long, closing connection' cyruslog | cut -d T -f 1 | uniq -c | awk '{step=10; bar = ""; for(i = $1; i>=1; i = i-step) bar = bar "#"; print $2, bar; }'
2010-03-01 ########################
2010-03-02 ################################
2010-03-03 #####################
2010-03-04 ##############################
2010-03-05 #################################
2010-03-06 ######################
2010-03-07 ###########################
2010-03-08 #####################
2010-03-09 ###################
2010-03-10 #####################
2010-03-11 ##################

My first approach was to filter by file and path names, but not every message includes a filename, so I was left with some lines like these:

2010-03-15T00:00:15.00+01:00 server php-cgi: Undeclared entity warning at line 54, column 8

To overcome the problem I introduce two new php.ini variables: syslog.program, to give the program name, and the boolean syslog.pid, to enable logging the PID. Default values are "" and Off, resulting in the previous behaviour (using the executable name as program name without PID). I use suphp, so every user has an own php.ini with syslog.program set to “php/username”, thus writing log lines like:

2010-03-19T09:00:14.00+01:00 server php/mschuett[87777]: Undeclared entity warning at line 54, column 8

With mod_php the variables can also be set per directory with php_admin_value. Changing them with ini_set() is not supported (not necessary because a new call to openlog() has the same effect). For enhanced security one could/should also use the php.ini setting disable_functions=openlog to prevent users from overriding these settings.

Because the source file is also affected by the Suhosin patch, I prepared two diffs:

• PHP 5.2.12:patch-php5.2.12.ext_standard_syslog.diff
• PHP 5.2.12 with Suhosin: patch-php5.2.12.suhosin.ext_standard_syslog.diff

[Update: the patches still work for PHP 5.3.6]
[Update2: finally got around to write a Feature Request #54144]

These messages have an interesting format like:
Sep 29 21:41:18 cathy ufs: [ID 845546 kern.notice] alloc /: file system full

I found the thought fascinating to have a system- (or at least kernel-) wide list of all possible log events; but it does not work like that. Actually the message ID is a hash value over the format string of the message (in the example the string is: "alloc: %s: file system full"); for a description see msgid(1M) and the calculation STRLOG_MAKE_MSGID(). In practice these hashes are probably unique (even more so in combination with the severity notice and the subsystem ufs), but I would expect to find collisions duplicate IDs for very simple or generic format strings (like printf("%s", some_string) or printf("%s: %d", some_key, some_value)).

• RFC 5424 on The Syslog Protocol
• RFC 5425 on Transport Layer Security (TLS) Transport Mapping for Syslog
• RFC 5426 on Transmission of Syslog Messages over UDP

Complete plot: occurance count by line length

Detail: occurance count for line lengths ≤ 250

As every real data set the selection is somewhat skewed. It includes Apache httpd access logs but no httpd error logs. The Windows logs are collected from NTSyslog but are non-representative because they include many WPKG debug messages. I also use ISO instead of BSD timestamps, so every line is 14 chars longer than with traditional syslogd; the timestamp, the hostname and a programname/tag use around 50 chars which is the minimum lenght here. The most notable artifact are the many long (>800 chars) messages. — These are syslog-ng log statistics from the logserver itself (used for monitoring).

Most applications and protocols use a request-response or a session model and do not have problems with this, because they simply reset their status on connection loss and start again. Syslog is different because it does not use a backchannel (for acks or other server responses) and completely relies on the one-direction channel for server and client to synchronize their states over long periods of time and across connection losses. After reading Rainer’s text it took me some time to believe the problem described, and even then I saw the words and accepted the reasoning, but it still felt wrong to believe in them.

This week I began my own Syslog programming with a small client/server program to test different IPC protocols; and the plain TCP variant nicely confirmed the observation above. Only after trying SCTP which did not have that problem, I realized that we asked the wrong question. (I did not dig into it but I guess the socket is marked broken really fast on connection shutdown, so the first write to it returns an error immediately). — sendmsg() is not our only interface to the TCP-layer and if sendmsg() does not tell us whether a connection is still established then we just have to use another interface.

My first experiment lead to a bad solution: I found that Linux and FreeBSD have a struct tcp_info with TCP state information that can be accessed with getsockopt(). This allows us to always check if the connection is still established:

getsockopt(sock, IPPROTO_TCP, TCP_INFO, &tinfo, &tinfo_size)
if (tinfo.tcpi_state != TCPS_ESTABLISHED) {
  if (sock) { close(sock); }
  /* reconnect */
}
/* send() */

After that I finally found the good and really simple solution: Just use recv() to check on the connection status. On connection shutdown recv() immediately indicates that the socket is unavailable and the application can react upon it by reconnecting:

rc = recv(sock, msgbuf, BUFSIZE, MSG_DONTWAIT | MSG_PEEK);
if (!rc) {
    if (errno == EAGAIN) {
        /* server closed connection */
        connected = 0;
    } else {
        perror("Error in recv()");
    }
}
if (!connected) {
    /* (re)connect */
    connected = 1;
}
/* send() */

Source code is online in my Trac. Note that the tcp_info version only works on FreeBSD; on Linux I did not find the necessary header files yet and NetBSD does not support tcp_info (bad solution anyway).

Update: Just a note on the intention and the kind of problem that is solved here. TCP has no atomic and synchronous send. Period and no need to argue.

With “real” network problems TCP does not show errors immediately because it was practically designed not to. (I have already worked on failur tolerance in a cluster computing environment and there are similar problems.)

The main problem I want to improve on is the simple LAN without connection or even packet loss. Where it is really unnecessary (and annoying) for a TCP-based syslog to derministically loose messages on a server reload, just because the client does not react upon a nice and clean connection shutdown.

Later Update (22.6.): One obvious problem is the additional system call per socket operation. So I am quite happy I do not have to use this in the BSD syslogd implementation. I found it much easier to register a BSD kernel event on the socket. That way the kernel immediately notifies the application when there is either data to read from the socket or when it is closed.

One example where one might rewrite syslog messages is to preserve multiple sending and receiving timestamps. This is useful because a wrong system clock makes the sending timestamp mostly worthless, so you want the logserver to write its own time into the mesage. But if there is a network outage and the client buffers its messages for some time, then sending and receiving timestamps are also different but in this case the sending timestamp is the accurate one. — Thus simply replacing the messages’ timestamp is no general solution, but having multiple timestamps is.

With SD every syslogd could just write its local time into every message, not unlike a Received: line in e-mails (in theory it could do so with BSD Syslog as well, but the important difference is that a new SD element will not mess up later parsing and analysis).

Consider these examples:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com logger - - [transport relay1="relay.example.com" timestamp1="2003-10-11T24:14:16.000-02:00" relay2="logserver.example.com" timestamp2="2003-10-11T18:14:16.702+04:00"] Hello World

<34>1 2003-10-11T22:14:15.009Z mymachine.example.com logger - - [transport relay1="relay.example.com" timestamp1="2003-10-11T24:14:15.840-02:00" relay2="logserver.example.com" timestamp2="2003-10-11T18:54:16.702+04:00"] Hello World

The first message shows a clean transport; but the second message indicates a network problem between the relays because logserver.example.com received it 40 minutes after relay.example.com, so it might have been buffered for that time.

But now enter syslog-sign: Once you sign your messages on one device you most not alter them on their way to their destination as every changes invalidates the signature!

But what if we really really want to add routing information? Would it be feasible to define a structured data element that might be inserted at will and had to be removed before checking signatures?

I think the following specification would suffice for this:

• Define some SD element with SD-ID “transport” to be used to track transport of syslog messages across different relays.
• Every device might insert a transport SD element into the SD field. If other SD elements are present then their order must not be changed; otherwise the new SD element replaces the dash (-).
• All parameters get natural numbers as suffixes. Every device that writes transport information has to check whether a transport SD element is already present. It has to use the parameters with the lowest unused number as suffixes.
• Before signing or signature checking the transport SD element is removed from the message. If it is the only SD element, then it is replaced by a dash (-) to keep the message well-formed.

Using the example above, we start with this message which might be signed at the originating device:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com logger - - - Hello World

This gets submittet to relay.example.com which inserts the transport SD element:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com logger - - [transport relay1="relay.example.com" timestamp1="2003-10-11T24:14:16.000-02:00"] Hello World

and later logserver.example.com which adds additional parameters to the transport SD element, yielding the message as above:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com logger - - [transport relay1="relay.example.com" timestamp1="2003-10-11T24:14:16.000-02:00" relay2="logserver.example.com" timestamp2="2003-10-11T18:14:16.702+04:00"] Hello World

Now say logserver.example.com checks the messages’ signature with syslog-sign. Then it would remove the transport SD element before doing so, thus calculating the hash over

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com logger - - - Hello World

which would be valid and match the signature as it is the same message as was sent from mymachine.example.com in the first place.

Today the participants in Google’s Summer of Code 2008 were announced. – And my project was chosen. :-)

So now I will work for NetBSD and implement the new IETF syslog protocols.