• RFC 5424 on The Syslog Protocol
• RFC 5425 on Transport Layer Security (TLS) Transport Mapping for Syslog
• RFC 5426 on Transmission of Syslog Messages over UDP

Now I am lost between different code versions in my development SVN, the sf.net CVS, and the NetBSD CVS so I will have to re-merge a little before I can really fix further problems. But I am grateful that Christos went through the code again before comitting it. Now I can continue an argument with my flatmate whether being a BSD contributer is more geeky prestigious than having a low Erdős number…  :-)

Just don’t use it in debug mode whith all input printed verbatim to the poor terminal… :-|

In theory a Trac is really nice because it combines several software development tools in one place and makes it easier to collaborate on one project; — but it practice I am mostly working alone and do not have to coordinate with anyone else (of course I should document my work, but it is never strictly necessary for further development). And then the Trac offers little immediate benefits, whereas my notebook is much faster to use, easier to edit and is also usable on trains and buses.

So while I should have much more data online like this …

… most of my notes and documentation actually look like this:

So this is a good time to re-read the draft and check its requirements against my current syslogd code:

The TCP port NNN has been allocated as the default port for syslog over TLS, as defined in this document.

Nothing to do here as syslogd depends on /etc/services for its port (unless configured otherwise).

Implementations MUST support TLS 1.2 [I-D.ietf-tls-rfc4346-bis] and are REQUIRED to support the mandatory to implement cipher suite, which is TLS_RSA_WITH_AES_128_CBC_SHA.  This document is assumed to apply to future versions of TLS, in which case the mandatory to implement cipher suite for the implemented version MUST be supported.

I depend on OpenSSL to implement this.

Both syslog transport sender (TLS Client) and syslog transport receiver (TLS server) MUST implement certificate-based authentication.
o  Certification path validation: The TLS peer is configured with one or more trust anchors (typically root CA certificates), which allow it to verify a binding between the subject name and the public key.

Check.

Implementations MUST support certificate fingerprints in Section 4.2.2 and MAY allow other formats for end-entity certificates such as a DER encoded certificate.

Check for both.

Both transport receiver and transport sender implementations MUST provide a means to generate a key pair and self-signed certificate in the case that a key pair and certificate are not available through another mechanism.

Coming soon…

Update (for completeness): Now implemented. When given a config option and the certificate/key files cannot be read, then a 1024-bit RSA key is created and used for a self-signed X.509 certificate with the hostname as CN.

The transport receiver and transport sender SHOULD provide mechanisms to record the end-entity certificate for the purpose of correlating it with the sent or received data.

In form of a log message:
Accept client certificate from 127.0.0.1 due to matching hostname/subject.Subject is "localhost", fingerprint is "SHA1:E4:E1:A6:1C:D4:31:D7:D4:9B:B8:DC:DF:DD:CE:30:71:46:00:92:C9"

Both client and server implementations MUST make the certificate fingerprints for their certificate available through a management interface.

Could I say the shell is my “management interface” and provides the fingerprints with “openssl x509 -in xyz.crt -noout -fingerprint“?  :-)

Anyway, the used certificate is logged on server start and looks like this:
syslogd: Initialize SSL context using library "OpenSSL 0.9.8h 28 May 2008". Load certificate from file "localhost.crt" with CN "localhost" and fingerprint "SHA1:E4:E1:A6:1C:D4:31:D7:D4:9B:B8:DC:DF:DD:CE:30:71:46:00:92:C9"

Implementations MUST support SHA-1 as the hash algorithm and use the ASCII label “SHA1″ to identify the SHA-1 algorithm.

Hashing is provided by OpenSSL again; syslogd will support/recognize all algorithms that OpenSSL’s EVP_get_digestbyname() supports.

Syslog applications SHOULD be implemented in a manner that permits administrators, as a matter of local policy, to select the cryptographic level and authentication options they desire.

There are different ways to authenticate the peer’s certificate.
The cryptographic level is not configurable, because I do not see the demand for that. (If indeed required then every OpenSSL setting, especially the list of usable ciphers, could be turned into a syslog.conf option.)

TLS permits the resumption of an earlier TLS session or the use of another active session when a new session is requested, in order to save the expense of another full TLS handshake.  The security parameters of the resumed session are reused for the requested session.  The security parameters SHOULD be checked against the security requirement of the requested session to make sure that the resumed session provides proper security.

I do not implement TLS session resumption, because I expect a syslogd TLS connection to last for some hours or days so the handshake cost is negligible.

All syslog messages MUST be sent as TLS “application data”.

Check.

A transport receiver MUST use the message length to delimit a syslog message.  There is no upper limit for a message length per se.  However, in order to establish a baseline for interoperability, this specification requires that a transport receiver MUST be able to process messages with a length up to and including 2048 octets.  Transport receiver SHOULD be able to process messages with lengths up to and including 8192 octets.

syslogd has no fixed limits on message lengths. On connection establishment a 2048 byte input buffer is allocated, longer messages will cause a realloc(). If that fails, then one message will be skipped.

A TLS client MUST close the associated TLS connection if the connection is not expected to deliver any syslog messages later.

I wrote a syslogd — it always expects to deliver more messages  ;-)

It MUST send a TLS close_notify alert before closing the connection.  A client MAY choose to not wait for the server’s close_notify alert and simply close the connection, thus generating an incomplete close on the server side.  Once the server gets a close_notify from the client, it MUST reply with a close_notify unless it becomes aware that the connection has already been closed by the client (e.g., the closure was indicated by TCP).

Check. Although I make no difference between client and server mode and always shutdown the TLS connection.

When no data is received from a connection for a long time (where the application decides what “long” means), a server MAY close the connection.

There is no timeout implemented.

Implementations MUST support specifying the authorized peers using certificate fingerprints, as described in Section 4.2.1 and Section 4.2.2.

Check.

Implementations MUST support specifying the authorized peers using locally configured host names, MUST support certification path validation [RFC5280] and matching the name with the certificate as follows: [...]

I match a given subject against SubjectAltName/dNSName, SubjectAltName/ipAddress, and commonName.
I just do not support the wildcard matching for hostnames (which is a MAY). I do want to but it has a really low priority on my TODO-list.

A typical task (like “how do I get the commonName from this?”) starts with fgrep -r xyz ./openssl, looking for likely function names in the .h files. Since many functions are not documented in man pages this is followed by grepping all #defines for it (preprocessor macros are used extensively) and finding the real implementation with source code to figure out if it does what I need. Sometimes the next step is even copying/reimplementing the found function, because often there are functions to output a structure to a BIO (the I/O abstraction layer) but not to copy its data as a string to work with it.

Several factors contribute to that:

• There is more than one way to do it. You can use BIOs for everything or manage sockets and file descriptors yourself; or you can call (e.g.) bio_print_NID(x) or get_data_by_NID(x) or get_data_by_OBJ(NID2OBJ(x)). While it is good that every object has a pretty complete set of methods/functions, this makes it hard to reuse existing code or follow a documentation.
• Documentation structure. Most parts of OpenSSL are object-oriented. It seems to me that a Javadoc-like documentation is better for that, because it maps the class/object structure and allows you to see what data is held and which methods exist to access it. The man page structure seems to be more apropriate for things like system calls where the functions are mostly independent and functionally orthogonal. OpenSSL shows this because although the existing man pages are pretty good, those written for a single function give too little information for the larger task, and many man pages describe 5-10 functions at once because they access the same object.
• Knowledge not written down. The OpenSSL mailing lists probably contain more information than the man pages. Searching the lists answers many questions because they were asked before and quickly answered by the developers. — Their answers just did not make it into a documentation.
• Not always clear APIs. Some technically unnecessary access methods are not implemented, probably because the developers knew that they could just access the underlying struct. I also found several code snippets (even given as advice on the mailing list) that implicitly assume data types to be equal or direct access to certain structure fields. This makes it harder to find the right methods.

Of course I know how an open source project works, and I am not the one to complain about missing functions because I do not have the time to implement them either (although I do think it is time to finally IPv6-enable the network-BIOs ;)

But OpenSSL seems to be too important to be left with so little good documentation on how to use it correctly (and securely). This all reminds me of Cyrus SASL, which has similar problems because we all have to rely on it but really few people understand it :-(

Most applications and protocols use a request-response or a session model and do not have problems with this, because they simply reset their status on connection loss and start again. Syslog is different because it does not use a backchannel (for acks or other server responses) and completely relies on the one-direction channel for server and client to synchronize their states over long periods of time and across connection losses. After reading Rainer’s text it took me some time to believe the problem described, and even then I saw the words and accepted the reasoning, but it still felt wrong to believe in them.

This week I began my own Syslog programming with a small client/server program to test different IPC protocols; and the plain TCP variant nicely confirmed the observation above. Only after trying SCTP which did not have that problem, I realized that we asked the wrong question. (I did not dig into it but I guess the socket is marked broken really fast on connection shutdown, so the first write to it returns an error immediately). — sendmsg() is not our only interface to the TCP-layer and if sendmsg() does not tell us whether a connection is still established then we just have to use another interface.

My first experiment lead to a bad solution: I found that Linux and FreeBSD have a struct tcp_info with TCP state information that can be accessed with getsockopt(). This allows us to always check if the connection is still established:

getsockopt(sock, IPPROTO_TCP, TCP_INFO, &tinfo, &tinfo_size)
if (tinfo.tcpi_state != TCPS_ESTABLISHED) {
  if (sock) { close(sock); }
  /* reconnect */
}
/* send() */

After that I finally found the good and really simple solution: Just use recv() to check on the connection status. On connection shutdown recv() immediately indicates that the socket is unavailable and the application can react upon it by reconnecting:

rc = recv(sock, msgbuf, BUFSIZE, MSG_DONTWAIT | MSG_PEEK);
if (!rc) {
    if (errno == EAGAIN) {
        /* server closed connection */
        connected = 0;
    } else {
        perror("Error in recv()");
    }
}
if (!connected) {
    /* (re)connect */
    connected = 1;
}
/* send() */

Source code is online in my Trac. Note that the tcp_info version only works on FreeBSD; on Linux I did not find the necessary header files yet and NetBSD does not support tcp_info (bad solution anyway).

Update: Just a note on the intention and the kind of problem that is solved here. TCP has no atomic and synchronous send. Period and no need to argue.

With “real” network problems TCP does not show errors immediately because it was practically designed not to. (I have already worked on failur tolerance in a cluster computing environment and there are similar problems.)

The main problem I want to improve on is the simple LAN without connection or even packet loss. Where it is really unnecessary (and annoying) for a TCP-based syslog to derministically loose messages on a server reload, just because the client does not react upon a nice and clean connection shutdown.

Later Update (22.6.): One obvious problem is the additional system call per socket operation. So I am quite happy I do not have to use this in the BSD syslogd implementation. I found it much easier to register a BSD kernel event on the socket. That way the kernel immediately notifies the application when there is either data to read from the socket or when it is closed.

Today the participants in Google’s Summer of Code 2008 were announced. – And my project was chosen. :-)

So now I will work for NetBSD and implement the new IETF syslog protocols.