The structured data (SD) in syslog-protocol offers some interesting possibilities when it comes to modifying a message in transit. What would be necessary to enable rewriting in transit while still be able to authenticate by syslog-sign?
Today the participants in Google’s Summer of Code 2008 were announced. – And my project was chosen. :-)
So now I will work for NetBSD and implement the new IETF syslog protocols.
Skimming through last year’s Informatik Spektrum I found an interesting master thesis by Steffen Tambach (german) examining the reliability of UDP transport. The analysis showed that no data was lost on the Fast Ethernet, but all lost packets were dropped by the receiver sowewhere on the network stack between NIC and socket API.
The following recommendations are given to prevent UDP package loss in a local network (besides the obvious ‘have enough bandwidth for your data’ and ‘be able to store/process it fast enough’):
One of the most important issues in a logging system (i.e. syslog) is reliability. But sometimes I get the impression most people asking for reliability do not really want it. IMO a reliability requirement has to be tested against the following extreme case: suppose the logging system fails and you enter a command — should the command be executed although it cannot be logged? If the answer is “Yes” then you do not really want reliable logging (at least for the command you thought about).
I admit that this constructs a dichotomy that might not be necessary. But so far I have not heard of some “weak” or “semi-reliability”, thus the binary distinction only reflects the state of discourse on the subject. I also admit that most related real-life problems were solved if the syslog-world would eventually throw out UDP in favor of TCP or TLS. So this whole reasoning is not about pressing needs but rather abstract and mainly a response to people claiming “just implement a rate-limit, then everything is solved”.
Ich habe diese Tage erst gesehen, dass das BSI im Dezember eine Studie zur Verarbeitung von Log- und Monitoringdaten [Link aktualisiert, 12.8.09] veröffentlicht hat. Der Text ist geeignet um sich in das Thema einzulesen; es werden einige grundlegende Probleme erläutert und verbreitete Produkte kurz vorgestellt. Allerdings liegt der Fokus nicht allein auf Open-Source-Lösungen; es geht um heterogene Daten und (auch) um “die großen” Monitoringprogramme OpenView und Tivoli, die für mich bisher kein Thema waren.
Für alle, die eher im Bereich “ein paar Unix-Server” denken verlinke ich hier auch meine Folien über Syslog (pdf) vom CLT 2007. ;-)
Und weil’s so schön ist noch ein zweiter Patch für pflogsumm: Diesmal um in den Fehlerlisten nur signifikante IPs/Domains anzuzeigen und nicht x-Tausend Einträge mit jeweils einem Fehler zu haben.