Comments on: reliable TCP reconnect made easy

By: jdmurray

jdmurray — Mon, 20 Oct 2008 19:17:01 +0000

For syslog TCP reliability, would it not be easier to just prevent TCP from buffering the outgoing syslog messages? This would allow the syslog client to immediately know if the syslog message was sent successfully or not by the send exception thrown. The app itself would then keep track of the “first unsent” message.

I have implemented a syslog TCP client using C# and .NET 2.0 that sends event log messages to a syslog TCP server. If the TCP connection is broken, the next syslog message sent will cause the TcpClient object to return a Winsock error (10061 or 10054) and my app can then keep track of which event messages were not sent until a new connection can be established. It works very well.

By: Rainer Gerhards

Rainer Gerhards — Mon, 09 Jun 2008 10:06:02 +0000

Hi Martin,

some more quick feedback: I have now implemented your approach in rsyslog, it will come out as part of 3.19.7. Initial testing confirms the findings – it works well in local, low-traffic environments, but still fails in others. However, even in the fail cases, the message loss is somewhat less than without the recv(). So I think it is useful to have this work-around in the code. Thanks for pointing it out :)

Rainer

By: Rainer Gerhards

Rainer Gerhards — Fri, 06 Jun 2008 10:10:19 +0000

Hi Martin,

I just wanted to let you know that I have written a final “analysis”: http://blog.gerhards.net/2008/06/getting-bit-more-reliability-from-plain.html . In short, I’ll see that I give it a try in rsyslog, too. It’s not a perfect solution (what on earth is that…), but it looks like it offers sufficient benefit to be worth implementing ;)

Thanks for bringing it up.

Rainer

By: Rainer Gerhards

Rainer Gerhards — Mon, 02 Jun 2008 13:40:34 +0000

OK, I have done some further stress testing. Unfortunately, the solution does not work as soon as the traffic gets higher. In client.c, you do a sleep. This sleep makes sure that the remote end has time to do its close and notify the client before the next message is sent. I modified you examples so that the sleep() is no longer used and 1000 messages are sent by the client. Then, the disconnect happens 1 byte into message 423 (stream was partly received). The next message I see is 701. The client also notices the problem at 701. So I have lost around 280 messages without anybody noticing it. Of course, on a non-busy server that does not happen, but it proves the point that even in a nice clean close scenario without packet loss you can lose data on a server restart. There is no reliable cure. I’ll now evaluate if it makes sense to add your suggested recv() logic for those cases where it helps. I am just unsure if these are a large enough sample to justify this code change. Anyhow… Please let me know any further findings you may have (did I overlook something)?

Rainer

By: Rainer Gerhards

Rainer Gerhards — Fri, 30 May 2008 13:02:42 +0000

Interesting. I’ve now done a wireshark trace. As it looks, the IP stack does NOT process the incoming connection request until after it actually sends the message. If you do the recv(), this forces this evaluation. I have not verified this with the tcp sources and I don’t think it makes much sense to do so. It’s probably a bit platform-specific, depending on what the IP stack does. Anyhow, it seems to greatly reduce the problem in non-failure cases and does not cause much performance pain. So it is worth adding it. I’ll now apply it once again to the real rsyslog and see how it performs there. I keep you posted. Rainer

By: Rainer Gerhards

Rainer Gerhards — Fri, 30 May 2008 12:34:36 +0000

OK, I am not just ignorant ;) I’ve given it another try, while racy it looks like it indeed solves the problem in the majority of simple cases. I just wonder what happens under the hood. I’ll have a look :) — Rainer

By: Rainer Gerhards

Rainer Gerhards — Fri, 30 May 2008 09:55:57 +0000

Martin, I have read your update. This use case isn’t fixed either, at least not reliably. It is racy. You get the same results be just doing the send() without the recv() – just give it a try. ;) — Rainer

By: Rainer Gerhards

Rainer Gerhards — Thu, 29 May 2008 15:37:18 +0000

OK, I have updated my last blog post with references to RFC 793, which provides clear evidence that there is no way around this problem (at least for a compliant implementation). Rainer

By: Rainer Gerhards

Rainer Gerhards — Thu, 29 May 2008 15:13:14 +0000

Martin, I saw you post from noon today only after posting my comment. You are now using two error states:

74 } else if (rc == -1 && errno == ECONNRESET) {
75 /* server closed connection */
76 printf(“got ECONNRESET, reconnect\n”);
77 connected = 0;
78 } else if (rc == -1 && errno == ENOTCONN)

However, you do get ECONNRESET and ENOTCONN also from the send() call, but ONLY after the TCP stack has detected that the connection is broken. I propose this test: make the client connect to the server. Pull the network cable from the server. Make the client continue to send some data to the server. The client will be unable to detect the server failure for “n” records. Plug in the network cable to the server. Server will now receive the “lost” messages. So the client was right in assuming the server was available.

Another test: connect to server as before. Do not pull the network cable but switch the machine off (do NOT shut it down). Client will not see any problems. Problem will be seen only after server comes back online (TCP connections can be reset at this point).

Rainer

By: Rainer Gerhards

Rainer Gerhards — Thu, 29 May 2008 15:04:52 +0000

I am sorry, but it doesn’t work out. I have now also been able to define why it is impossible to do reliable delivery over plain tcp without app-level acks: http://blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html

Rainer