2 Sep 2017 • Detecting TCP server crashes

I was wondering what happens when an HTTP server gets killed in the middle of a request. The OS should close all the open sockets, but what happens on the client? Can it tell that the server was shutdown forcefully, or does recv return 0 like it does for a normal shutdown?

I was wondering specifically because my HTTP client doesn't look at the response headers, and I wasn't sure if that would lead to problems down the line.

The Arch/OpenBSD man pages don't have anything to say about it. The junky and often outdated die.net man pages talk about ECONNRESET, and SO has a few questions that mention it, but nowhere else does. Grepping the OpenBSD kernel sources doesn't make it obvious.

So let's just test it. The client code is trivial and of no use to you because it's written against my libraries. On the server we can do nc -l 13337 and then pkill -9 nc from another terminal. The result:

socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(13337), sin_addr=inet_addr("108.61.209.87")}, 16) = 0
recvfrom(3, "", 16, 0, NULL, NULL)      = 0

It looks just like a normal shutdown! And that's a bug in my HTTP client: I don't look at Content-Length so truncated responses are not considered an error.

I guess the bigger picture here is that TCP is not as high level as you might expect. Your protocol has to be able to distinguish between premature and intentional shutdowns. Somewhat related is that you can't tell when the other party has processed your TCP packet. ACK just means it's sitting in a buffer somewhere in the TCP stack, and there's no guarantee that the application has actually seen your data yet. So if your application need acks, you have to add them to your protocol too!

(The even bigger picture is that if you trust the other party at all, it will bite you)