EventSource / TRtcHttpServer Client Disconnect Detection

[peteness]

I'm adding support for Javascript EventSource on my RTC-based HTTP server.  EventSource needs a long-running connection, and uses the standard HTTP protocol to negotiate.  When the server wants to tell the client about something, it pushes a message over the client connection.  I'm doing this in the httpProviderDataReceived event for the http server something like this (phsudo-code):

        S.Response.ContentType := 'text/event-stream';
        S.Response.AsString['Cache-Control'] := 'no-cache';
        S.WriteHeader(TRUE);
        S.Write('event: startup'+#13#13);
        S.Response.Sending := TRUE;
        S.Flush;

        while (S.Request.Active) and (not S.isClosing) // <- Problem here - what to look for?
        do begin
          // Wait for notifications read, then push it back to the client.
          EventHandles[0] := HasDataUpdatesEvent.Handle;
          EventHandles[1] := HasStatusUpdatesEvent.Handle;
          case WaitForMultipleObjects(2, @EventHandles, FALSE, 10000) of
            Windows.WAIT_TIMEOUT: begin
              s.Write('event: heartbeat'+#13#13, TRUE);
            end;
            Windows.WAIT_OBJECT_0: begin
              S.Write('event: update_data'#13#13);
              HasDataUpdatesEvent.ResetEvent;
            end;
            Windows.WAIT_OBJECT_0 + 1: begin
              S.Write('event: update_status'#13#13);
              HasStatusUpdatesEvent.ResetEvent;
            end;
          end;
        end;

The problem here is that I can't figure out a way to determine that the client (web browser) has closed a connection.  The check in the RTC code is much too deep to get access to any variables that show disconnect, and no events ever come back.  I'm assuming that I'm doing something wrong in my setup or ordering, but I can't figure it out by looking at the RTC code.

Also, this is very threaded (the only way to keep a long-running request open) - and thread-safe (even if some of the test code above doesn't look entirely thread-safe).

Any guidance on how I might do this?  This is on a shared HTTP connection that also serves up other pages, so I can't just replace this with a full-on self-managed TCP connection.

Thanks (and Happy New Year!),
Pete

 

[D.Tkalcec (RTC)]

Short answer:

Don't use "EventSource". While it might work in controlled environments like your LAN, it is not Proxy-friendly, it is heavy on Servers like Apache and IIS (no ISAPI support) and it does not work on all Web Browsers. It also does not give you any kind of feedback about communication success or failure, because the Client isn't expected to give a response to the Server.

Long answer:

You can not have loops like this inside RTC events. The event has to process the data it receives, send a response (if there is some) and complete execution. What you are doing can not work, because your loop inside the RTC event is blocking the RTC code from continuing its execution. This is why you don't get any events about changes in the connection status.

Would you have find a way to implement this by using a loop inside a RTC event (which is a very bad idea), because you are blocking a thread with your connection, you would end up with a Server refusing to accept futher connections once the number of open connections reaches your thread limit. Because TCP/IP is a protocol where packets are allowed to get "lost" and there is no mechanism in TCP/IP to ensure that a "disconnect" packet is re-sent, your Server would easily end up blocking all your threads while waiting on a lot of dead connections and refusing to handle requests from your "live" Clients.

If you wanted to send a response from the Server outside of the RTC event (so there is no blocking loop inside the RTC event), you could use the "PostEvent" method of the TRtcConnection component to post a message to the thread pool and execute your custom event from the context of a connection thread, which is what Load Balancer and Router components use to send requests and responses asynchronously when data is available.

But with "EventSource", you still have the problem that a connection might get closed by the Client or a Proxy without a "close connection" packet reaching the Server. With long running connetions and no requirements for the Client to contact the Server between each response, chances are high that you will be sending responses to Clients which are no longer there. And because there is no built-in mechnism in TCP/IP to let the Server know if a response was received by the Client, your Server runs a risk to continue "sending" data in to nirvana for a long time, before it "realizes" that there is nobody listening.

Other than this, a protocol like "EventSource" is very heavy on Proxy Servers and most likely won't work with some Proxies (connections will often be closed by the Proxy), so even if you did make it work in your test environment, chances are high that you will start encoutering problems with customers behind corporate firewalls and Proxy Servers. With a standard request/response protocol, a single physical connection can be shared by multiple clients, and a new connection can be opened as needed when requests need to be sent to the Server. Proxies make heavy use of this functionality by reusing open connections.

Best Regards,
Danijel Tkalcec

[peteness]

Thanks for the detailed reply.

Your concerns with EventSource and ws:// style-callbacks are valid, although in this case it's primarily a LAN-based system, so some of the concerns of proxies and lost connections aren't as big as they might be on a WAN or internet-based system.

The client is primarily Javascript running on IOS or Android devices.  This was previously working using comet-style (long-polling) calls from the client to the server with the server returning a response as soon as an action happened where the client would update something.  This worked fine until iOS6, where the long-poll action often blocks other http requests on the web browser.

Thanks for the advice - I'll try and go after the problem from the angle you're suggesting (PostEvent) and see if I can get things working a bit better.

- Pete

[D.Tkalcec (RTC)]

If you are unable to get enough active connections from iOS6, I'm afraid the idea to use "EventSource" won't solve your problem. If long polls block other communication, then using "EventSource" will have the exact same effect.

If you are limited to using a single connection for all communication, I recommend making periodic polls from the Client in shorter intervals. When a Client makes a poll request, if the Server does not have anything to return and nothing gets prepared within a short time interval (a second or two), the Server should return an empty result. Provided you have already allowed the Server to return with no data to the Client, there is no need for changes on the Client-side. You only need to reduce the timeout on the Server which you have reserved for waiting on new data.

Best Regards,
Danijel Tkalcec

[peteness]

If you are unable to get enough active connections from iOS6, I'm afraid the idea to use "EventSource" won't solve your problem. It will actually make your situation worse. If you are limited to using a single connection for all communication, I recommend making periodic polls in shorter intervals instead of long polls. If long polls block all the other communication, using "EventSource" will have the exact same effect.

Indeed - you're correct (until you try things, it's difficult to know what barriers still need to be overcome!).

However, using the HTML5 WebSocket *seems* (still needs testing) to work around the problem - at the cost of some complexity.  The upside of WebSockets is that there can be two-way communication between the server and the client.  I'm sure there are some unseen gotchas out there still, though...

Thanks,
Pete

[D.Tkalcec (RTC)]

If you want to use WebSockets, you will probably have to use raw TCP/IP components (TRtcTCPServer) instead of HTTP/s (TRtcHttpServer), because WebSockets are only using HTTP headers to initiate the communication, but after the initial handshake, the protocol would have to change to raw TCP/IP and this is something the HTTP/S components don't expect.

Best Regards,
Danijel Tkalcec