SSL protocol error and certificate problems on Apple products

[thomh]

Issue
A WebServer service running on Windows Server 2019 with all updates using RTC 9.3 and Streamsec Crypt plugin components generates “could not establish a secure connection” in Safari on any Mac device and “ERR_SSL_PROTOCOL_ERROR” in Chrome or Mozilla Firefox on any Mac device after a period of normal behavior. However all PC and Android devices perform as they should using any browser. So the issue is Mac OS specific.
 
On start of the service a TRtcHttpServer component is set to Listen. All browsers running on MacOS, PC and Android work and behave as expected. However, after a number of hours/days depending on request load a “limit” or something is reached and all MacOS device browsers stop working at the same time.
 
This appears to have started since Mac/Apple certificate/TLS changes last September, based on no evidence of the issue prior to then.
RTC 9.3 and Streamsec have been in use for a considerable period of time in multiple locations using the same certificate.
The certificate is issued by DigiCert. “Chain certificate” issues do not seem relevant as the certificate works perfectly for a period of time before failing.
 
Temporary solution
At the moment stopping the service and restarting resolves the issue. Effectively that takes the TRtcHttpError object out of listen mode and then returns to listen mode on restart. After restart all device types work as they should. Several hours/days later Mac devices fail again.
 
The issue has been discussed with Streamsec and there is suspicion that the RTC framework has changed how “BeforeDisconnectEx” is executed in the crypt plugin. Tests at Streamsec revealed problems with Mac devices in RTC v8 and RTC v9. But RTC v7 worked correctly.
 
Feels like some sort of leak related to Mac device connections.
 

[K. Okada (RTC)]

We had experienced a similar problem.

While runnig a RTC + StreamSec IV based web server for a long period (about a couple of week or so), apple Safari started to fail connecting to the web server, but Chrome/IE on Windows still could connect.

With the help of a packet capture software (Wireshark), we found that Safari browsers on Apple devices and browsers on Windows devices use different cypher suite set.

What cypher suite is used is decided during a negotiation between the web server and the browser.
The server offers a set of available cypher suites to the browser, and the browser chooses one from the list.

Safari prefers, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384   ECDH secp521r1
Chrome/IE prefers TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384   ECDH secp256r1

We thought that there'd be some resource leak in the secp521r1 implementation, and avoided using this by setting
Options.EphemeralECDHKeySize := ecs256  for the SimpleTLSInternalServer object .
and this workaround cleared our problems away successfully, so far.

But, be careful that if you use StreamSec IV , version older than 4.0.1.318. You should add
      D.EphemeralECDHKeySize := FEphemeralECDHKeySize;
to procedure TTLSOptions.AssignTo(Dest: TPersistent) function in StreamSec.Mobile.StreamSecII.pas file,
otherwise, setting EphemeralECDHKeySize might not work.

[K. Okada (RTC)]

But this experience was in several years ago, so it cannot be related to a change in last  September  directly
Maybe your new SSL server certificate has different structure than your older certificate, which result in using a different code with apple browsers.

[D.Tkalcec (RTC)]

I've taken a closer look at the way RTC SDK works with SSL/TLS Encryption Plugins, and ... I might have found a potential problem. If I am right, then there might also be a simple fix for that problem, which you could apply to your RTC SDK copy. All you need to do, is open the "rtcSocketSrvProv.pas" unit in the RTC SDK "Lib" folder, search for the "TRtcSocketServerProvider.CryptPluginError" implementation and comment out the line that sets "Lost:=True", after which the final method implementation should look something like this ...

Code:

function TRtcSocketServerProvider.CryptPluginError:boolean;
  begin
  Result:=True;
  // Lost:=True; <- COMMENT THIS LINE OUT
  InternalDisconnect;
  // raise Exception.Create('Encryption error.');
  end;

Let me know if that helps.

So ... why do I think that this little change might help?

I think that, for whatever reason, one of the events of the ST encryption plugins returns "cpsClosed" as a result, which signals RTC that the underlying TCP/IP connection should be closed. After that, RTC sets the "Lost" flag on that connection to TRUE, which prevents it from sending or receiving more data, assuming that the connection is no longer valid. But ... before the TCP/IP connection is closed by RTC, a call to BeforeDisconnectEx event will be made to the ST plugin, from where ST can prepare some data to be sent back to the Client, using the "s_out" parameter. And I think this might be exactly what ST is doing. It is probably preparing a packet for the Client, with a notification about a SSL/TLS connection closing, probably expecting this packet to go out to the Client, before the TCP/IP connection closes.

With the original implementation in RTC SDK (which has NOT changed between RTC SDK v7 and the latest version), the "Lost" flag was set to TRUE after an encryption plugin event returned with "cpsClosed", because of which any data that might have been prepared by the plugin in its "BeforeDisconnectEx" event, was NOT sent out before the Server closed its TCP/IP connection. As a result, the Client was left with a TCP/IP connection closed by the Server, without receiving a notification about the SSL/TLS stream closing. What happens next, depends on the Client implementation. My assumption is that Apple has recently made changes to their SSL/TLS layer, which keeps that connection open and waiting, while other implementations simply close it. And I think that this change by Apple, is now causing problems with all clients running on Apple devices that use their SSL/TLS implementation.

Best Regards,
Danijel Tkalcec