https server not responding

[Henrick (StreamSec)]

@Henrick - It seems to be triggered by connections from the same external IP. We know who the users are so it is not something they are doing wrong - they are requesting the front page, using chrome mostly. Can you think of a browser TLS configuration that might intermittently cause this?

Did you ever test with the ST 4.0 trial? Please do.

FWIW this looks like a thread issue. The ST 2.x implementation will sometimes misbehave when the same tlslayer instance is used from multiple threads, when socket closure is triggered multiple times, etc.

[Bryn Lewis]

Did you ever test with the ST 4.0 trial? Please do.

Not yet, because I can't switch a production server without testing etc.

FWIW this looks like a thread issue. The ST 2.x implementation will sometimes misbehave when the same tlslayer instance is used from multiple threads, when socket closure is triggered multiple times, etc.

Any advice on how to mitigate these issues? The server is multithreaded so it seems likely this might happen.

[D.Tkalcec (RTC)]

Here is a "quick fix" for problems caused by multi-threading ...

If you can change your Project to run as a normal Windowed Process instead of a Service, you could disable the thread pooling mechanism used by the RTC SDK by setting MultiThreaded:=FALSE on the TRtcHttpServer component. This would eliminate threading issues with components which might misbehave in a multi-threaded environment, but it won't work from a Windows Service if you do NOT create a message queue in the Main Thread (which is created by default in normal Windowed Projects, but NOT in a Windows Service).

Alternatively, you could change RTC_THREAD_POOL_MAX and RTC_THREAD_POOL_LIMIT variables to 1 (use the rtcThrPool unit) before calling "Listen" on the Server. This would limit the number of threads used by the RTC Thread Pool to 1 and should also eliminate issues with components and code which doesn't work well when used from multiple threads.

Both of these options would make all your events triggered by RTC components run from a single thread (serialized, one at a time), so keep that in mind if you have any longer-running events, because these will block everything else until they are finished.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Is it possible that your problem with multi-threading in a Service using StreamSec Tools 2.x is somehow related to this?

The topic above is rather old (from 2013) and talks about problems with StreamSec Tools 2.1 in a multi-threaded RTC Server running as a Service, but since the issue you are having seems to be related to multi-threading, unless you are already creating StreamSec components from the Main Thread (as suggested in the topic above by Henrick), you might want to try that and see if it makes a difference.

Best Regards,
Danijel Tkalcec

[Bryn Lewis]

IOW, are you creating StreamSec components from the Main Thread, as Henrick suggested in his reply in that topic?

Thanks, but I am creating the components in the main thread already.

@Henrick - I am trying to install the StrSecIV_DXE8 tools, but am getting StrSecIIMobile not found when compiling rtcSDK_STMT?

[Henrick (StreamSec)]

Setting your application to single thread would probably work, but is not strictly necessary. ST 2.x has been successfully used with RTC SDK in multi thread setups in the past, so the issue I am guessing might occur here is probably more esoteric.

For instance, do the close calls on specific sockets ever occur in a different thread from the other calls on the same socket? Such as, is there a controller thread for time outs that calls directly on the sockets, rather than send a message to the socket thread that it should close the socket?

[Henrick (StreamSec)]

@Henrick - I am trying to install the StrSecIV_DXE8 tools, but am getting StrSecIIMobile not found when compiling rtcSDK_STMT?

As always, the requires clause of the plugin package has to be edited prior to installation. The package to be required is StrSecIV220, the case of Delphi XE8. There is further information here https://support.streamsec.net/forum/?cmd=viewtopic&topic_id=7&section_id=3 and here https://support.streamsec.net/forum/?cmd=viewtopic&topic_id=2&section_id=3.

You also have to set ST40 as a conditional define, either in project options, or directly in the plug in unit.

[D.Tkalcec (RTC)]

@Henrick: You are right. I forgot to ask which RTC SDK version is being used.

In some older RTC SDK versions, when using RTC components in Multi-Threaded mode, RTC Timer events (for example, to trigger automatic disconnects) were being triggered in a separate "Timer Thread", which could have resulted in the same RTC connection component to be accessed from multiple threads at the same time. I've fixed that in one of the later RTC SDK updates, so this should NOT be the case anymore with the latest RTC SDK release.

Using RTC SDK v8.14 (2017.Q2) and newer, all Timer events are posted to the "virtual thread" associated with the connection for which they were triggered, then executed in the context of that "virtual thread". This ensures that any components used by the same connection (as well as events executed on the same connection) will NEVER be used from more than one thread at a time. There is no guarantee about the actual Worker Thread used to execute each "job", though, which means that one "job" could be executed from one Worker Thread, but once that "job" finishes (and returns), the next "job" could be executed from the same or any any other Worker Thread.

Best Regards,
Danijel Tkalcec

[Bryn Lewis]

I am using v8.19. The structure is based on the rtcForum demo, so if there is anything in there that might cause this it could be the culprit.

Given there are a small number of users, but they can be simultaneous, I don't want to reduce it a single thread. Would it be best to reduce the RTC_THREAD_POOL_MAX and RTC_THREAD_POOL_LIMIT variables to a smal number (4?), to reduce the likelihood of sockets being open/closed in different threads. At the same time, I don't want performance to be adversely effected too much.

thanks, Bryn

[D.Tkalcec (RTC)]

I don't remember who wrote the WebForum Demo (it wasn't me), but I do remember that it was done by a developer who didn't have prior knowledge of the RTC SDK. As I remember, he was learning to use the RTC SDK while writing that Demo, so ... I wouldn't dismiss the possibility that parts of that code aren't thread-safe. The same goes for the WebPackageManager Demo (it was written by the same developer).

There are actually quite a few Projects which were implemented by other developers and later included as part of RTC Demos and Examples. Like QuickStart examples "rtcParse1" - "rtcParse3", which were written by the same developer who wrote the original TRtcParse class, BCB* examples were converted from Delphi by another developer, and ... the Remote Function Wizard was written by Glynn Owen, who was a part of the "RTC Team" about 10 years ago.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Even using 2 threads is multi-threading, so ... if you want to eliminate all issues caused by code which isn't thread-safe, you have to limit the number of threads used by that code to only 1.

If you are using the latest RTC SDK version, you don't have to worry about sockets being accessed from multiple threads at the same time, but only about code which isn't thread-safe and about code which uses the ThreadID to decide in which "thread" it is running.

Anyway ... unless there are long-running events (your own code which takes a long time to finish), your users most likely won't even notice a difference between a Multi-Threaded and a Single-Threaded RTC Server, since everything is split into small jobs, most of which take a few milliseconds to finish and jobs from all sockets are being executed with the same priority, so all Clients will get their share of a CPU time on the Server, even if only 1 "Worker Thread" is used.

The advantage of using multiple threads is that longer-running code can run in its separate thread and doesn't block the rest. But if all the jobs your Server does are short, it's not really a big deal if your Server is using only one "Worker Thread".

Best Regards,
Danijel Tkalcec

[Henrick (StreamSec)]

Given there are a small number of users, but they can be simultaneous, I don't want to reduce it a single thread. Would it be best to reduce the RTC_THREAD_POOL_MAX and RTC_THREAD_POOL_LIMIT variables to a smal number (4?), to reduce the likelihood of sockets being open/closed in different threads. At the same time, I don't want performance to be adversely effected too much.

There is one thing. Seems there is a change to rtcSTMTPlugin and rtcSSecPlugin I am using internally, but doesn't seem to have been propagated to RTC SDK

Code:

function TRtcSSecPlugin.BeforeDisconnectEx(var ConnCryptObj: TObject;
   var OutData: RtcByteArray): TRtcCryptPluginState;
var
  lObj: TRtcSSecConnCryptObject;
  lRes: TRtcByteArrayStream;
  lTLS: TCustomTLS_ContentLayer;
begin
  Result := cpsReady;
  if Assigned(ConnCryptObj) then begin
    lObj := ConnCryptObj as TRtcSSecConnCryptObject;
    lTLS := lObj.fTLSLayer;
    lObj.fTLSLayer := nil;
    if Assigned(lTLS) then begin
      lRes := TRtcByteArrayStream.Create(nil);
      try
        lTLS.Close(lRes);
        OutData := lRes.GetBytes;
      finally
        lRes.Free;
      end;
      lTLS.Release;
    end;
    FreeAndNil(ConnCryptObj);
    if Assigned(FTLSServer) then
      FTLSServer.ClearExpiredSessionKeys(False);
  end;
end;

Code:

function TRtcSTMTPlugin.BeforeDisconnectEx(var ConnCryptObj: TObject;
   var OutData: RtcByteArray): TRtcCryptPluginState;
var
  lObj: TRtcSTMTConnCryptObject;
  lRes: TRtcByteArrayStream;
  lTLS: TCustomTLS_ContentLayer;
begin
  Result := cpsReady;
  if Assigned(ConnCryptObj) then begin
    lObj := ConnCryptObj as TRtcSTMTConnCryptObject;
    lTLS := lObj.fTLSLayer;
    lObj.fTLSLayer := nil;
    if Assigned(lTLS) then begin
      lRes := TRtcByteArrayStream.Create(nil);
      try
        lTLS.Close(lRes);
        OutData := lRes.GetBytes;
      finally
        lRes.Free;
      end;
      lTLS.Release;
    end;
    FreeAndNil(ConnCryptObj);
    if Assigned(FTLSServer) then
      FTLSServer.ClearExpiredSessionKeys(False);
  end;
end;

[D.Tkalcec (RTC)]

Thanks, Henrick. I must have missed an E-Mail from you with these updates (I'm getting way too much spam these days). Anyway ... I've updated these files now and released them in the RTC SDK v8.27 (see Downloads area).

Best Regards,
Danijel Tkalcec

[Bryn Lewis]

@Henrick - Tried using StreamSec4 with TRtcSTMTServerPlugin

First question - if the pfx has no password, do I pass an empty password, or nil?

smSimpleTLSInternalServer1.ImportFromPFX(global.APP_PATH+'Server.pfx',nil);

OR

PW:=TSecretKey.CreateBMPStr('');smSimpleTLSInternalServer1.ImportFromPFX(global.APP_PATH+global.sServerCertificate,PW);


Using the below, I get a handshake failure in StreamSec4 (XE8):

  //Used to do this with streamsec2.3 - not sure if need with 4?
  //while not MPYarrow.YarrowHasReseeded do begin
  //  Sleep(100);
  //  Application.ProcessMessages;
  //end;

    smSimpleTLSInternalServer1.ImportFromPFX(pathToPFX,PW);

  if pkaRSA in smSimpleTLSInternalServer1.PublicKeyAlgorithms then
  begin
    smSimpleTLSInternalServer1.Options.SignatureRSA := prPrefer;
    smSimpleTLSInternalServer1.Options.KeyAgreementRSA := prAllowed;
    smSimpleTLSInternalServer1.Options.KeyAgreementDHE := prPrefer;
    smSimpleTLSInternalServer1.TLSSetupServer;
  end;

Using the same certificate in StreamSec2.3 (d7), it works. What is the problem for XE8?  Is it the options being chosen? I initially went with the default options, then tried changing them to match what was working in the ss2.3 version - still doesn't work though.

thanks, Bryn

[Henrick (StreamSec)]

First question - if the pfx has no password, do I pass an empty password, or nil?

If the PFX truly doesn't have a password, it means it has a very unusual configuration and can't have any password encrypted components. In such case you can't use the ImportFromPFX method, which expects the kind of PFX files you get when exporting certificates and private keys from Microsoft software. Instead, you have to use the classes and interfaces in the StreamSec.DSI.Pkcs12 unit. It is doable, but outside of the scope of this discussion.

If you rather mean that the password is an empty string, you pass in a tSecretKey instance with the password set to the empty string.

Needless to say, you really shouldn't be doing either of those things in neither demo code nor production code.

Using the below, I get a handshake failure in StreamSec4 (XE8):

You will get handshake failures if the server certificate PFX hasn't been properly loaded server side. For instance, if it contains a 1024 bit RSA key and the LeastKeyBitSize property is set to 2048, it won't load properly.