Load Balancer with SSL

[Dany]

Hello!

First off i need to say that i have been working with the TRtcDataRouter and TRtcLoadBalancer components and it is a very enjoyable experience.

I want to create a service that is very similar to the LoadBalancer3 demo. The "balancer" is actually a router that uses the "VirtualPath" property of the TRtcLoadBalancer to decide where to route the traffic. All "routed to servers" honours the same virtual path for all requests and responses (root URI and VP are the same for each server /server1/, /server2/ and so on).

All servers also uses a cookie for the session id. They may set different cookies at any time during a session (ie they may exchange the coockie for another one within the same logical "session" at any time, any roundtrip, any method. This works excellently when accessing one of the servers directly and it works when using the LoadBalancer3 demo "out of the box".

However, the servers are "exposed" (they are not residing on an internal net) so i need to use SSL between the router and the servers. Thus i change the code in the sample LoadBalancer3 like so:

Code:

procedure TRtcLoadBalancerMainForm.LoadBalancer_PrepareConnection(Sender: TRtcDataClient);
  begin
  if Sender is TRtcHttpClient then
    with Sender as TRtcHttpClient do
      begin
      MultiThreaded:=FCliMultiThreaded;
      Blocking:=FCliBlocking;
      UseSSL:= true;  // *** ADDED
      FixupRequest.ForceOldHttp10:=FCliFixupRequest;
      end;
  end;

But now the cookies are "blurred". Session handling on the respective servers are rendered void. Some cookies get through others not. The result is more than one cookie, sometimes none at the clients (webbrowsers).

I should add that the servers use StreamSecII for server side SSL and that the certificates they have does not match the IP (self signed) - the certs are only for encryption. Also that all session events in the demo are removed and when testing i use the following settings:

Code:

/server1/ 0 0 0 89.200.100.10:443/server1/ 0 0 0

Any ideas? If it turns out that clients side SSL can not be used this way i need to find another solution. However, if scrutinized by a client, they will not accept unencrypted traffic between the router and the servers.

TIA,

/Dany

[D.Tkalcec (RTC)]

In order to use SSL with the Load Balancer, you need to use StreamSec Tools components with TRtcDataClient components for SSL encryption and leave the "Blocking" property FALSE. Load Balancer can ONLY work with TRtcHttpClient components in asynchronous WinSock mode, but NOT in blocking WinSock (used when Blocking=TRUE), WinInet (used when useProxy=TRUE) or blocking WinHTTP (used when useWinHTTP=TRUE).

Best Regards,
Danijel Tkalcec

[Dany]

In order to use SSL with the Load Balancer, you need to use StreamSec Tools components

My apologies for missing that fact.

I have now incorporated StreamSecII on the client side, modelled on your File_Client demo. I had to disable AES-CBC on the servers to make it work at all. That flag is global so it affects the router too, but that is for another topic entirely.

I have two different test setups and two different executables to test with. It seems the session problems that i wrote about initially has gone away. But the result is even worse than using "UseSSL".

In one test setup where i use virtual machines (extremely fast connections) on my workstation requests get trough more than in the "real word" test setup where the router is on one virtual server and the servers are on another (two parallel web-apps). But there are lots of lost packets. The router reports rows in the timeout log.

In order to eliminate my own misdoings i have two executables. The one i'm working on (Router) and the LoadBalancer3 demo with added Client side SSL.

Here's some test results from the "real world" test-setup (80, 90 is without SSL and 443 and 444 is with StreamSecII):

LoadBalancer3 Demo (modified)
IN: 80, OUT: 80, 90	Works perfect, sessions, uploads, downloads
IN: 80, OUT: 443, 444	Sessions seems to work (better), gets through very seldom, only "first" couple of requests seems to make it - unusable
Router
IN: 80, OUT: 80, 90	Works perfectly, sessions, uploads, downloads
IN: 443 (RC4), OUT: 80, 90	Works, sessions, uploads get stuck*, downloads - won't pass
IN: 80, OUT: 443, 444	Sessions seems to work (better), gets through very seldom, only "first" couple of requests seems to make it - unusable

* This works when testing with virtual machines, not with the "real" servers.


The two virtual servers are running around the clock now so testing is possible. How do i go about making this work?

TIA,

/Dany

[D.Tkalcec (RTC)]

Are you using a separate SSL plugin component for each RTC connection component? One for each TRtcHttpClient (when using SSL on the Client side) and one for each TRtcHttpServer component (when using SSL on the Server side)?

From RTC point of view, adding SSL encryption through 3rd-party plugins is a pass-through process for all incoming and outgoing data. In other words, all the received data goes through the SSL plugin for decryption before being used by HTTP-related code in RTC, and all the HTTP data from RTC will go through the SSL plugin for encryption before sending it out. How the actual encryption works is handled entirely and transparently by the SSL plugin.

If you have set up everything correctly but using SSL still does not work, please contact StreamSec support for help.

Best Regards,
Danijel Tkalcec

[Dany]

Are you using a separate SSL plugin component for each RTC connection component? One for each TRtcHttpClient (when using SSL on the Client side) and one for each TRtcHttpServer component (when using SSL on the Server side)?

All my servers has only on TRtcHTTPServer instance serving via SSL so server side should be ok.

I'll dig into the sources to see how i can create one client instance per rtc component.

Thanks for your prompt answer,

/D

[Dany]

Daniel, i'm looking more closely at the rtcSSecTest.pas unit and i do not get how to create a" separate SSL plugin component" for each connection.

Do you mean that i should re-write the GetClientCryptPlugin function to create one instance of TSsPrivateKeyRingComponent, TSimpleTLSInternalServer and TRtcSSecClientPlugin per TrtcHTTPClient instance?

What am i missing?

TIA,

/Dany

[Dany]

I rewrote the rtcSSecTest.pas to produce one instance per call and rooted out all the globals. There does not seem to be any change. I'll take this to SteamSec. Thank for all your help so far.

/D

[Henrick (StreamSec)]

I'll have a look at this problem and will try to find a solution. However, it seems to be related to the nature of SSL/TLS compared to the logic of the load balancing client. SSL/TLS requires seperate states for each TCP/IP connection, and it is essential that packets are processed in the same order they were sent. If packets get queued in anything but a FIFO queue, or if packets arriving from different TCP/IP connections get mixed up before they have been processed, then the SSL/TLS handling will, by design, inevitably fail.

[Henrick (StreamSec)]

Some comments:

• There should be no reason to use separate TRtcSSecClientPlugin objects for different clients. Using a single component is fine, since it separates the connections with ConnCryptObj that RTC SDK couples with a unique TCP/IP connection. There is also no reason to remove the event handler from rtcSSecTest, because that event handler does all SSL/TLS related logging and is your best shot at figuring out what is happening.
• However, there might be reasons to modify rtcSSecTest to set Client_SimpleTLSInternalServer.LockAlways := True and Server_SimpleTLSInternalServer.LockAlways := True. This is in particular the case if you are mixing blocking and non-blocking sockets.
• Obviously, you must make sure that both end points of each connection both use SSL/TLS or both don't use SSL/TLS, and if they do, that their certificates match. If the server side uses a self-signed certificate, that certificate should be added as a root certificate client side of the connection.
• There is nothing in the SSL/TLS handling that might cause blurred or missing cookies. At the plugin level, however, this might hypothetically happen if the RTC SDK layer passes in a ConnCryptObject corresponding to a TLS object that has already been destroyed. The TLS object is normally destroyed when the connection is closed, but might also be prematurely closed if there is a TLS error (such as failure to verify the remote certificate), in which case the RTC SDK layer must react by also closing the connection and destroying the ConnCryptObject.

[Dany]

Henrick!

it seems to be related to the nature of SSL/TLS compared to the logic of the load balancing client

If this is the case, maybe i'm going about things the wrong way. I could in stead use VPN, implement an encryption of my own (but that seems a bit over the top) or move to a provider that offers "private networks".

My "LoadBalancer" is actually a router. There's only one receiver per "virtual path". Also, the dropping of session information happened with "UseSSL" not with StreamSecs TLS client.

I will work through my setups with regard to your much appreciated input here.

Again, thanks!

/Dany

[Henrick (StreamSec)]

Just to rule out that I have misunderstood exactly what you are trying to do:

You have a client, a router/load balancer, and two back end servers. The client connects only to the router/load balancer. The router/load balancer forwards the request to either of the back end servers, and returns the response from the back end server to the client. Is that correct?

In such case, there is no reason why it wouldn't work to have SSL/TLS only for the client<->router/load balancer connections. Adding SSL/TLS will slow down the transfer some, but not more than what a VPN would do, and the network latency is likely to have a greater effect.

[Dany]

Hmm... perhaps.
M1, M2... are different machines. Could also be on the same machine with different ports.

Code:

Client [many, browsers] ==>
 Router [one, M1] ==>
  Server [M2]
  Server [M3]
  Server [M4]

The virtual path of the client will decide what Server to target. Servers are aware of the virtual path so there's not trickery going on with changing requests in route at all.

The problem is using SSL/TLS between the Router and the Server(s).

Another problem that was mentioned is that when SSL is used for INcoming to the Router uploads to server(s) get stuck. But i think i need to concentrate on one thing at a time.

Perhaps it's a bad idea to use SSL between Router and Servers is my main question now.

I have tried using the same root cert, setting LockAlways to true (though there's no blocking involved).
The results are "patchy" connections when router is tested in a VM and one server listening on the host. This usually gets lot worse when i move to test "in the cloud", ie with real machines with real network cards. But i get no warnings or fatals now. When running with RTC_DEBUG i get reports in a log called TIMEOUT from the router and server both.

TIA,

/Dany

[Henrick (StreamSec)]

What you do should work. There might of course be time outs if the load is too great. There are more initial round trips with SSL/TLS, so it is generally best for the router to keep the connections to the back end servers open and queue the requests that get passed to them. Opening a new connection for each request will consequently degrade performance a lot more in the HTTPS case, than it does in the HTTP case.

[Dany]

I have decided to leave this issue for the time being. I need to concentrate on functionality for a while. I simply can not get the RTC Client-RTC Server encryption working. I'd like to thank you both for the input and your efforts. I will have to get back to this eventually so no efforts have been in vain.

I have now moved around some stuff and for now the Router communicates with the Severs(s) without SSL.

[Dany]

Hello!

I have a very painful problem. If you have access to StreamSecII or Eldos you can reproduce it like so:

Compile the "BrowserUpload" project from the QuickStart directory.

Open the "LoadBalancer3" demo from the Demos directory. Add the following to the end of the TRtcLoadBalancerMainForm.FormCreate procedure:

Code:

    // Set to false to disable RC4 and enable AES-CBC.
  ExpectOldBrowsers := false;

    //  Files will be IGNORED if they do not exist.
  AddServerRootCertFile(
    ExtractFilePath(AppFileName) + 'root.cer');
  AddServerPFXFile(
    ExtractFilePath(AppFileName) + 'server.pfx',
    'yourCertsPasswordIfProtected');

  try
    Server.CryptPlugin := GetServerCryptPlugin;
  except
    On E: Exception do
      XLog(E.Message);
  end;

and add rtcSSecTest unit to Unit1.pas.

Compile this project too.

Move the two exe-files to a machine with port 443 open.

In order to make this fail it seems that it should be a machine on "the net". If the connection is too fast it does not fail as often or as "reliably". Thus making this issue very difficult to pursue in a test-setup of course.

Start the executables. The BrowserUpload should already listen.

Do not change any settings on the LoadBalancer3 form except for having one row in the server list that reads:

Code:

/upload/ 0 0 0 localhost:80/upload/ 0 0 0

and change the port for listening to 443. Then start the balancer.

Access https://theserver/uploads/ from a browser and upload a medium sized file. You will see how browsers that count stops at a percentage relative to the file size.

I have put up such a server at https://89.221.253.82/upload/ and you are welcome to try and upload something. I'm interested to know if it fails from any machine or just some. I have tried most things and googled my brains out and there seems like there are potentially a lot of factors that can cause this.

A note on that server is that you have to accept the server cert since it does not match the ip. I have tested with a legit cert on another machine with the same results. Another not is that the demos are not compiled into services so if the machine has restarted for some reason you will not be able to get the BrowserUpload "welcome" page at all. Just post here and i'll restart things.

Any input is appreciated, TIA.

/Dany

Server: Windows Server 2012 - One network card
Delphi: Delphi XE2 - Updated
RTC: v6.31 (2014.Q1)
StreamSecII: 2.2.0.254