RTC SDK v3.76: Detailed stress-testing Screenshots and all Test Results

[D.Tkalcec (RTC)]

Here are a few Screenshots of a Server showing how the RealThinClient SDK 3.76 is being stress-tested by simulating 9.900 concurrent clients, where each Client is sending a remote function call to the Server, waiting for the result and then sending the next call in a continuous loop, flooding the Server with requests.

What you see below are 11 Client PCs, each running 3 instances of the AppClient demo, each AppClient instance using 64 threads in MultiThreaded mode to send remote function calls through 300 concurrent connections to the Server, which is also running in MultiThreaded mode and using 64 threads to handle all connections simultaneously.

Each vertical line on each AppClient instance represents a connection to the Server, where the red color shows the percentage of calls to be made in the current loop and the green/cyan color shows the percentage of results received in the current loop. AppClients are working in multi-threaded mode and are set up to use different settings, while every connection is set up to make 50 remote calls to the Server per loop, and each loop is repeated one second after the last one was finished, which results in an endless loop of remote calls (until stopped):







The small windows titled ibm-1 to ibm-c are remote desktop views from 11 Client PCs, displayed in real-time by using Nexus Portal (Nexus Portal Gateway, Control and Host compiled with RTC SDK 3.76).

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

All 11 Client PCs are running in MultiTreaded mode, each using 3 instances of the AppClient demo compiled with the RealThinClient SDK 3.76, each AppClient using 300 connections to send remote function calls to the Server, wait for a result and then send the next call and so on, and so on - until I remove the "auto-repeat" checkbox.

Clients ibm-1 to ibm-4 are using the RTC SDK with asynchronous (non-blocking, message-based) WinSock API and StreamSec Tools 2.1.9.240 for SSL-TLS encryption, Clients ibm-5 to ibm-8 are using the RTC SDK with synchronous (blocking) WinSock API and StreamSec Tools 2.1.9.240 for SSL-TLS encryption, Client ibm-a is using the RTC SDK with the blocking WinInet API and built-in WinInet SSL support, while Clients ibm-b and ibm-c are using the RTC SDK the WinHTTP API and build-in WinHTTP SSL support. All Clients are running inside the same LAN as the Server.

The Server is using the RTC SDK in MultiThreaded mode with the asynchronous (non-blocking, message-based) WinSock API and StreamSec Tools 2.1.9.240 for SSL-TLS encryption.

I have set up this stress-test more than 24 hours ago (see timestamp on the "DEBUG.log" file in the folder on the Screenshot). Until now, there have been no errors on either Client or the Server. All Nexus Portal Hosts have remained connected to the Gateway running on the Server and the Control has continued updating remote screens of all Hosts at near real-time. There were no errors on either client nor the Server and memory usage is stable on all Clients as well as the Server.

As you can probably see from the Screenshots, the Server has responded to over 24 million requests from Clients in this test for which over 330.000 connections have been opened and closed by Clients.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

 Here are two new Screenshots from today (taken a few minutes ago):





RTC AppServer and all the AppClients are still running at full speed without any problems.
Portal Host clients have also remained connected to the Portal Gateway and Portal Control.

This stress-test has now been running for more than 50 hours, during which the RTC AppServer has responded to more than 150 million requests and AppClients have opened and closed over 900 thousand connections to the Server.

Best Regards,
Danijel Tkalcec 

[D.Tkalcec (RTC)]

And ... two more Screenshots from today:





Now the stress-test is running more than 72 hours (3 days) without a break and without any problems. No errors, no stopped clients, everything works as it should. So far, the RTC AppServer has responded to more than 165 million requests and AppClients have opened and closed over 1,1 million connections to the Server. In the last 24 hours, all AppClients have been using complex structures (over 100KB per request).

Best Regards,
Danijel Tkalcec 

[D.Tkalcec (RTC)]

And here are 4 new Screenshots from today:









The RTC SDK 3.76 stress-test has now been running for five full days, during which time I have been changing Clients test parameters at least once a day. The last change was a few hours ago, when I've changed 2 out of 3 AppClient instances on each Client PC to use the standard XML-RPC protocol instead of the RTC protocol for remote functions.

As you can probably see from the last Screenshot above, the RTC AppServer has processed over 267 million requests in the last 5 days, for which AppClients have opened and closed over 1,9 million connections. So far, all Clients as well as the Server are running without any problems and without creating any error logs. Would there have been exceptions on either Client or the Server, error logs would have been created by the problematic App because all the test RTC Apps have been compiled with madExcept and the RTC_DEBUG compiler directive declared.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Two Clients, one using the WinInet API and the other one using the WinHTTP API, got me worried a bit because there were a few connections which seemed to not be "moving" in the last two Screenshots, so I went back to check again an hour later and now these "late connections" have also moved forward.

Below are two new Screenshots from a few minutes ago. If you are wondering which Clients I am talking about, look closer at miniaturized windows titled IBM-A and IBM-B in the last two Screenshots above and look at the same windows below, you will see how there are a few long and unchanging red lines in the two Screenshots above which have changed in the Screenshots below:





I guess, I am getting a bit too jumpy towards the end of this bug hunting event, starting to see bugs where there are none.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

The RTC SDK 3.76 stress-test is now getting close to 6 (six) full days of running time (no errors nor any other issues) and here are three new Screenshots from today:







As you can probably see from the Screenshot above (RTC App Server demo window), the RTC App Server demo has processed more than 281 million requests through more than 2,2 million connections since this test was started. Memory usage on the Server and all the Clients is stable (it does not appear as if there were memory leaks).

In about 24 hours tomorrow, once this stress-test has been running 24/7 (seven days straight), I will be finalizing this test and publishing detailed log files from all Clients and the Server. Unless there will be a problem during shutdown tomorrow, all Clients and the Server will then also be creating a detailed memory leak report. Even if there will be no memory leaks, a memory leak report will still be created because a single TForcedMemLeak object is created but not destroyed on purpose, to force a memory leak and thus force generation of a memory leak report.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

And here are two new Screenshots from today, taken roughly 6 days and 13 hours after the test was started ...





As visible on the last Screenshot above, the RTC AppServer has responded to over 345 million requests from Clients, which have opened and closed over 2,5 million connections. There are no errors on either of the Clients and Memory usage also looks stable, but we will know for sure only after finalizing the test later today (in about 10 hours).

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Here are the last five Screenshots from this stress-test, showing the process of finalizing all Clients test loops and closing the Server ...











This 24/7 stress-test of the RealThinClient SDK 3.76 is now completed.

There were NO errors throughout the whole test, there were NO problems closing any of the RTC Clients nor the RTC Server and there were a absolutely NO memory leaks on either of the Clients nor the Server (memory leak report generated by FastMM4).

Detailed log files from ALL test Clients (IBM-1 to IBM-8 and IBM-A to IBM-C) as well as the test Server (SRV) can be downloaded HERE (a 23,5 MB zip file, CLICK to download)!

Important thing to note here is that "madExcept" was used on all the Clients and the Server throughout this stress-test with "detect application freezing" option enabled and a "freezing alarm" set to 60 seconds. Because some of the functions on the Client took longer to complete than 60 seconds (for example opening 300 connections, preparing 300 complex remote function calls and sending them to the Server), "madExcept" has created a few "bugreport.txt" files containing snapshots of some AppClient instances at times when their main thread was busy for longer than 60 seconds, but the madExcept report dialog was automatically closed once madExcept has realized that the main thread is still running normally.

I have decided to include "bugreport.txt" files even though they do not contain real bug reports (only false warnings about the application appearing to be frozen, which were proven to be wrong after the app has continued normally), because I found it interesting to see a complete snapshot including call stacks of all running threads, which clearly shows that the Clients were running in multithreaded mode and that most threads were busy opening connections, preparing new remote function calls, sending data, receiving data or processing received results. Would there have been any real bugs, there would have been errors in RTC log files, the Clients would not have continued running, there would probably have been a real exception entry in the "bugreport.txt" file and a memory leak report would either have been missing or it would contain leaked objects.

In this stress-test, the RTC Server has responded to more than 394 million requests from RTC Clients, which have opened and closed more than 2,7 million connections to the Server. I have been changing test parameters on all Clients at least once a day, so that most RTC SDK properties and settings have been tested for at least 24 hours each.

Best Regards,
Danijel Tkalcec