Server stopping responding

[ultralogic]

Hi,

 We have setup a simple web server which makes database requests using Remote Functions. Everything runs fine during tests but when clients get above 1500 - 2000 connecting to server becomes a bingo. Pure luck. Would this be caused by threads being ocupied ? We get lots of connection problems .

Best regards,

Daniel Bichuete

[ultralogic]

Another question would be about upgrading my recently bought plan to Gold subscription. Would it be possible without waiting next year ?

Best regards,

Daniel Bichuete

[D.Tkalcec (RTC)]

When there are no free threads from the RTC thread pool to handle connection requests for a relatively long time (time depends on WinSock API), new connection requests will get "refused".

If your Server needs to handle thousands of active connections and the events you wrote on the Server (for example, the OnExecute event to handle remote function calls) take a rather long time to complete, there are two things you can do:

1) If your Server has a lot of free CPU time and enough memory to handle all these clients, simply increase the RTC thread pool size (see the "rtcThrPool.pas" unit for all variables you can set in your code).

2) If your Server is under heavy load and increasing the RTC thread pool size does not help, you will have to use more Servers (install the Server app on more PCs) and place a Load-Balancer before them so that all traffic from Clients will first go to a Load Balancer and then be forwarded to one of your Servers (depending on their load).

PS. For questions about subscriptions, please contact me directly by E-Mail:


Best Regards,
Danijel Tkalcec

[ultralogic]

Hi,

 The weird behavior is that server is huge (Quad Xeon with 8 GB), threads are setup to astronomical values 2048. Still getting no answer when running remote functions. Component throw an erro like if none is listening on other side. Any ideas ?

Best regards,

Daniel Bichuete

[D.Tkalcec (RTC)]

1) Sorry, I thought you were using the default values. Windows has a thread pool limit. If you try to create more threads than Windows can handle, Windows will stop working properly.

As far as I know, Windows can handle 2000 threads max. Since you should also leave some space for other applications, you should keep the max thread count in your Server below 1000. But you also need to keep in mind that each active thread takes CPU resources and more active threads will reduce your overall performance, so you should try to keep the thread number as low as possible.

2) Since your application is 32-bit, it won't have access to more than 3 GB RAM and that ONLY if you use a Memory Manager which can use it. Most Memory Managers will stop at 2 GB RAM.

Also - the more threads are active, the more RAM will be used at the same time. So ... you might have to try running the Server with default thread pool size or reduce the thread pool size gradually until you find a value that works best for you.

More active threads means more memory usage but higher connection acceptance. If your application has high CPU or memory requirements, you might not be able to find a setup where a single Server will be enough and should consider using a Load Balancer.

Best Regards,
Danijel Tkalcec

[ultralogic]

This is one behavior that I was troubleshooting just now. Running from local development machine worked fine, getting up to the server it did not respond anymore even with few clients.Tested with about 100 connected clients and with 3000. Everytime it did same way. Tried diagnosing from another station and no answer too.

Every request is made from RtcHttpClient to one RtcHttpServer. Memory usage was pretty low, around 400 MB, and CPU at 18%. We have plenty room for more clients. We have put the threads on 768 and still got same issue and could not see any increase on connected clients limit.

Best regards,

Daniel Bichuete

[D.Tkalcec (RTC)]

Is your Server PC accessible over the Internet and is the firewall on the Server side configured to allow incoming connections to the Port where your Server application is listening?

If RTC Client and Server applications work inside LAN, they will also work over the Internet - provided there are no firewalls or blocking proxies between them. But you need to configure the router on the Server side to forward incoming connections to the Port you are using on the Server application (usually 80 for HTTP, 443 for HTTPS).

Also ... if you have installed the Server application as a Windows Service and you are using configuration files to set it up (especially the Port number), make sure you are looking for the file at the correct location. Applications running as Services have different default paths, so you should always use absolute paths.

The easiest way to test if a Server is accessible would be to add a simple TRtcDataProvider to the Server and implement it to respond to a specific request. Then you can use any browser to check if your Server is accessible. If it isn't, check the router or firewall at the Server side and check which Port the Server is using.

PS. Use the "Log"() or "xLog()" procedures (rtcLog.pas unit) to write logs during Server startup, shutdown and other important actions - so you will have a better idea about what is happening on the Server. If you haven't done so already, you might also want to log "Sender.TotalConnectionCount" in OnConnect and OnDisconnect events (TRtcHttpServer component) - just so you know how many clients are currently connected.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

If none of the above mentioned helps, please answer these questions:

1) How much bandwidth does your Server have to the Internet?

2) How many Client *PCs* were you using to access the Server? Was each Client application running on a separate PC or were you emulating a lot of Clients using only a hand full of PCs?

3) How much dedicated bandwidth does each of your Client PCs have to the Internet?

4) What do your Clients do when accessing the Server over the Internet? Are they operated by people using your application, or are they running automated flood tests?

Best Regards,
Danijel Tkalcec

[ultralogic]

After using a load balancer component to distribute traffic across some servers we could get rid of such errors. It appears that since we are using database intensively on each remote function (SQL Server 2008), threads get stuck during some time. So using 512 thread per server and two RTC Servers per physical server show as a good choice. CPU is nice and RAM is really low, around 300MB per server.

I just try to reach your example with 9900 concurrent connections, but I think they did not use database, did they ? Any recomendations when using SQL Server ?We are using throw AnyDAC.

PS: Clients are running one system that each 30s contacts server looking for information. It's a biometric control access system.
Best regards,

Daniel Bichuete

[D.Tkalcec (RTC)]

I'm glad the problem was solved.

The amount of load a single Server will be able to handle without losing connections or getting in trouble because of high RAM usage depends on what the Server does. In your case, it looks like your Server is spending most of its time waiting for a response from a database, which is why it needs a lot of threads to process all the requests.

Now that you have placed a Load Balancer before your Server(s), you should be able to reduce the number of threads on the Server. I would recommend you to go back to the default value (64 threads) and increase the number of threads in the RTC thread pool only if necessary. Less threads means faster code execution and smaller load on your database.

PS. I hope you are using a database connection pool and are NOT creating and destroying database connections inside every remote function call. If you aren't using a database connection pool, you will be wasting a lot of time just opening and closing database connections and your response times will be a lot slower.

Best Regards,
Danijel Tkalcec