Server Lockup Issue

[seiapplication]

I'm using remote procedure calls with http client/server and I'm having a problem with the server locking up.  The lockup interval is random.  Sometimes it happens in less than an hour and other times it will go a day or more.  The memory consumption is consistent and the cpu utilization is normal.  An exception is not being raised when the server locks up in the IDE.  I've turned on rtc debug logging, but the log files haven't revealed any errors.  Any suggestions on how I should proceed to isolate the issue?

Thanks,

Kevin

[D.Tkalcec (RTC)]

Unfortunaltely, your description does not tell me anything about a possible cause for your problem, but a "deadlock" is usually a result of trying to acquire a lock on two different critical sections from two different threads in a different order. In other words, when Thread 1 acquires a lock on A and tries to lock B, while Thread 2 has already acquired a lock on B and tries to get a lock on A. In this situation, Thread 1 and 2 will end up in a deadlock.

1. Which RTC SDK version are you using?

2. Which Delphi version are you using?

3. Which platform (Windows, MacOSX, ...) is the Server running on?

4. How are TRtcHttpServer and TRtcServerModule components configured (published property values)?

5. Are you using remote functins with "Delayed Calls"?

6. Does your Server require access to one or more shared resource? If yes, have you secured access to those resources with one or more critical sections or some other exclusive-access-mechanisms (events, mutexes or semaphores)?

7. Is your Server running 100% in the background without any access to the GUI or do some of your functions modify elements on a Form? If your functions access the GUI, are you using the Sync() method to synchronize access to GUI with the Main Thread?

Best Regards,
Danijel Tkalcec

[seiapplication]

1. We are using the latest version of RTC SDK.

2. XE7

3. Windows

4. (I will provide properties later this evening)

5. Not using delayed calls.

6. It accesses database, but each session has it's own database connection.

7. Yes.  We check "InMainThread" property and call Sync() if necessary.


Thanks! 

[D.Tkalcec (RTC)]

6.A) What do you mean by "Session"? How exactly are you managing Database connections?

6.B) How many Clients does a Server manage and how long does a Client usually have an open "Session" on the Server?

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

If the lockup you are experiencing is not a result of wrong critical section use (see my initial reply), the only other thing that comes to mind which could result in a situation resembling a "lockup" are requests or remote functions processed by the Server which take a very long time to execute or never complete.

Every request and remote function occupies one thread from the RTC thread pool while being executed. If the Thread Pool is not allowed to grow and all the threads from the pool are in use, the Server will stop accepting new connections and processing requests from other connected Clients until at least one thread becomes available again.

All RTC Thread Pool parameters are defined in the "rtcThrPool.pas" unit. The most important parameter there is "RTC_THREAD_POOL_MAX", which is set to 256 by default. That should be more than enough for a very busy Server, handling thousands of Client connections. You can modify that value in your Project before starting the Server Listener to see if it has any effect on your problem. Reducing the maximum thread pool size should lock the Server up sooner, while increasing the value would make the Server last longer. When changing RTC Thread Pool parameters, please keep in mind that Windows has a limit of about 1020 threads, which are shared by all the running processes, so you shouldn't go too high with this number, or you will end up with other kinds of problems.

RTC also comes with simple thread-safe logging functions, available in the "rtcLog.pas" unit (see "Log" and "XLog" procedures there), which you could use to log function entry and exit points if you need to find out which one of your functions take too long to execute or never completes, probably causing this "lockup" issue.

If nothing helps and you need my assistance, then I will need a lot more details about the way you are using the component, or ... an example Project using only the RTC SDK and standard Delphi components which can be used to reproduce the problem, so I can debug it on my machine.

Best Regards,
Danijel Tkalcec