freeze on disconnect

[Director]

In unstable internet link, our application freezes on disconnects.

  Stack trace in attach. You may see, that last function of our
  application is _Connect in TRtcSocket.Sock_Connect.

  It is called internally by your components, so we have no idea how
  to control this behaviour. Please, fix this if it is an error in
  your components, or tell, how we can control rtc behaviour to avoid
  freezes.

first freeze log:

Module = $13140000 to $001EC000;
Base = $00130000; Limit = $0012C000:
Return Addr $71A356E8 in Module mswsock.dll
Return Addr $71A355AF in Module mswsock.dll
Return Addr $71A3542C in Module mswsock.dll
Return Addr $71A940BD in Module ws2_32.dll at Function WEP + $00000053
Return Addr $1321723A in Module Terminal.exe
Call Addr: $13217235
Function TRtcSocket.Sock_Connect + $00000061
 at synsock\winapi.inc[170]
[169]
[170]   if Sock_Err(_Connect(FSocket, Sin, SizeOfSockAddr(Sin))) then Exit;
[171]

Return Addr $1321783A in Module Terminal.exe
Call Addr: $13217835
Function TRtcSocket.Connect + $00000029
 at rtcSynAPI.pas[278]

Return Addr $1321A0D3 in Module Terminal.exe
Call Addr: $1321A0CE
Function TRtcSynSocket.api_Listen + $00000002
 at rtcSynSocket.pas[139]
[138] function TRtcSynSocket.api_Listen: boolean;
[139]   begin
[140]   Need_HSock;

Return Addr $13219712 in Module Terminal.exe
Call Addr: $1321970D
Function LoopMsg + $0000003D
 at rtcSockBaseSyn.pas[1037]
[1036]         Result:=Exec_Message(GetMsg);
[1037]         until Result or MsgQ.Empty;
[1038]     finally

Return Addr $1321978C in Module Terminal.exe
Call Addr: $13219787
Function TRtcSocketBaseSyn.MsgLeave + $0000003B
 at rtcSockBaseSyn.pas[1047]
[1046]       if LoopMsg then
[1047]         Free;
[1048]   end;

Return Addr $132196CE in Module Terminal.exe
Call Addr: $132196C9
Function TRtcSocketBaseSyn.MsgEnter + $00000001
 at rtcSockBaseSyn.pas[1027]
[1026]   begin
[1027]   Inc(MsgLevel);
[1028]   end;

Return Addr $1321BFA6 in Module Terminal.exe
Call Addr: $1321BFA1
Function TRtcSocketClientProvider.OpenConnection + $00000001
 at rtcSocketCliProv.pas[709]
[708] procedure TRtcSocketClientProvider.OpenConnection(Force:boolean;Reconnecting:boolean);
[709]   begin
[710]   if (State=conActive) or (State=conActivating) then Exit; // already connected !!!

Return Addr $13274C5C in Module Terminal.exe
Call Addr: $13274C57
Function TDataCommon.StartMainConnection + $00000067
 at ..\Common Files\uDataModuleCommon_Term.pas[670]
[669]            FSpecConnect:=True;
[670]            HTTPClient.Connect
[671]           end

-------------------------------------------------------------------------------

rest of stack is in our application.
HTTPClient.Connect called from main (GUI) thread.
HTTPClient: TRtcHttpClient; // component on form

second freeze log:

Module = $13140000 to $001EC000;
Base = $00130000; Limit = $0012C000:
Return Addr $71A356E8 in Module mswsock.dll
Return Addr $71A355AF in Module mswsock.dll
Return Addr $71A3542C in Module mswsock.dll
Return Addr $71A940BD in Module ws2_32.dll at Function WEP + $00000053
Return Addr $1321723A in Module Terminal.exe
Call Addr: $13217235
Function TRtcSocket.Sock_Connect + $00000061
 at synsock\winapi.inc[170]
[169]
[170]   if Sock_Err(_Connect(FSocket, Sin, SizeOfSockAddr(Sin))) then Exit;
[171]

Return Addr $1321783A in Module Terminal.exe
Call Addr: $13217835
Function TRtcSocket.Connect + $00000029
 at rtcSynAPI.pas[278]

Return Addr $1321A0D3 in Module Terminal.exe
Call Addr: $1321A0CE
Function TRtcSynSocket.api_Listen + $00000002
 at rtcSynSocket.pas[139]
[138] function TRtcSynSocket.api_Listen: boolean;
[139]   begin
[140]   Need_HSock;

Return Addr $13219712 in Module Terminal.exe
Call Addr: $1321970D
Function LoopMsg + $0000003D
 at rtcSockBaseSyn.pas[1037]
[1036]         Result:=Exec_Message(GetMsg);
[1037]         until Result or MsgQ.Empty;
[1038]     finally

Return Addr $1321978C in Module Terminal.exe
Call Addr: $13219787
Function TRtcSocketBaseSyn.MsgLeave + $0000003B
 at rtcSockBaseSyn.pas[1047]
[1046]       if LoopMsg then
[1047]         Free;
[1048]   end;

Return Addr $132196CE in Module Terminal.exe
Call Addr: $132196C9
Function TRtcSocketBaseSyn.MsgEnter + $00000001
 at rtcSockBaseSyn.pas[1027]
[1026]   begin
[1027]   Inc(MsgLevel);
[1028]   end;

Return Addr $1321BF3F in Module Terminal.exe
Call Addr: $1321BF3A
Function TRtcSocketClientProvider.Connect + $0000007A
 at rtcSocketCliProv.pas[695]
[694]     begin
[695]     if assigned(Client_Thread) then
[696]       TRtcThread.PostJob(Client_Thread, Message_WSOpenConn)

Return Addr $1320950C in Module Terminal.exe
Call Addr: $13209507
Function TRtcClient.TriggerReadyToRelease + $00000003
 at rtcConn.pas[3577]
[3576] procedure TRtcClient.TriggerReadyToRelease;
[3577]   begin
[3578]   if InsideEvent then

Return Addr $132084FA in Module Terminal.exe
Call Addr: $132084F5
Function TRtcClient.OnReconnect_Timer + $000000A9
 at rtcConn.pas[2694]
[2693]     try
[2694]       TriggerReconnect;
[2695]     except

Return Addr $13205CC7 in Module Terminal.exe
Call Addr: $13205CC2
Function TRtcTimer.Timer + $0000007A
 at rtcTimer.pas[320]
[319]           end
[320]         else if assigned(FThr) then
[321]           begin

Return Addr $13205F01 in Module Terminal.exe
Call Addr: $13205EFC
Function RtcTimerWindowProc + $00000040
 at rtcTimer.pas[417]
[416]     try
[417]       TRtcTimer.Timer(Obj);
[418]     except

Return Addr $13202026 in Module Terminal.exe
Call Addr: $13202021
Function RtcMainHWNDWindowProc + $00000071
 at rtcHWndPool.pas[125]
[124]           Result:=proc(ahWnd,auMsg,awParam,alParam);
[125]           Exit;
[126]           end;

Return Addr $77D38709 in Module user32.dll at Function GetDC + $00000072
Return Addr $77D387EB in Module user32.dll at Function GetDC + $00000154
Return Addr $77D389A5 in Module user32.dll at Function GetWindowLongW + $00000127
Return Addr $77D3BCCC in Module user32.dll at Function DispatchMessageA + $0000000F
Return Addr $131AE2E8 in Module Terminal.exe
Call Addr: $131AE2E3
Function TApplication.ProcessMessage + $00000083
 at DECData.pas[5042]
[5041]
[5042] end.
[5043]

Return Addr $132B9E58 in Module Terminal.exe
Call Addr: $132B9E53
Function Terminal + $000000FF
 at D:\....\Terminal.dpr[126]

Return Addr $7C816D4F in Module kernel32.dll at Function RegisterWaitForInputIdle + $00000049

[D.Tkalcec (RTC)]

The stack trace you've just posted does not tell me anything. If you think there is a BUG in the RealThinClient SDK and you are certain that you are doing everything right, please send a Delphi project which reproduces the BUG using only standard Delphi components and the RealThinClient SDK, along with a short BUG description and ways to reproduce it, to:


Without a project which shows how to reproduce the problem you are having, I can only make wild guesses but I can NOT help you solve the problem.

Best Regards,
Danijel Tkalcec

[Director]

Dear Danijel,
Except the callstack and our project we have nothing. They are repeated irregular and in conditions that we dont understand. We dont manage to repeat them by ourselves.
It is very diffcult and almost impossible to make deug-version (retrive allocated code) from big project. We hoped that we would manage to determine the couse on the callstack and call sequence.
Maybe you will tell us in what direction we should act.

Best regards,
Anatoly

[D.Tkalcec (RTC)]

What I can tell you without seeing your code is that the RTC SDK will NOT be doing anything on its own. If a method is executed in the RTC SDK, it means that you have set up one of the RTC components to act this way, or your project is making a method call into the RTC SDK. In either case, the code executed is called because you've told the components to do so.

For example, if you set the AutoConnect property on a TRtcHttpClient component to TRUE, you are telling the RTC SDK to open a new connection automatically when a connection is required. This also means opening a new connection if a Request is waiting on the Request queue while you call Disconnect from another thread.

Another important thing to note is that there is a big difference between "freezing" and "long delays". For example, if you do not explicitly set "TimeoutsOfAPI" values for your connection components, default WinSock API values will be used for every WinSock call, which could mean *very long* waiting times in case a connection can not be established. And if your connections are slow and unstable (as you've mentioned earlier), it is possible that your application is not frozen, but that it simply takes a very long time to connect.

Also, when setting Timeout values, especially when working with slowand unstable connections, you have to be careful to NOT set the Timeout value too short, or the Client might not be able to get a connection at all, because a connect attempt would always time-out too soon. If you want failed connect attempts to timeout sooner, you need to set a high enough Timeout value to make sure the connection can get established in that time period, but short enough for your customers to not go crazy in case the Server is overburdened and unable to accept new connections.

Anyway ... before I continue theorizing, I need to see how you are using the components. If you can not localize the problem in your own Project for which you have full source code, including full RTC SDK source code, how am I supposed to do it without even having a clue about what you are doing?

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

By the way ... if I interpret your 2 call stacks correctly (I am now assuming that these are 2 call stacks of the Main Thread and not current execution points inside multiple running threads), then it looks like your application is NOT frozen, but is simply waiting for the WinSock API call "Connect" to return.

At least from the call stack, I don't see anything wrong. A call to "Connect" would definitely become a problem if you would have set the component up to automatically try to reconnect infinite number of times to a bad IP Address or bad Port number or to a Server which became inaccessible for whatever reason, but the call stack you've posted does not tell me if that is the case or not.

If AutoConnect is FALSE and you are manually opening the Connection by calling Connect, "ReconnectOn" property will still be used to auto-reconnect every time a connection gets closed and if you are NOT closing the connection manually when you do not need it, this could be an even worse solution that using AutoConnect, because then you would be telling the RTC components to force the connection to stay open and reconnect every time it gets closed for what ever reason.

Well, as said, I can only make wild guesses about what you are doing. If you have absolutely no idea where or why this "freezing" is happening and your project is too big for you to debug or to localize the problem, please answer a few questions so I can get a general idea about your problem ...

How many Server listeners (TRtcHttpServer) and how many Client connections (TRtcHttpClient) are actively used in that application and how are these components set up (their property values)?

Are other threads running when the application appears to be frozen?

Can you send me (E-Mail) the complete call stack of ALL running threads at the time the application appears to be frozen?

Are you using "madExcept" or "EurekaLog" with their "detect application freezing" enabled to get a stack trace of your running process, or how did you retrieve that stack trace? If you are using "madExcept" or "EurekaLog" with ther "detect application freezing" feature, what time interval did you use? In other words, how long can your applications main thread remain busy before it as declared "frozen"?

Did you try leaving the application running for a few some time to see if it will "un-freeze"?

Which Windows version is used on the PC where the application appears to freeze randomly?

Best Regards,
Danijel Tkalcec

[Director]

Hello,

Thanks for your answer.
First, some notices:

N.0. Yes, your understanding is right, it was standart stack traces of one (frozen) thread. All other threads are function normally. Two examples of ST is maked in different times.
N.1. It is long delays, definitely. Not a infinite freeze, but very long to wait (we never wait so long to see, will it unfreeze or not).
N.2. Freezes appears only at unstable link to server.
N.3. It is really hard to localize problem, to make demo project. If I will have demo project, it means that I already solved the problem =)
N.4. I meet this problem (very long wait) first time, because in winsock, for example, call to connect is never waits more than minute (even if it is problems with network).

Let me answer to your questions:

A.1. The AutoConnect is set to false, the event OnReconnect is nil.
A.2. Application is client side, so this is used: 0 TRtcHttpServer and 2 TRtcHttpClient. Their props:

Code:

 object HTTPClient: TRtcHttpClient
   Timeout.AfterConnecting = 32
   OnConnect = HTTPClientConnect
   OnDisconnect = HTTPClientDisconnect
   OnException = HTTPClientException
   ReconnectOn.ConnectError = True
   ReconnectOn.ConnectLost = True
   ReconnectOn.ConnectFail = True
   ReconnectOn.Wait = 5
   OnConnectFail = HTTPClientConnectFail
   OnConnectLost = HTTPClientConnectLost
   OnConnectError = HTTPClientConnectError
   OnBeginRequest = HTTPClientBeginRequest
   Blocking = True
   TimeoutsOfAPI.ConnectTimeout = 32
 end
 object HTTPEvents: TRtcHttpClient
   MultiThreaded = True
   Timeout.AfterConnecting = 32
   OnConnect = HTTPClientConnect
   OnDisconnect = HTTPClientDisconnect
   OnException = HTTPClientException
   OnConnectFail = HTTPClientConnectFail
   OnConnectLost = HTTPClientConnectLost
   OnConnectError = HTTPClientConnectError
   OnBeginRequest = HTTPClientBeginRequest
   TimeoutsOfAPI.ConnectTimeout = 32
 end

A.3. Other threads definitely runnung. At least, stack dump thread is running. It seems, that freeze appears only in main (GUI) thread.
A.4. It is hard to send you an other threads stack traces. And they will not contain interest information. If you wish, I can get stack trace of some certain thread, but not all threads.
A.5. We are not using "madExcept" or "EurekaLog". It is self-written mechanism, separate thread, that dumps stack if some thread freezes. Time interval before stack will be dumped is equal to 16 seconds (now 48 seconds, when I set RTC timeouts to 32s). But freeze time is much more.
A.6. Not long, but not less than 1-2 minutes.
A.7. WinXP SP2.

Have any ideas?

[D.Tkalcec (RTC)]

Looking at call stacks and your "HTTPClient" and "HTTPEvents" component properties, I would say that your problem is related to how you've set up your "HTTPClient" component.

It is using blocking WinSock (Blocking=True) in single-threaded mode (MultiThreaded=False) with a Connect timeout (Timeout.AfterConnecting) of 32 seconds and all "ReconnectOn" parameters set to TRUE with a "Wait" period of 5 seconds between each renewed connect attempt, which means that each call to Connect from the Main Thread will block your Application (it will appear frozen) until a connection is established, or ... if a connection can not be established within your specified timeout (32seconds) WinSock will abort its attempt but because of your "ReconnectOn" settings (all parameters TRUE), a Timer component will be created to make a new connect attempt 5 seconds later, which will be repeated infinite number of times or until you remove all requests from the request queue and call Disconnect to stop further connection attempts. But since your connection is Blocking and single-threaded, doing this could be difficult since the user would need to act during the 5 second delay between connect attempts, or a new connect attempt will be made which will probably take about 32 seconds or longer (depending on your timeout parameters and WinSock) if there are problems connecting to the Server.

Another problem with the scenario you've created in your application is that every time you use the "HTTPClient" component, you will be blocking the main thread (application appears "frozen") and any Synchronized calls from a background threads will be forced to wait until the main thread is idle again. Synchronized calls can be made from events triggered for the "HTTPEvents" component if you use the "Sync" method, or for every Result you receive after a remote function call made using a TRtcClientModule component linked to the "HTTPEvents" components if you have "AutoSyncEvents" set to TRUE on the TRtcClientModle component.

What I'm wondering now is why none of your components have "AutoConnect" property set to TRUE, but the Blocking component has all "ReconnectOn" parameters set to TRUE. First thing I would recommend you to do is stop making Connect and Disconnect calls manually and set the AutoConnect property to TRUE for both of your TRtcHttpClient components. When AutoConnect is TRUE, "ReconnectOn" parameters will be used to re-try connecting ONLY if a connection is really required. Also, in case a connection is closed, a new connection attempt will be made automatically if a request is waiting in the request queue (for example, when you make a remote function call).

On the other hand, when AutoConnect=FALSE (as in your case) and you have already called Connect, "ReconnectOn" properties will be used to force the connection open until every time a connection gets lost - until you manually call Disconnect. And that is a very bad idea in scenarios where your connection is slow or unstable. Forcing a connection open when there is nothing being sent nor received is generally a bad idea.

I would also like to know why you are using 2 connection components set up with completely different properties? Yes, you can do this as there are no restrictions in the RTC SDK which would disallow that, but I don't see the point.

Do you know that you can make blocking remote function calls (using the "Execute" method) even when the Blocking property of the connection component is FALSE and also when the MultiThreaded property is TRUE? It will result in a different behavior, with a message queue running while waiting for a result, but if part of your code is event-driven, using a blocking connection in single-threaded mode is pretty much the worst thing you can do to your application.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

I've just checked the implementation in the RTC SDK and I have to say that there is an error in RTC SDK *documentation* for the "TimeoutsOfAPI.ConnectTimeout" property, where it says that this property is available for WinInet, WinHTTP *and* WinSock APIs.

Unfortunately, the only "TimeoutsOfAPI" values you can set when using WinSock are "SendTimeout" and "ReceiveTimeout". All other Timeout values are available ONLY when you are using the WinInet or WinHTTP APIs (and some only for WinHTTP). This is NOT a RTC SDK limitation, but a WinSock API limitation.

When using non-blocking WinSock, you can use the "Timeouts.AfterConnecting" property to force a disconnect if a connect attempt takes longer than you would like to wait. But when using blocking WinSock, I'm afraid there is no way to explicitly set a connect timeout, which means that any application using the Blocking WinSock API will have to live with any connect timeouts WinSock has by default (which is normally 1 - 2 minutes).

In other words, you are probably using the worst possible setup for your "HTTPClient" component. The property set for "TimeoutsOfAPI.ConnectTimeout" has absolutely no effect, which means that a call to Connect will always take as long as WinSock thinks it should (up to 2 minutes), but your "Timeouts.AfterConnecting" property could trigger after 32 seconds (your value currently set for "Timeouts.AfterConnecting"), closing the connection even if it was opened later.

"Timeout" properties are usually used in combination with non-blocking Sockets, because with Blocking sockets you can't "inject" a Disconnect if a Connect call is in progress, which means that your current "Timeouts" setup on the Blocking "HTTPClient" component will never do you any good, but the "Timeouts" property could result in a Disconnect message to be placed in the message queue just before the last Connect call returns, which would in your case most likely result in a Reconnect Timer component to be started immediately before a disconnect event would get triggered because of a short "Timeout" property value for "AfterConnecting".

If you want to continue using blocking WinSock, you would need to set the "Timeout.AfterConnecting" value to 0 (default) on the Blocking "HTTPClient" component, because it will only cause problems when used in combination with blocking sockets and all "ReconnectOn" properties set to TRUE.

But I really see no reason for you to use Blocking sockets in single-threaded mode for one connection when your other connection is non-blocking and MultiThreaded. If you are having problems with slow and unstable connections, you should either set "Blocking" to FALSE on both of your connections, in which case you can use "Timeouts.AfterConnecting" to have the connection closed if it takes too long to connect, or ... you can set "useProxy" and/or "useWinHTTP" properties to TRUE to use the WinInet or WinHTTP API which have the option to explicitly set the "TimeoutsOfAPI.ConnectTimeout".

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Did anything I wrote help you solve the problem or do you have more information about the issue now?

Best Regards,
Danijel Tkalcec

[Director]

Did anything I wrote help you solve the problem or do you have more information about the issue now?

Thanks for your comments, they was helpful, because they make us think in right way. Bug is still here, but it seems that we have localize it. Now both clients are switched to non-blocking mode, so app not freezes at all (but have other errors, internal). It seems that error was in app's architecture.

We will write to you if this problem will show unknown behaviour in RTC components.