Problems freeing components in router

[Dany]

Hello!

I have a long standing problem. Sometimes when i terminate my application i get the following AV (here's the call stack):

Code:

:7740c42d KERNELBASE.RaiseException + 0x58
System.SyncObjs.TEvent.SetEvent
rtcThrPool.TRtcWorkerThread.PostWork($647D5A0)
rtcThrPool.TRtcThread.PostJob($647D5A0,$3261350,True)
rtcSocketCliProv.TRtcSocketClientProvider.Release
rtcConn.TRtcConnection.ReleaseProvider
rtcConn.TRtcClient.Destroy
rtcDataCli.TRtcDataClient.Destroy
rtcHttpCli.TRtcHttpClient.Destroy
System.Classes.TComponent.DestroyComponents
System.Classes.TComponent.Destroy
System.Classes.TDataModule.Destroy
System.TObject.Free
rtcRoutingProvider.Finalization
System.FinalizeUnits
System._Halt0
FLIIRTC_SecurityServer.FLIIRTC_SecurityServer
:00406f34 NotifyNonDelphiException + $1C
:0042ae28 Exception.GetBaseException

The finalization of my unit rtcRoutingProvider is taken from the webserver demo:

Code:

finalization

if assigned(Routing_Provider) then
begin
  Routing_Provider.Free;
  Routing_Provider := nil;
end;

The code stopping this provider looks like this and is inspired by the router demos:

Code:

procedure TRouting_Provider.CloseClients;
begin
  FClosing := true;

  try
    XLog('Disconnecting clients...');
    SearchHttpClient.AutoConnect := false;
    SearchHttpClient.Disconnect;
    FileHttpClient.AutoConnect := false;
    FileHttpClient.Disconnect;
    ApplicationHttpClient.AutoConnect := false;
    ApplicationHttpClient.Disconnect;
    PageHttpClient.AutoConnect := false;
    PageHttpClient.Disconnect;

    XLog(Format('Waiting for %d client connections to close...', [rtcClientConnectionCount]));
    while (rtcClientConnectionCount > 0) do
      Sleep(10);

    Sleep(500);

      //  Strange because these are published properties, should be
      //  able to set at design time and keep them through start-stop
      //  cycles?
    XLog('Uncoupling datarequest components...');
    SearchDataRequest.Client := nil;
    FileDataRequest.Client := nil;
    ApplicationDataRequest.Client := nil;
    PageDataRequest.Client := nil;

    XLog('Cleaning router queues...');
    SearchDataRouter.CleanUpQueues;
    FileDataRouter.CleanUpQueues;
    ApplicationDataRouter.CleanUpQueues;
    PageDataRouter.CleanUpQueues;
  finally
    FClosing := false;
  end;
end;

The linked server is stopped like so

Code:

    ServerHTTP.StopListenNow;
    while (ServerHTTP.isListening) or (ServerHTTP.TotalServerConnectionCount > 0) do
      Sleep(250);

before CloseClients is called. This is done similarly to the webserver demo.

Just for completion i'm including the code starting the provider (after the server has started):

Code:

procedure TRouting_Provider.ConnectClients;
begin
  SearchDataRequest.Client := SearchHttpClient;
  FileDataRequest.Client := FileHttpClient;
  ApplicationDataRequest.Client := ApplicationHttpClient;
  PageDataRequest.Client := PageHttpClient;

  SearchHttpClient.AutoConnect := true;
  FileHttpClient.AutoConnect := true;
  ApplicationHttpClient.AutoConnect := true;
  PageHttpClient.AutoConnect := true;
end;

So these four parallel routers are each three components on the datamodule. The TRtcDataRouter's are connected to a common TRtcDataServerLink at design time. They have the OnCheckRequestI, OnPostNewRequestI and OnRequestReceivedI events connected to separate handlers at design time. The TRtcDataRequest components have "HyperThreading" checked. The TRtcHttpClient components have "MultiThreaded" checked.

OnCheckRequestI's calls Sender.Accept the OnPostNewRequestI's assignes the DataRequest parameter to the correct TRtcDataRequest component and the OnRequestReceivedI's only adds some info to Sender.Request using the Values property.

All event handlers except for OnCheckRequestI will exit first thing if FClosing is true.

I can reproduce the problem when testing and thus i can know that this provider does not get any new requests during server stop and application termination.

Any idea about what i'm doing wrong is *much* appreciated.

TIA,

/Dany

[D.Tkalcec (RTC)]

Judging by the Call Stack, my guess would be that some Client connections were still active (they had an active thread) when the Application was terminating. Instead of setting AutoConnect to FALSE and calling Disconnect, try using the new DisconnectNow method on all TRtcHttpClient components and see if it makes any difference. I'm not sure if this is already the case, but also make sure to stop all Server listeners before you start disconnecting any Clients (using StopListenNow), so there won't be any new incoming requests which might try accessing the Clients afterwards.

Best Regards,
Danijel Tkalcec

[Dany]

Changed to

Code:

    XLog('Disconnecting clients...');
    //SearchHttpClient.AutoConnect := false;
    SearchHttpClient.DisconnectNow;
    //FileHttpClient.AutoConnect := false;
    FileHttpClient.DisconnectNow;
    //ApplicationHttpClient.AutoConnect := false;
    ApplicationHttpClient.DisconnectNow;
    //PageHttpClient.AutoConnect := false;
    PageHttpClient.DisconnectNow;

and removed "AutoConnect := true". No go

The CloseClient procedure is called after the server is stopped. The code for that is in my OP.

I can add that there are other providers to this server and as long as the routing provider has not been used (only requests that the other providers respond to was made) this does not happen.

Need to skip and hop now so i'll do some digging tomorrow and get back.

Thanks,

/Dany

[D.Tkalcec (RTC)]

1. Does the Routing Provider create any other TRtcHttpClient components on-demand?

2. Is the Routing Provider accessing anything that it hasn't created (not the owner)? Maybe even something created outside of the unit?

3. The fact that the Routing Provider is being destroyed from the unit Finalization section could mean that some things required by the Routing Provider have already been destroyed before reaching the finalization section of that unit. If the Routing Provider is used in a Windows Service, instead of destroying it from the finalization section of the unit, you could try closing the connections and destroy the Routing Provider when the Service receives the Shut Down notification, to make sure no objects used by the Routing Provider are destroyed before its connections are closed and the DataModule destroyed.

Best Regards,
Danijel Tkalcec

[Dany]

1. Does the Routing Provider create any other TRtcHttpClient components on-demand?

No, i'm absolutely sure.

2. Is the Routing Provider accessing anything that it hasn't created (not the owner)? Maybe even something created outside of the unit?

No, it does not even have a DataModuleCreate handler. It does use HaveSession, FindSession and UnlockSession though.

3. The fact that the Routing Provider is being destroyed from the unit Finalization section could mean that some things required by the Routing Provider have already been destroyed before reaching the finalization section of that unit. If the Routing Provider is used in a Windows Service **CUT**

It is used in a setup very similar to the webserver demo. It is extremely convenient to start the exe as an application when setting parameters (database, ports et.al.) and test and then terminate the application and start the service. When running as a service sometimes i have to kill the process. Running as an application the problem is more reproduceable (and debuggable) of course.

So, i create a new global procedure in parallel with "GetRoutingProvider" called "FreeRoutingProvider" and call that in the "Stop" method that gets called from the service on shutdown and from the application btnStopListening. This seems to make things better. It is, however awkward to "break" the general layout of it all.

But, now that i put some more XLog's into it all i can see the following, and this might be my problem. This code:

Code:

  if lSettings.Port <> 0 then
  begin
    XLog('Stopping HTTP Server...');
    ServerHTTP.StopListenNow;
    while (ServerHTTP.isListening) or (ServerHTTP.TotalServerConnectionCount > 0) do
      Sleep(250);
    XLog('Finished HTTP Server...');
  end;

  if lSettings.SSLPort <> 0 then
  begin
    XLog('Stopping HTTPS Server...');
    ServerHTTPS.StopListenNow;
    while (ServerHTTPS.isListening) or (ServerHTTPS.TotalServerConnectionCount > 0) do
      Sleep(250);
    XLog('Finished HTTPS Server...');
  end;

that gets executed before cleaining and stopping. Produces the following log result:

Code:

2014-10-26 12:05:25.178; SERVER ServerHTTP STARTED on Port 80...
2014-10-26 12:05:25.459; SERVER ServerHTTPS STARTED on Port 443...
2014-10-26 12:06:11.446; Stopping HTTP Server...
2014-10-26 12:06:11.478; SERVER ServerHTTP STOPPED.
2014-10-26 12:06:11.531; Finished stopping HTTP Server...
2014-10-26 12:06:11.539; Stopping HTTPS Server...
2014-10-26 12:06:11.594; Finished stopping HTTPS Server...
2014-10-26 12:06:11.644; Disconnecting clients...
2014-10-26 12:06:11.646; Waiting for 0 client connections to close...
2014-10-26 12:06:12.148; Cleaning router queues...
2014-10-26 12:06:12.207; SERVER ServerHTTPS STOPPED.

the "SERVER ServerHTTPS STOPPED" line should be printed before the "Finished stopping HTTPS Server..."! The lines "SERVER %s STOPPED." are printed in the ServerHTTPListenStop event connected to both servers in the server module. The three lines before the last bold line is from the routing providers CloseClients.

I really must apologize for not seeing this before now, my bad definitely  .

Perhaps i need some other way of waiting for the HTTPS server to stop?

[D.Tkalcec (RTC)]

I'm thinking ...

Server connections are all running in background threads. When you stop a Server Listener, no new Client connections can come in, but active events won't imediately be terminated and it could take some time for those events to complete. As long as you do not have direct control over currently running events on individual connections (which you probably don't), you will have to give those events enough time to complete execution before you move on.

Looking at the log files, one second should be enough, so adding a Sleep(1000) after both Server Listeners were stopped and all connections are closed might fix the problem of the "Stop Listen" event triggering after the Data Module has already been destroyed. But, you should change your code to have a single loop for both Server components, because the "TotalServerConnectionCount" method is directly mapped to the global "rtcServerConnectionCount" function in the rtcConn unit, which returns the total number of all connections to all Server components.

Code:

  if lSettings.Port <> 0 then
  begin
    XLog('Stopping HTTP Server Listener ...');
    ServerHTTP.StopListenNow;
    XLog('Finished HTTP Server.');
  end;

  if lSettings.SSLPort <> 0 then
  begin
    XLog('Stopping HTTPS Server Listener...');
    ServerHTTPS.StopListenNow;
    XLog('Finished HTTPS Server...');
  end;

XLog('Stopping Server connections ...');
while rtcServerConnectionCount > 0 do
      Sleep(250); // Wait for Server connections to close
Sleep(1000); // Wait for active Server events to complete
XLog('Finished Server connections.');

The same applies to closing Client-side connections which are running in Multi-Threaded mode, so adding another Sleep(1000) after all the Client connections have been closed might fix the problem there, too.

Code:

    XLog('Disconnecting clients...');
    SearchHttpClient.DisconnectNow;
    FileHttpClient.DisconnectNow;
    ApplicationHttpClient.DisconnectNow;
    PageHttpClient.DisconnectNow;

    XLog('Waiting for Client connections to close ...');
    while rtcClientConnectionCount > 0 do
      Sleep(250); // Wait for open Client connections to close
    Sleep(1000); // Wait for active Client events to complete
    XLog('Client connections closed.');

Once all Client and Server connecions have been closed and all the related events have completed execution, it should be safe to destroy the components and all related objects.

Best Regards,
Danijel Tkalcec

[Dany]

Yesterday i tried this latest suggestion, but unfortunately it does not change much. Other things are demanding my attention and i think that the only sound way of getting anywhere with this is for me to create a project to reproduce it. During such an effort i might find what i'm doing wrong or - not that common i'll admit - if there's something in the SDK.

Destroying the provider at server shutdown does improve things enough for now.

Thanks for your attention and effort,

/Dany