Access Violation in CompareText()

[joepasquariello]

Hello,

I'm building an HTTP server with RTC. Each client (browser) requests a simple web page (HTML and javascript), and then afterward only makes requests for JSON data. I'm using SQLite and FireDAC database components. RAD Studio XE5 on Windows 7 Pro 64-bit. I had some problems at first with multiple clients, but I thought I had solved those by making sure that my RTC data provider handlers (C++) look as shown below. I've read all of the FireDAC and RTC articles on multi-threading, and I think I'm doing things correctly now. My RtcHttpServer has MultiThreaded = true.

My problem is that if I open a relatively small number of clients (10), and increase their polling rate to about 10 times/sec, I pretty quickly get an access violation that is reported to be in the VCL CompareText() function. I checked the source for rtcSrvModule.pas, and it does make one call to CompareText() in TRtcBaseServerModule.DoExecute(). I did some googling, and I found an article on EDN about an issue regarding this function and non-thread-safe VCL (link below). The problem they discuss seems to be related to creating/destroying TDataModule. I am not doing that. I create/destroy FireDAC and RTC objects, as shown in my handler code. My server is built as described in the tutorials, with a TRtcServerDataLink and TRtcDataProvider components in a TDataModule, and a TRtcHttpServer on a separate TForm. Am I doing something wrong in my design or code that causes this problem? Is the problem really occurring in CompareText(), and is the issue described on EDN a problem for RTC?

Thanks for any advice you can provide.

Joe
 
https://forums.codegear.com/message.jspa?messageID=595331

void __fastcall THttpDataModule::RTEVENTDataProviderDataReceived(TRtcConnection *Sender)
{
  TRtcDataServer *DS = (TRtcDataServer*)Sender;
  if (DS->Request->Complete) {
    // create a new TFDConnection with settings identical to Connection
    TFDConnection *TempConnection = new TFDConnection( this );
    TempConnection->ConnectionString = Connection->ConnectionString;
    TempConnection->UpdateOptions->LockWait = true;
    // create a new TFDQuery, apply settings, assign SQL, execute, FetchAll()
    TFDQuery *FDQuery = new TFDQuery( this );
    FDQuery->Connection = TempConnection;
    FDQuery->ResourceOptions->CmdExecMode = amNonBlocking;
    FDQuery->SQL->Assign( RTEventQuerySQL );
    FDQuery->Active = true;
    FDQuery->FetchAll();
    // use TRtcArray to get "flat" JSON compatible with D3
    TRtcArray *RtcArray = new TRtcArray;
    DelphiDataSetToRtcArray( FDQuery, RtcArray );
    DS->Write( RtcArray->toJSON() );
    delete( RtcArray );
    // destroy temporary TFDQuery and TFDConnection
    FDQuery->Active = false;
    FDQuery->SQL->Clear();
    delete( FDQuery );
    TempConnection->Connected = false;
    delete( TempConnection );
  }
}

[joepasquariello]

More information:

I have opened 12 browsers on the same PC that is running the server, with each browser polling the server 50-100 times/sec, and this seems to run forever with no problems. If I then open one browser on another PC, and make a connection to the server, I pretty quickly get an access violation. So far, the violation is always in function TRtcBaseServerModule.DoExecute(), though not always in CompareText(). The debugger shows the most recent error occurring on line 1035 of rtcSrvModule.pas.

Is there something different about local and remote connectioins? Is this a clue to something I am doing wrong in the server? I'm doing all of my testing so far with the server running from the IDE.

Thanks,

Joe

[D.Tkalcec (RTC)]

Im not familiar with the datavase components you are using, but with the BDE, you need to create and manually assign a separate Database Session component (TSession) for each database connection (TDatabase) running in a separate thread. If you dont, a single global database session component will be silently created and assigned to all database connections without a database session component assigned, so the code would work fine in a single threaded app, but cause Access Violations in multithreaded app.

In general, you need to make sure to nor use any global objects from multiple threads, if they were not especially designed for such use.


Best Regards,
Danijel Tkalcec

[joepasquariello]

Thanks, Danijel. I am creating a separate TFDConnection (like TDatabase) and TFDQuery for each request, so I think I am meeting that requirement. Can I ask a few follow-up questions?

1) In my reply to original post, I explain there is no problem if the clients are on the same PC as the server, but if the clients are on a different PC, communicating over the wireless the network, the problem occurs. Do you have any thought on why this would occur? Is it a clue to what is wrong?

2) Is there any way to associate objects (e.g. TConnection) with a thread within RTC, so I create the minimum necessary number of TConnection objects?

3) Is there a way to limit the number of threads used by RTC? Can I limit to 1 thread for testing?

Joe

[D.Tkalcec (RTC)]

1) This is just a conicidence. When your App is multithreaded, it's all about timing.

2) No, you can not associate a TConnection component with a specific thread.

3) Yes. Check the rtcThrPool unit and you will see the global variables which you can set to define how the RTC thread pool works.

Best Regards,
Danijel Tkalcec

[joepasquariello]

Thank you, Danijel.

I have had a discussion with FireDAC creator Dmitry Arefiev, and he has assured me the way I create/delete TFDConnection and TFDQuery within the event handler is thread-safe. Assuming that is correct, is the RTC-related portion of the OnDataReceived handler shown below okay? Maybe there is a problem with using "RtcArray->toJSON()" as the argument to DS->Write, or I don't understand a limit to the size of JSON string that is okay to send this way.

BTW, there is a connection pool capability in FireDAC, so it's possible to avoid the creation/destruction of TFDConnection on each request. The default number of connections in the pool is 50, and can be set to match the number of threads in the RTC server. When I get this working well, I would be happy to make an example of using SQLite/FireDAC with RTC.

Joe

void __fastcall THttpDataModule::DEVICEDataProviderDataReceived(TRtcConnection *Sender)
{
  TRtcDataServer *DS = (TRtcDataServer*)Sender;
  if (DS->Request->Complete) {
    ...
    create/populate FDQuery (thread-safe)
    ...
    TRtcArray *RtcArray = new TRtcArray;
    DelphiDataSetToRtcArray( FDQuery, RtcArray );
    DS->Write( RtcArray->toJSON() );
    delete( RtcArray );
  }
}

[D.Tkalcec (RTC)]

The RTC portion of the code you have posted is thread-safe.

Is there anything else you are doing in the Server?

Best Regards,
Danijel Tkalcec

[joepasquariello]

There are 3 DataReceived handlers, all practically identical except for the associated URL and query text, and a file data provider from the QuickStart example. During the test, all of the requests are to the same URL, so it's really only one handler that is being executed.

I need to try again, but yesterday it seemed like the test was okay (no errors) with all clients on local PC, and was also okay (no errors), with all clients on remote PC. The problem occurs when clients are on the local PC AND a remote PC. I haven't tried more than one remote PC.

Joe

[D.Tkalcec (RTC)]

The problem with Access Violations is that very often they are raised at locations which are completely unrelated to the actual location that caused the problem. This is why I can't really tell you where to look. Anything that uses memory could be causing an Access Violation, if used in a wrong way.

For example, copying memory from one location to another without allocating enough memory at the destination would cause memory overwrites, which will cause an Access Violation when you try to access an object which was previously stored at the location that got overwirtten. And because memory allocation and decallocation is dynamic, it is possible that a memory location holding an Object, an Array, a String or anything else could get overwritten by completely unrelated code accessing memory directly.

So, you need to check all the code being executed on the Server while your test is running. Looking at the code arround the location where the Access Violation is happening usually isn't enough.

Best Regards,
Danijel Tkalcec

[joepasquariello]

Thanks, Danijel. I just did a test that seems to point back to the database access not being threadsafe.

Normally, the client sends requests for a URL that results in executing one of the database data providers. I modified the client to send requests for a file containing JSON data, so all of the requests for data are handled by the file data provider from the RTC example.

I started with 3 clients on the same PC as the server, then added 9 clients on a remote PC. With all 12 clients reading the file 50-100 times/sec, there were no errors.

I then changed the HTML page to send requests that cause database access. First I restarted the clients on the same PC as the server. With those 3 clients reading from the database, and the 9 remote clients reading files, all was okay. As soon as I restarted a remote client to read from the database, I got an AV on the server.

These are not large queries, so I don't think it's the amount of data, but rather something in the Connection/Query that is not thread-safe. Would you agree with this conclusion?

Joe

[D.Tkalcec (RTC)]

The biggest problem is that the place where the Access Violation happens doesn't usually have much to do with the location that caused the problem. But if you only get AVs when a specific piece of code is being executed, then my conclusion would also be that there is something wrong with that piece of code.

Best Regards,
Danijel Tkalcec

[joepasquariello]

Okay, I will refine the test to try to isolate the code that causes the problem.

Joe

[joepasquariello]

Danijel, after many tests, and always getting AV with database access, I found the solution below. I can only guess that this avoids add/remove from a list that is not thread-safe?

    // set Owner=NULL for TFDConnection and TFDQuery. Owner=this causes AV.
    TFDConnection *TempConnection = new TFDConnection( NULL );
    TFDQuery *FDQuery = new TFDQuery( NULL );

The server is running now fine with clients on multiple PCs, with no AV, and the change above is the ONLY change.

Thank you for your help. I know I keep saying this, but RTC is a great product.

Joe

[D.Tkalcec (RTC)]

Thank you for sharing your findings.

Best Regards,
Danijel Tkalcec

[joepasquariello]

You're welcome. Now that FireDAC is the primary database tool in RAD Studio, I would expect more people to be using it with RTC. Perhaps I should update the database example? I'll let you know when I have converted my current project to use FireDAC's connection pool. This would be the best fit with RTC's multi-threaded design.

Joe