Bug in RtcClientModule.Execute if AllowMessageProcessing=false

[Max Terentiev]

Hi, Danijel,

Just found another bug in RtcClientModule:

If we call RtcClientModule.Execute(true,5,false); // AllowMessageProcessing=false, timeout 5 seconds
Request is not sent and timeout exception raised after 5 seconds.

If we call RtcClientModule.Execute(true,0,false); // AllowMessageProcessing=false, no timeout
Request not sent and Execute call goes to never ending infinity wait.

This happens with RtcMessageClient connected to local RtcMessageServer (RtcMessageClient.Server:=RtcMessageServer).

You can easy reproduce it with my previous example project (about Execute error events) if you change Execute call from Execute(true,1,true) to Execute(true,0,false);

p.s. Additional note: if we call RtcClientModule.Execute with AllowMessageProcessing=false - OnBeginRequest event not fired !

[Max Terentiev]

Any response about this bug  ? Is it's confirmed ?

[D.Tkalcec (RTC)]

I will get back to you, as soon as I have news worth sharing.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Your example with the "Execute" method using "AllowMessageProcessing=False" does NOT work, because you are using RTC components in MultiThreaded mode, while trying to access a TMemo (GUI element) directly from RTC events. Since GUI elements may ONLY be accessed from the Main Thread, but events triggered on RTC components running in MultiThreaded mode are executed from inside background threads, your attempt to write to the TMemo component directly from the "OnBeginRequest" event results in a "deadlock".

In short, you should NOT use the "Execute" or "WaitForCompletion" methods with "AllowMessageProcessing=False" if your events have to access the GUI (Forms or any Visual Components or Controls), because synchronizing with the GUI and some methods on GUI components require message processing.

Also, for your code work correctly, you should NEVER access GUI elements directly from background threads. If most of your events implemented on the TRtcClientModule component need access to the GUI, you can set the "AutoSyncEvents" property on the TRtcClientModule component to TRUE, which tells the component to synchronize all events triggered by that component with the GUI. This includes the OnResult event on the TRtcResult component.

Or ... if you ONLY need to access the GUI from some of your events, but want the rest of your code to run in the background, you can use the Sync() method (available on the Sender:TRtcConnection component) from any event triggered by a RTC connection component to synchronize that event with the GUI.

Here is an example:

procedure MyForm.MyEvent(Sender:TRtcConnection);
  begin
  // call the event synchronized if we are in a background RTC thread ...
  if Sender.Sync(MyEvent) then // returns TRUE if the event was called (executed in the Main Thread)
    Exit; // exit here if "Sync" returned TRUE
  // From here on, were are in the Main Thread, so access to the GUI is allowed ...
  end;

Best Regards,
Danijel Tkalcec

[Max Terentiev]

Hi Danijel,

I test it again and finally found bug(s) and steps to reproduce:

At this time my app have Client and Server in one .exe to make it easy for develop and debug. I use RtcMessageClient+RtcMessageServer for local transport.

Here is my components setup:

ClientSide:
--------------
(TDataModule) ClientDM + RtcMessageClient + RtcClientModule
RtcClientModule.Client:=RtcMessageClient;
--------------

ServerSide:
--------------
(TDataModule) ServerDM + RtcMessageServer + RtcServerModule + RtcFunctionGroup + Many RtcFunctions
RtcMessageServer.MultiThreaded:=true;
--------------

ClientDM.RtcMessageClient.Server:=ServerDM.RtcMessageServer; // For local transport

RtcMessageClient and RtcClientModule does NOT have any assigned events like OnBeginRequest, OnResultError, etc. ALL their events are NOT assigned.

With different combinations of AutoSyncEvents, MultiThreaded, AllowMessageProcessing and Timeout - RtcClientModule.Execute method works very different !!! Execute calls bellow is calling long running remote function wich works much longer than Execute timeout !

Case 1: RtcMessageClient.MultiThreaded:=false, RtcClientModule.AutoSyncEvent:=false:

RtcClientModule.Execute(true,5,true);     // Timeout just not works ! Execute waits for result until it arrived in 15 seconds
RtcClientModule.Execute(true,5,false);    // Timeout just not works ! Execute waits for result until it arrived in 15 seconds

Case 2: RtcMessageClient.MultiThreaded:=false, RtcClientModule.AutoSyncEvent:=true:

RtcClientModule.Execute(true,5,true);     // Again, Timeout just not works ! Execute waits for result until it arrived in 15 seconds
RtcClientModule.Execute(true,5,false);    // Again, Timeout just not works ! Execute waits for result until it arrived in 15 seconds

Case 3: RtcMessageClient.MultiThreaded:=true, RtcClientModule.AutoSyncEvent:=false:

RtcClientModule.Execute(true,5,true);     // Timeout works as expected !
RtcClientModule.Execute(true,5,false);    // Timeout works as expected !

Case 4: RtcMessageClient.MultiThreaded:=true, RtcClientModule.AutoSyncEvent:=true:

RtcClientModule.Execute(true,5,true);     // Timeout works as expected !
RtcClientModule.Execute(true,5,false);    // Timeout fired always, no matter received reply or not !!! If callled fast/instant remote function - timeout exception raised anyway !
RtcClientModule.Execute(true,0,false); //!!!! Execute call NEVER returns because of infinite loop inside RtcThrPool.RtcWaitFor (lines 487-498) !!!!!! No matter instant or slow remote function called - remote function executed and send reply back but RtcClientModule just not see it

And I check many times: NO events assined for RtcMessageClient and RtcClientModule ! No GUI access ! All Execute calls is made inside main thread.

As you see Execute timeout works very strange depends on MultiThreaded, AutoSyncEvent and AllowMessageProcessing combinations ! From  "Not works" to "works as exepected" and "very buggy" ! I sure it's a bug ! Please fix it if possible !

Thanks !

[Max Terentiev]

One addition:

I just try to set RtcClientModule.AutoSyncEvents:=false, assign my events back to RtcClientModule and (in each event) write manual synchronization code:

if not Sender.InMainThread then
    Sender.Sync(EventName);
else
     DoGuiWork;

And I got same problems like with AutoSyncEvents:=true and unassigned events (case 4 in previous post) ! Problem happens after entering to Sender.Sync(EventName). Looks like bug somewhere in synchronization routines. Hope it's help to fix it !

[D.Tkalcec (RTC)]

Your cases 1 and 2 are working as expected. When RtcMessageClient is running in single-threaded mode, remote functions will be executed in blocking mode and WaitForCompletion will be called afterwards, which is why the timeout parameter won't have any effect. This is by design and is simply how single-threaded blocking communication works.

Your case 3 is also working as expected. When RtcMessageClient is running in multi-threaded mode, all the remote function code will be executed from the background, while the WaitForCompletion method (called by Execute) waits in the Main Thread, up to the specified timeout. This is also working as designed.

As for your case 4 ...

Because the default implementation of the "Sync" method from RTC is to call the "TThread.Synchronize" method from the Delphi RTL, which (on Windows) sends a Windows Message from the Background Thread to the Main Thread and waits in the calling/background Thread until the Main Thread processes that Message, if you are using the Sync method in your code running from the background thread or if you set the AutoSyncEvents:=TRUE, the Execute and WaitForCompletion methods should be called with "AllowMessageProcessing:=TRUE", which is also the default value for this parameter.

If you call the Execute or WaitForCompletion method with AllowMessageProcessing:=FALSE (on Windows), after using the Sync method from a background thread or using AutoSyncEvents:=TRUE, your background thread will be waiting for your Main Thread to process the message sent, but your Main Thread will NOT be Processing Messages, so ... your background thread will be waiting forever for the Main Thread and your Main Thread will be waiting forever for your background thread.

Please note that this behavior is specific to Windows. On all the other platforms, where the standard TThread.Synchronize method from the Delphi RTL doesn't work reliably, RTC will be using a custom implementation for the "Sync" method which does NOT use Messages, but instead uses a TTimer class from VCL or FMX.

If you absolutely HAVE TO use the Execute or WaitForCompletion methods with AllowProcessMessages:=FALSE (all Windows Message processing disabled while waiting) and you want to use communication components in Multi-Threaded mode and also have the option to use the Sync method from background threads or set AutoSyncEvents:=TRUE, you will have to use one of the following two units in your Project, depending on the framework it uses:

A) for FMX projects use the "rtcFMX.GUI" unit, which uses the TTimer class from FireMonkey to implement synchronization with the Main Thread outside of "Execute" and "WaitForCompletion" loops, while directly calling the "rtcSyncProc" procedure from inside "Execute" and "WaitForCompletion" loops.

B) for VCL projects use the "rtcVCL_GUI" unit, which uses the TTimer class from the Delphi VCL to implement synchronization with the Main Thread outside of "Execute" and "WaitForCompletion" loops, wile directly calling the "rtcSyncProc" procedure from inside "Execute" and "WaitForCompletion" loops.

Alternatively, if your projects are NOT using FMX nor VCL and you do NOT want any dependencies on VCL or FMX in your Project, you can copy all the code from the "rtcVCL_GUI.pas" unit into your own unit, then copy code from the "rtcFMX_GUI.inc" file (where {$include rtcFMX_GUI.inc} is) and modify your unit to do the same without relying on the TTimer class from FMX or VCL for synchronization with the Main Thread, if you want your Sync method calls to work (synchronize your code with the Main Thread) even when you are NOT currently calling the Execute or WaitForCompletion methods in the Main Thread.

Best Regards,
Danijel Tkalcec

[Max Terentiev]

If you absolutely HAVE TO use the Execute or WaitForCompletion methods with AllowProcessMessages:=FALSE (all Windows Message processing disabled while waiting) and you want to use communication components in Multi-Threaded mode and also have the option to use the Sync method from background threads or set AutoSyncEvents:=TRUE

Yes, I need exactly this settings because one of my task is fetching dataset on demand then DBGrid scrolls. If AllowMessageProcessing=true - I got series of fetch calls because scroll events is accumulated somewhere in VCL while Execute works and then it's done they call my fetch procedure again and again... And also I need to call remote functions at background from some of my threads using StartCalls/post - because of it I need multithreading. And I need a timeout because it's handy )

I added rtcVCL_GUI unit and it's eleminate problem with infinity loop and timeout !!!

But it's adds little new issue: if Execute call aborted by timout and I immediately call Execute again (for another remote function) - I have hang for about 1-2 seconds. Looks like RTC waits for release something for this time and after it allow next Execute call to run (after pause ends I see lots of "module unload" for ADO dlls in delphi's events window). I not have this delay if MultiThreaded=false and without rtcVCL_GUI unit. This is not big problem (because timiout will be rare) but it's will be nice if you can suggest how to avoid or decrease this delay.

And very last question: does rtcVCL_GUI affect behaviour of non-RTC threads and Synchronize ?

Thank you very very much !

[D.Tkalcec (RTC)]

The "rtcVCL_GUI" unit affects your Application by creating a TTimer object which periodically calls the rtcSyncCheck function (from the Main Thread) to check if anything is waiting in the RTC "Sync" queue, and executes it. Depending on the frequency of this timer and the code you call using the "Sync" method from RTC events, this can (obviously) have an affect on the rest of your Application.

Disabling message processing while waiting in the Main Thread for something to be executed in a background thread, which in turn waits for something else to be executed in the Main Thread will also have an effect on your Application.

There is always a cost associated with code synchronization between background threads and the Main Thread, so try to keep the number of "Sync" calls from background threads at the absolute minimum.

Check "rtcVCL_GUI" unit implementation if you need more details about how this works and make a copy of that unit (as explained above) if you need to make adjustments for your Application.

Best Regards,
Danijel Tkalcec

[Max Terentiev]

Clear, thank you !

[D.Tkalcec (RTC)]

Just for the info ...

Starting with RTC SDK v8.00 (2017.Q2), to avoid problems with thread synchronization when using blocking methods like Execute, WaitForCompletion, DisconnectNow or StopListenNow (see problem case 4 above), the default "Sync" implementation in the RTC SDK now uses a dedicated (non-user) background thread to call the "TThread.Synchronize" method for all RTC Threads which need to execute code in the Main Thread, eliminating the problem of code running in RTC Threads being blocked infinitely by a "Sync" call while waiting for one of the blocking calls to return when running with message processing disabled.

The main difference between the new  "Sync" method implementation - starting with RTC SDK v8.00 (2017.Q2) - and the one previously provided when using the "rtcFMX.GUI" (for FireMonkey Projects) or "rtcVCL_GUI" (for VCL Projects), is that "rtcVCL_GUI" and "rtcFMX.GUI" units were using a TTimer instance, which is platform-specific and was triggered periodically (by default, every 50 milliseconds) to check if any events are waiting for execution in the Main Thread.

The reason for using a "TTimer" instead of calling "TThread.Synchronize" was poor performance of the "TThread.Synchronize" implementation in older Delphi versions on some platforms, resulting in very poor performance of mobile RTC Client Applications running in single-threaded mode or setting the AutoSync property to TRUE, where every user event has to be synchronized with the Main Thread, while all the other code is running in background threads. This performance difference between using a "TTimer" with a high frequency and using "TThread.Synchronize" directly from RTC Threads is the ONLY reason why "rtcFMX.GUI" and "rtcFMX_GUI" units are still available in the "Lib" folder.

Best Regards,
Danijel Tkalcec