Server eating up all cpu

[Cesar]

Hello,

I've been having a strange bug in which my server app starts to eat up the 100% of my cpu

It's a basic rtchttpServer with the following config

Code:

object Server: TRtcHttpServer
    MultiThreaded = True
    Timeout.AfterConnecting = 60
    Timeout.AfterConnect = 60
    Timeout.AfterDataReceived = 60
    Timeout.AfterDataLost = 60
    Timeout.AfterDataSend = 60
    Timeout.AfterDataOut = 60
    Timeout.AfterDataIn = 60
    Timeout.AfterDataSent = 60
    ServerPort = '8082'
    OnException = ServerException
    OnListenLost = ServerListenLost
    OnListenError = ServerListenError
    TimeoutsOfAPI.ResolveTimeout = 60
    TimeoutsOfAPI.ConnectTimeout = 60
    TimeoutsOfAPI.SendTimeout = 60
    TimeoutsOfAPI.ReceiveTimeout = 60
    TimeoutsOfAPI.ResponseTimeout = 60
    Left = 16
    Top = 256
  end

all of my functions onExecute event point to my procedure petitionExecute which looks like this:

Code:

procedure TForm1.PetitionExecute(Sender: TRtcConnection;
  Param: TRtcFunctionInfo; Result: TRtcValue);
begin
  try
    cache.petition(Param.FunctionName, Param, Result);
  Except
    on E:exception do begin
      smartlog('error en petition execute: '+E.ToString,tllWarning);
    end;
  end;
end;

cache. petition basically just queries the DB and gives the result of the petition

The weird thing is: i'm sure cache.petition never enters in a loop or something that could eat up all the 100% of my cpu i even logged method enter and exit of my cache.petition to ensure it never hangs inside of it.

I enabled the DEBUG mode of RTC to see if it has something to do with it and i get this messages.

in my .exe.ERROR.log i get just this 2 kinds of errors:

2013-06-21 18:33:20.944; TRtcSocketClientThread.Destroy RtcConn.Free Exception! EAccessViolation: Access violation at address 00000001. Write of address 00000001
2013-06-21 18:33:21.022; SRV:TRtcSocketClientThread.RunJob "DoMessage(2)" Exception! EAccessViolation: Access violation at address 00000000. Write of address 00000000

in the .exe.INFO.log i got this

2013-06-21 20:34:35.897; TRtcAbsValue.FromCode(0) Exception! ERtcInfo: Unknown object type specifier at 1: 164612 - AVAL1|CASA|8346";lat:F
2013-06-24 08:40:05.251; TRtcRecord.from_Code(416) Exception! ERtcInfo: Not enough data.
2013-06-24 08:40:05.251; TRtcRecord.from_Code(416) Exception! ERtcInfo: Not enough data.
2013-06-24 08:40:05.251; TRtcRecord.from_Code(416) Exception! ERtcInfo: Not enough data.
2013-06-24 08:40:05.251; TRtcAbsValue.FromCode(416) Exception! ERtcInfo: Not enough data.
2013-06-24 10:11:17.222; TRtcAbsValue.FromCode(10) Exception! ERtcInfo: Unknown object type specifier at 11: form:S
2013-06-24 10:46:46.100; TRtcRecord.from_Code(417) Exception! ERtcInfo: Not enough data.
2013-06-24 10:46:46.100; TRtcRecord.from_Code(417) Exception! ERtcInfo: Not enough data.
2013-06-24 10:46:46.100; TRtcRecord.from_Code(417) Exception! ERtcInfo: Not enough data.
2013-06-24 10:46:46.100; TRtcAbsValue.FromCode(417) Exception! ERtcInfo: Not enough data.
2013-06-24 11:00:37.130; TRtcAbsValue.FromCode(0) Exception! ERtcInfo: Unknown object type specifier at 1: 99.891157;typ:I
2013-06-24 11:53:39.676; TRtcAbsValue.FromCode(0) Exception! ERtcInfo: Unknown object type specifier at 1: 2"P063121758450 - TITUL|LABO|10108";lat:F
2013-06-24 13:45:10.434; TRtcAbsValue.FromCode(0) Exception! ERtcInfo: Unknown object type specifier at 1: :S

i tracked down the one at 11:53:39 and this is the data i  tried to response

47=(fid=50; fha=tm0dq4; id=1036124; form=EXTRAJUDICIAL; fol=P063121758450 - TITUL|LABO|10108; lat=19.024571; lon=-98.247895; typ=0; tex=2 NORTE 2005, PUEBLA CENTRO, PUEBLA, PUEBLA, 72000; nid=-1; nha=);

I know it because in a future attempt it passes. (the above is the dump of Result.asRecord.GetAsText)

so down to the questions:

How do I interpret the log files?
Is there something i'm doing wrong or is it a bug in the component?
is there a way to force the stop of a thread that has going rogue?

[Kevin Powick]

At first glance,  it appears that the object "cache" is not being managed in a way that is thread safe.

What is the "cache" object?  An instantiation of a data module?  If so, is it a data module upon which you've placed database access components?  That would explain the issues.

[Cesar]

Hi Kevin,

the cache object is a class with a set of functions that queries a Database using MyDAC.

this is mostly a "read-only" class in which writes are only done at creation. it has no extra threads besides the ones of the RTC

I've made sure the objects are not shared beetwen request to avoid locks and the only list is needed to be shared its properly isolated in a critical section.

also i've been profiling it with several tools to avoid something is wrong with the cache class.

I tested it with eurekalog to check for any memoryleaks.
I profiled it with aQtime to check for performance and resource leaks and it passed.
I implemmented the SMartInspect Log to ensure all my petitions are behaving like it should a no function get looped.

I am also sure its not a exception raising in any of my code and that the "cpu spike" is not fired by a specific petition or a specific data trying to retrieve. it has happened at different times with different petitions.



One test that i did and might give you guys a hint of something i might be doing wrong.

I tested it with the multithreaded property set to false.

after several minutes running good the application completely froze. the logs showed that the last request i did receive was answered correctly, but since that it stopped receiving new requests.

Another thing is i haven't been able to reproduce it dev environment it only happens in the prod server.

I'm using the v6.14 (2013.Q2) version of the SDK and compiling it in Delphi XE2

[D.Tkalcec (RTC)]

1) My first guess is that the AV you see in the ERROR LOG is a result of the Server being terminated when a connection is still active. I don't think it has anything to do with your actual problem, unless your code is overwriting memory which it does not own.

2) I see that you are catching all exceptions and calling "smartlog", but are NOT re-raising them. This means that your Client will NOT be notified of any Exceptions that might happen inside the remote function, but will instead get a partial Result from the Server with no clue what happened. If you don't handle exceptions gracefully, you should NOT silently "swallow" them. After you write the exception message to your log (if that is what you want to do), you should call "raise", so the exception goes through to RTC components, where it will be sent back to the Client as "rtc_Exception" type.

3) As Kevin pointed out, when you are working with MultiThreaded=TRUE, any code you use inside RTC events has to be 100% thread safe. This means that each thread has to get its own instance of your objects to work with, or your objects have to be capable of executing methods from multiple threads at the same time. I don't know MyDAC, but all the Database Access components I've been working with require a separate set of all database related components per thread (TSession, TDatabase, TQuery, ...). Using a global set of any DB component from multiple threads is a guarantee for a disasaster.

4) When using critical sections, watch out for a "dead-lock" trap, which is what would happen if two thread instances are trying to lock 2 critical sections in a different order (thread 1 locks CsA, thread 2 locks CsB, thread 1 tries to lock CsB and thread 2 tries to lock CsA).

5) INFO exceptions could mean that memory is being overwritten by something, RAM on the Server is corrupt, or there is a missmatch in the encryption (SecureKey).

6) Without seing more of your code, I can only make guesses and assumptions. But first, you need to find out where the exceptions are happening and where the Server freezes. For example, by changing your function code to ...

// uses "Log" function is from the "rtcLog.pas" unit
procedure TForm1.PetitionExecute(Sender: TRtcConnection;
  Param: TRtcFunctionInfo; Result: TRtcValue);
begin
  Log(Param.FunctionName+' enter', 'MAIN');
  try
    cache.petition(Param.FunctionName, Param, Result);
    Log(Param.FunctionName+' exit', 'MAIN');
  Except
    on E:exception do
      begin
      Log(Param.FunctionName+' #ERROR# '+E.ClassName+':'+E.Message,'MAIN');
      smartlog('error en petition execute: '+E.ToString,tllWarning);
      raise; // -> required to forward Exception to the Client!
      end;
  end;
end;

Even if the exceptions happen in different "petitions", this will show you which exceptions happen where, and if one of the "petitions" causes the app to freeze. If you say that the freezing also happens when you are running the Server in single-threaded mode, then this is exactly what you should do. Run the Server with MultiThreaded=FALSE and once it freezes check the logs.

Best Regards,
Danijel Tkalcec

[Cesar]

i did just as you said logging when i enter and leave each of my functions and sending the error to the client

the last thing i logged was leaving the onExecute Event an then i got this from the call stack.

any guess where it might be the problem?

EurekaLog 7.0.5.1  

Exception:
--------------------------------------------------
  2.2 Address: 00870BE9
  2.5 Type   : EFrozenApplication
  2.6 Message: The application seems to be frozen.
  2.7 ID     : 1AFF0000
  2.11 Sent  : 0

User:
-------------
  3.2 Name :
  3.3 Email:

Steps to reproduce:
------------
  8.1 Text:


Call Stack Information:
---------------------------------------------------------------------------------------------------------------------------------------------------------
|Methods |Details|Stack   |Address |Module          |Offset  |Unit                |Class                       |Procedure/Method              |Line     |
---------------------------------------------------------------------------------------------------------------------------------------------------------
|*Exception Thread: ID=5256; Parent=0; Priority=0                                                                                                       |
|Class=; Name=MAIN                                                                                                                                      |
|DeadLock=0; Wait Chain=thread: [ 1488 / 5256 ] is blocked                                                                                              |
|Comment=                                                                                                                                               |
|-------------------------------------------------------------------------------------------------------------------------------------------------------|
|7FFFFFFE|04     |00000000|00870BEE|GestiiServer.exe|00470BEE|rtcInfo             |                            |PosEx                         |8526[16] |
|00000020|04     |0017FBB4|0096DCC4|GestiiServer.exe|0056DCC4|rtcSocketHttpSrvProv|TRtcSocketHttpServerProvider|InternalRead                  |477[5]   |
|00000020|04     |0017FBDC|0096D8A5|GestiiServer.exe|0056D8A5|rtcSocketHttpSrvProv|TRtcSocketHttpServerProvider|TriggerDataReceived           |420[213] |
|00000020|04     |0017FCA0|00968FA6|GestiiServer.exe|00568FA6|rtcSocketSrvProv    |TRtcSocketServerProvider    |wsOnDataReceived              |1217[70] |
|00000020|04     |0017FCD4|009583E0|GestiiServer.exe|005583E0|rtcSockBase         |TRtcSocketBase              |On_DataReceived               |399[7]   |
|00000020|04     |0017FCE4|0095F9D4|GestiiServer.exe|0055F9D4|rtcWinSocket        |TRtcWinSocket               |DoRead                        |1624[8]  |
|00000020|04     |0017FCFC|0095F876|GestiiServer.exe|0055F876|rtcWinSocket        |TRtcWinSocket               |DoXRead                       |1580[1]  |
|00000020|04     |0017FD04|0095FE80|GestiiServer.exe|0055FE80|rtcWinSocket        |TRtcWinSocket               |DoMessage                     |1761[9]  |
|00000020|04     |0017FD28|00958139|GestiiServer.exe|00558139|rtcSockBase         |TRtcSocketBase              |Call_Message                  |278[18]  |
|00000020|04     |0017FD40|0096000B|GestiiServer.exe|0056000B|rtcWinSocket        |TRtcWinSocket               |WMASyncSelect                 |1780[10] |
|00000020|04     |0017FD48|00960385|GestiiServer.exe|00560385|rtcWinSocket        |                            |RtcSocketWindowProc           |1913[45] |
|00000020|04     |0017FDA8|00955EA0|GestiiServer.exe|00555EA0|rtcHWndPool         |                            |RtcMainHWNDWindowProc         |126[6]   |
|00000020|03     |0017FE18|76498989|USER32.dll      |00018989|USER32              |                            |(possible GetWindowLongW+962) |         |
|00000020|03     |0017FE90|76498AB4|USER32.dll      |00018AB4|USER32              |                            |(possible GetWindowLongW+1261)|         |
|00000020|03     |0017FEF4|76498B0B|USER32.dll      |00018B0B|USER32              |                            |DispatchMessageW              |         |
|00000020|03     |0017FF04|0062B9D7|GestiiServer.exe|0022B9D7|Vcl.Forms           |TApplication                |ProcessMessage                |         |
|00000020|03     |0017FF20|0062BA1A|GestiiServer.exe|0022BA1A|Vcl.Forms           |TApplication                |HandleMessage                 |         |
|00000020|03     |0017FF44|0062BD59|GestiiServer.exe|0022BD59|Vcl.Forms           |TApplication                |Run                           |         |
|00000020|04     |0017FF74|0099C257|GestiiServer.exe|0059C257|GestiiServer        |                            |Initialization                |72[6]    |
|00000020|03     |0017FF8C|763FF26F|kernel32.dll    |0008F26F|kernel32            |                            |BaseThreadInitThunk           |         |
---------------------------------------------------------------------------------------------------------------------------------------------------------

Assembler Information:
------------------------------------------------------------------------------------------------
; rtcInfo.PosEx (Line=8522 - Offset=104)
; --------------------------------------
00870BC8  dec     edx
;
; Line=8522 - Offset=105
; ----------------------
00870BC9  jnz     rtcInfo.PosEx (Line=8517)
;
; Line=8522 - Offset=107
; ----------------------
00870BCB  jmp     rtcInfo.PosEx (Line=8550)
;
; Line=8524 - Offset=112
; ----------------------
00870BD0  mov     dword ptr [ebp-$04], $FFFFFFFF  ; ANSI: '...'; UNICODE: '...'
;
; Line=8525 - Offset=119
; ----------------------
00870BD7  mov     eax, esi
;
; Line=8525 - Offset=121
; ----------------------
00870BD9  call    System._DynArrayLength
;
; Line=8525 - Offset=126
; ----------------------
00870BDE  mov     edx, eax
;
; Line=8525 - Offset=128
; ----------------------
00870BE0  dec     edx
;
; Line=8525 - Offset=129
; ----------------------
00870BE1  test    edx, edx
;
; Line=8525 - Offset=131
; ----------------------
00870BE3  jle     rtcInfo.PosEx (Line=8550)
;
; Line=8525 - Offset=137
; ----------------------
00870BE9  mov     eax, $00000001                  ; ANSI: '...'; UNICODE: '...'  ; <-- EXCEPTION
;
; Line=8526 - Offset=142
; ----------------------
00870BEE  movzx   ecx, byte ptr [esi+eax-$01]
;
; Line=8526 - Offset=147
; ----------------------
00870BF3  cmp     cl, byte ptr [ebx]
;
; Line=8526 - Offset=149
; ----------------------
00870BF5  jnz     rtcInfo.PosEx (Line=8530)
;
; Line=8526 - Offset=151
; ----------------------
00870BF7  movzx   ecx, byte ptr [esi+eax]
;
; Line=8526 - Offset=155
; ----------------------
00870BFB  cmp     cl, byte ptr [ebx+$02]
;
; Line=8526 - Offset=158
; ----------------------
00870BFE  jnz     rtcInfo.PosEx (Line=8530)
;
; Line=8528 - Offset=160
; ----------------------
00870C00  mov     [ebp-$04], eax
;
; Line=8529 - Offset=163
; ----------------------
00870C03  jmp     rtcInfo.PosEx (Line=8550)
;
; Line=8530 - Offset=168
; ----------------------
00870C08  inc     eax
;
; Line=8531 - Offset=169
; ----------------------
00870C09  dec     edx

Registers:
-----------------------------
EAX: ?       EDI: ?    
EBX: ?       ESI: ?    
ECX: ?       EBP: ?    
EDX: ?       ESP: ?    
EIP: ?       FLG: ?    
EXP: 00870BE9   STK: 045FF7FC

Stack:               Memory Dump:
------------------   ---------------------------------------------------------------------------
045FF838: 00000000   00870BE9: B8 01 00 00 00 0F B6 4C 06 FF 3A 0B 75 11 0F B6  .......L..:.u...
045FF834: 00000000   00870BF9: 0C 06 3A 4B 02 75 08 89 45 FC E9 85 00 00 00 40  ..:K.u..E......@
045FF830: 00000000   00870C09: 4A 75 E2 EB 7F C7 45 FC FF FF FF FF 8B C6 E8 E0  Ju....E.........
045FF82C: 00000000   00870C19: A3 B9 FF 8B D0 2B 55 F4 42 85 D2 7E 67 B8 01 00  .....+U.B..~g...
045FF828: 00000000   00870C29: 00 00 0F B6 4C 06 FF 3A 0B 75 55 0F B6 0C 06 3A  ....L..:.uU....:
045FF824: 05EB8000   00870C39: 4B 02 75 4C 0F B6 4C 06 01 3A 4B 04 75 42 89 45  K.uL..L..:K.uB.E
045FF820: 00001488   00870C49: FC 8B 4D F4 49 83 E9 03 7C 30 41 89 4D F0 C7 45  ..M.I...|0A.M..E
045FF81C: 045FFB00   00870C59: F8 03 00 00 00 8B C8 03 4D F8 0F B6 4C 0E FF 8B  ........M...L...
045FF818: 0064131F   00870C69: 7D F8 47 3A 4C 7B FE 74 09 C7 45 FC FF FF FF FF  }.G:L{.t..E.....
045FF814: 045FFB0C   00870C79: EB 08 FF 45 F8 FF 4D F0 75 DB 83 7D FC 00 7F 04  ...E..M.u..}....
045FF810: 045FFB00   00870C89: 40 4A 75 9E 8B 45 FC 5F 5E 5B 8B E5 5D C3 90 55  @Ju..E._^[..]..U
045FF80C: 00641305   00870C99: 8B EC 53 56 8B DA 8B F0 8B C3 8B D6 E8 5A 80 B9  ..SV.........Z..
045FF808: 045FF814   00870CA9: FF 8B 13 B8 E0 0C 87 00 E8 96 93 B9 FF 8B F0 85  ................
045FF804: 045FFB00   00870CB9: F6 7E 13 8B C3 E8 2D 82 B9 FF 0F B7 15 1A 55 AC  .~....-.......U.
045FF800: 006412D1   00870CC9: 00 66 89 54 70 FE 5E 5B 5D C3 00 B0 04 02 00 FF  .f.Tp.^[].......
045FF7FC: 045FFB00   00870CD9: FF FF FF 01 00 00 00 2E 00 00 00 55 8B EC 83 C4  ...........U....

according to the assembler information it might be related with unicode or ansi but in all my functions i use normal strings and assign them with .asString

[D.Tkalcec (RTC)]

Thank you for further investiragions. To me, the call stack you've just sent means that you have a serious case of corrupted memory. This either means that something you are using is overwriting memory it does not own, or the PC you are using has a faulty RAM unit.

Why?

1) The implementation of the PosEx function which is being called is very simple. It loops through all elements of a dynamic byte array (you can see the code in the rtcInfo unit) to scan for a substring, which is (in this case) CRLF (=13,10). The only reason for that function to end up in a very long loop, would be that the length element of a dynamic byte array being passed to the function was corrupt, and the Length() function returned a very large positive integer.

2) The location from which the PosEx function is being called in the rtcSocketHttpSrvProv unit could ONLY be reached if the FChunked private variable was set to TRUE, which would ONLY be the case if the Client is using Chunked Transfer encoding, or if the memory location of the FChunked variable was overwritten. Since RTC components do NOT implement data sending with Chunked Transfer encoding, it is impossible for the FChunked variable to be set to TRUE when using RTC components on the Client side.

3) Errors which you have posted previously from the INFO logs, indicate that data being deserialized was corrupt.

If the error is ONLY happening on one machine, then I would recommend replacing all RAM units on that PC and see if the error goes away. If the error persists after replacing all RAM units on that PC, or if you can reproduce the error on another PC when running your Application with MultiThreaded=FALSE (in single-threaded), then the problem is either caused by some direct memory manipulation code (like the Move function), or by the use of pointers (either to memory or to objects). For example, the use of objects beyond their destruction (calling Free but not setting the variable to NIL), use of pointers after the memory they are pointing to was released, or by badly written direct memory manipulation code (writing beyond the end of a reserved memory block).

PS. In Delphi 2009 and later (you areu sing Delphi XE2), the default String type is UnicodeString. If you are using "asString" to read and write String variables, only the lower 8 bits of each String character will be copied, ignoring the higher 8 bits.

Best Regards,
Danijel Tkalcec

[Kevin Powick]

I'd really like to see your "cache" class and how it's accessing the database.

[Cesar]

I could send it to you via email, where should i send it?

I'm already testing some of scenarios you gave me.

One important thing is we're not using RTC components in the client side because our clients are mobile apps.

[Kevin Powick]

One important thing is we're not using RTC components in the client side because our clients are mobile apps.

Ok, but the issue is on the server side, right? 

Since you're using functions on the server, then I assume clients communicate via XML-RPC?

Could you paste the class to Pastebin or TinyPaste?

[D.Tkalcec (RTC)]

Since you've just said that you are NOT using RTC components on the Client side, then I have some questions about the Client:

1) How are you preparing requests to send remote function calls to the Server using the RTC format? From your INFO error log, I see that you are using the RTC format to deserialize data (and not XML-RPC).

2) Are you using encryption or compression with remote functions? If yes, then how do you compress and/or encrypt the data on the Client side?

3) Are you using Chunked Transfer encoding on the Client side to send data to the Server? If yes, how is this implemented?

Best Regards,
Danijel Tkalcec

[Cesar]

there are the cache class TNewMemory

and 2 others that have all the db stuff

basically i just get my "master connection" and my "agency connection" which are two different servers and pass it as parameter to whatever functions that need acces to it.

https://gist.github.com/pacharrin/10b457afe3232e320ef9

[D.Tkalcec (RTC)]

Now that you've uploaded some of your units, in addition to the 3 questions above about your Client (see my post above), can you also upload the unit where the TMyConnection and TMyQuery classes are declared and implemented, as well as the unit where the "getiiConfig" class/object is declared and/or implementated? What I see so far, is that a lot of TMyConnection instances are being created, but I don't see what they contain or what they do. Also, I see that a "connection" variable is being passed on to a TMyQuery, but I don't see what type it is or what TMyQuery does.

Best Regards,
Danijel Tkalcec

[Cesar]

TMyQuery and TMyCOnnection are standard components to access mysql its a VCL widely used (http://www.devart.com/mydac/)

and gestii config is just a class that read an ini file on initialization and  expose the variables to all the other classes, its never write again after the initialization so no multithread issues could happen. i could show you only the definition part but not the implementation because it contains sensitive data

Code:

type TGestiiConfig = class(TConfig)
  protected
    procedure loadconfig;override;
  public
    //Base de datos
    dbServer : string;
    dbPort : integer;
    dbUsername : string;
    dbPassword : string;
    dbDatabase : string;
    dbWorkerDatabase : string;
    //worker
    worker_id:string;
    destroy_failed_jobs:boolean;
    sleep_delay : Integer; // 5
    max_attempts : Integer; //25;
    default_priority :Integer; // 0;
    delay_jobs : boolean; //true
    read_ahead : integer; //5
    job_queues : string;

    //server
    serverport: String;

    //directories
    dirupload:string;
    dirtemplates:string;
    dirtemp:string;
    dirimages:string;
    dirreports: string;
    //keys
    key_locationapi:string;
    key_google:string;
end;

[Cesar]

my answers:

1) we have an inhouse implementation in java j2me and android that build the parameters in the RTC format, once the data is built, its send via http post.
2)nope, we are not compressing or encrypting
3)we use the standard http methods java provides, we are not specifically setting the chunked = true.

to clarify this third point i just traced the traffic on my server and this a sample of one of my petitions with the resonse included. this is exactly how its traveling.

Host: gestii.com:8082

User-Agent: UNTRUSTED/1.0 3gpp-gba

Content-Length: 320

X-Nokia-MusicShop-Version: 13.1109.31

X-Nokia-MusicShop-Bearer: GPRS/3G



FC=location;RE=11;userid:S=4"2061";imei:S=15"354875047035164";lat:F=24.759599008246003;lon:F=-107.38657992405301;accu:F=14.550616264343262;bat:S=2"28";free:F=47472.0;total:F=1176576.0;location:RE=1;0:RE=5;cellid:S=7"1331274";lac:S=0"";mnc:S=14"050 (Iusacell)";mcc:S=3"334";count:I=11;ts:D=2013-6-25 17:19:18.0;log:S=0"";HTTP/1.1 200 OK

CONTENT-LENGTH: 53



RE=3;code:I=;ts:D=2013-6-25 17:19:22.251;res:S=2"OK";

Since you've just said that you are NOT using RTC components on the Client side, then I have some questions about the Client:

1) How are you preparing requests to send remote function calls to the Server using the RTC format? From your INFO error log, I see that you are using the RTC format to deserialize data (and not XML-RPC).

2) Are you using encryption or compression with remote functions? If yes, then how do you compress and/or encrypt the data on the Client side?

3) Are you using Chunked Transfer encoding on the Client side to send data to the Server? If yes, how is this implemented?

Best Regards,
Danijel Tkalcec

[Kevin Powick]

Cesar,

Just a thought.  You say it only happens on the production server.  Is there a chance that you are exhausting available database connections because the production server's MySQL database has a different configuration than your test server?  I don't see you managing any type of connection pool.

Even if the above was the case, I would expect more "normal" trapped errors to be seen, but I did notice the following. 

In the unit uGestiiDBUtils, you have the function getConnection(), where you define the connection, but never actually open the connection via something like a .Open method.  Instead, the connection object created by that function is later passed as a parameter to a TMyQuery object.  I'm not sure how MyDAC works, but does TMyQuery auto-open the defined connection? I guess it must or none of your queries would work.

Also, though you do .Free your connections, I don't see that you ever explicitly .Close them first.  Again, maybe MyDAC objects do that for you automatically?  Even if closing is implicit in freeing, connections at MySQL may not immediately become available again until after some type of wait/timeout period on recently lost or closed connections.

Even if I'm totally wrong on the above, it would be wise, IMO, to manage a database connection pool to recycle a small group of existing connections instead of constantly creating new ones.