|
Title: Server eating up all cpu Post by: Cesar on June 24, 2013, 08:19:37 PM Hello,
I've been having a strange bug in which my server app starts to eat up the 100% of my cpu It's a basic rtchttpServer with the following config Code: object Server: TRtcHttpServer all of my functions onExecute event point to my procedure petitionExecute which looks like this: Code: procedure TForm1.PetitionExecute(Sender: TRtcConnection; cache. petition basically just queries the DB and gives the result of the petition The weird thing is: i'm sure cache.petition never enters in a loop or something that could eat up all the 100% of my cpu i even logged method enter and exit of my cache.petition to ensure it never hangs inside of it. I enabled the DEBUG mode of RTC to see if it has something to do with it and i get this messages. in my .exe.ERROR.log i get just this 2 kinds of errors: Quote 2013-06-21 18:33:20.944; TRtcSocketClientThread.Destroy RtcConn.Free Exception! EAccessViolation: Access violation at address 00000001. Write of address 00000001 2013-06-21 18:33:21.022; SRV:TRtcSocketClientThread.RunJob "DoMessage(2)" Exception! EAccessViolation: Access violation at address 00000000. Write of address 00000000 in the .exe.INFO.log i got this Quote 2013-06-21 20:34:35.897; TRtcAbsValue.FromCode(0) Exception! ERtcInfo: Unknown object type specifier at 1: 164612 - AVAL1|CASA|8346";lat:F 2013-06-24 08:40:05.251; TRtcRecord.from_Code(416) Exception! ERtcInfo: Not enough data. 2013-06-24 08:40:05.251; TRtcRecord.from_Code(416) Exception! ERtcInfo: Not enough data. 2013-06-24 08:40:05.251; TRtcRecord.from_Code(416) Exception! ERtcInfo: Not enough data. 2013-06-24 08:40:05.251; TRtcAbsValue.FromCode(416) Exception! ERtcInfo: Not enough data. 2013-06-24 10:11:17.222; TRtcAbsValue.FromCode(10) Exception! ERtcInfo: Unknown object type specifier at 11: form:S 2013-06-24 10:46:46.100; TRtcRecord.from_Code(417) Exception! ERtcInfo: Not enough data. 2013-06-24 10:46:46.100; TRtcRecord.from_Code(417) Exception! ERtcInfo: Not enough data. 2013-06-24 10:46:46.100; TRtcRecord.from_Code(417) Exception! ERtcInfo: Not enough data. 2013-06-24 10:46:46.100; TRtcAbsValue.FromCode(417) Exception! ERtcInfo: Not enough data. 2013-06-24 11:00:37.130; TRtcAbsValue.FromCode(0) Exception! ERtcInfo: Unknown object type specifier at 1: 99.891157;typ:I 2013-06-24 11:53:39.676; TRtcAbsValue.FromCode(0) Exception! ERtcInfo: Unknown object type specifier at 1: 2"P063121758450 - TITUL|LABO|10108";lat:F 2013-06-24 13:45:10.434; TRtcAbsValue.FromCode(0) Exception! ERtcInfo: Unknown object type specifier at 1: :S i tracked down the one at 11:53:39 and this is the data i tried to response Quote 47=(fid=50; fha=tm0dq4; id=1036124; form=EXTRAJUDICIAL; fol=P063121758450 - TITUL|LABO|10108; lat=19.024571; lon=-98.247895; typ=0; tex=2 NORTE 2005, PUEBLA CENTRO, PUEBLA, PUEBLA, 72000; nid=-1; nha=); I know it because in a future attempt it passes. (the above is the dump of Result.asRecord.GetAsText) so down to the questions: How do I interpret the log files? Is there something i'm doing wrong or is it a bug in the component? is there a way to force the stop of a thread that has going rogue? Title: Re: Server eating up all cpu Post by: Kevin Powick on June 24, 2013, 08:54:10 PM At first glance, it appears that the object "cache" is not being managed in a way that is thread safe.
What is the "cache" object? An instantiation of a data module? If so, is it a data module upon which you've placed database access components? That would explain the issues. Title: Re: Server eating up all cpu Post by: Cesar on June 24, 2013, 09:17:32 PM Hi Kevin,
the cache object is a class with a set of functions that queries a Database using MyDAC. this is mostly a "read-only" class in which writes are only done at creation. it has no extra threads besides the ones of the RTC I've made sure the objects are not shared beetwen request to avoid locks and the only list is needed to be shared its properly isolated in a critical section. also i've been profiling it with several tools to avoid something is wrong with the cache class. I tested it with eurekalog to check for any memoryleaks. I profiled it with aQtime to check for performance and resource leaks and it passed. I implemmented the SMartInspect Log to ensure all my petitions are behaving like it should a no function get looped. I am also sure its not a exception raising in any of my code and that the "cpu spike" is not fired by a specific petition or a specific data trying to retrieve. it has happened at different times with different petitions. One test that i did and might give you guys a hint of something i might be doing wrong. I tested it with the multithreaded property set to false. after several minutes running good the application completely froze. the logs showed that the last request i did receive was answered correctly, but since that it stopped receiving new requests. Another thing is i haven't been able to reproduce it dev environment it only happens in the prod server. I'm using the v6.14 (2013.Q2) version of the SDK and compiling it in Delphi XE2 Title: Re: Server eating up all cpu Post by: D.Tkalcec (RTC) on June 24, 2013, 09:55:04 PM 1) My first guess is that the AV you see in the ERROR LOG is a result of the Server being terminated when a connection is still active. I don't think it has anything to do with your actual problem, unless your code is overwriting memory which it does not own.
2) I see that you are catching all exceptions and calling "smartlog", but are NOT re-raising them. This means that your Client will NOT be notified of any Exceptions that might happen inside the remote function, but will instead get a partial Result from the Server with no clue what happened. If you don't handle exceptions gracefully, you should NOT silently "swallow" them. After you write the exception message to your log (if that is what you want to do), you should call "raise", so the exception goes through to RTC components, where it will be sent back to the Client as "rtc_Exception" type. 3) As Kevin pointed out, when you are working with MultiThreaded=TRUE, any code you use inside RTC events has to be 100% thread safe. This means that each thread has to get its own instance of your objects to work with, or your objects have to be capable of executing methods from multiple threads at the same time. I don't know MyDAC, but all the Database Access components I've been working with require a separate set of all database related components per thread (TSession, TDatabase, TQuery, ...). Using a global set of any DB component from multiple threads is a guarantee for a disasaster. 4) When using critical sections, watch out for a "dead-lock" trap, which is what would happen if two thread instances are trying to lock 2 critical sections in a different order (thread 1 locks CsA, thread 2 locks CsB, thread 1 tries to lock CsB and thread 2 tries to lock CsA). 5) INFO exceptions could mean that memory is being overwritten by something, RAM on the Server is corrupt, or there is a missmatch in the encryption (SecureKey). 6) Without seing more of your code, I can only make guesses and assumptions. But first, you need to find out where the exceptions are happening and where the Server freezes. For example, by changing your function code to ... // uses "Log" function is from the "rtcLog.pas" unit procedure TForm1.PetitionExecute(Sender: TRtcConnection; Param: TRtcFunctionInfo; Result: TRtcValue); begin Log(Param.FunctionName+' enter', 'MAIN'); try cache.petition(Param.FunctionName, Param, Result); Log(Param.FunctionName+' exit', 'MAIN'); Except on E:exception do begin Log(Param.FunctionName+' #ERROR# '+E.ClassName+':'+E.Message,'MAIN'); smartlog('error en petition execute: '+E.ToString,tllWarning); raise; // -> required to forward Exception to the Client! end; end; end; Even if the exceptions happen in different "petitions", this will show you which exceptions happen where, and if one of the "petitions" causes the app to freeze. If you say that the freezing also happens when you are running the Server in single-threaded mode, then this is exactly what you should do. Run the Server with MultiThreaded=FALSE and once it freezes check the logs. Best Regards, Danijel Tkalcec Title: Re: Server eating up all cpu Post by: Cesar on June 25, 2013, 06:59:06 PM i did just as you said logging when i enter and leave each of my functions and sending the error to the client
the last thing i logged was leaving the onExecute Event an then i got this from the call stack. any guess where it might be the problem? Quote EurekaLog 7.0.5.1 Exception: -------------------------------------------------- 2.2 Address: 00870BE9 2.5 Type : EFrozenApplication 2.6 Message: The application seems to be frozen. 2.7 ID : 1AFF0000 2.11 Sent : 0 User: ------------- 3.2 Name : 3.3 Email: Steps to reproduce: ------------ 8.1 Text: Call Stack Information: --------------------------------------------------------------------------------------------------------------------------------------------------------- |Methods |Details|Stack |Address |Module |Offset |Unit |Class |Procedure/Method |Line | --------------------------------------------------------------------------------------------------------------------------------------------------------- |*Exception Thread: ID=5256; Parent=0; Priority=0 | |Class=; Name=MAIN | |DeadLock=0; Wait Chain=thread: [ 1488 / 5256 ] is blocked | |Comment= | |-------------------------------------------------------------------------------------------------------------------------------------------------------| |7FFFFFFE|04 |00000000|00870BEE|GestiiServer.exe|00470BEE|rtcInfo | |PosEx |8526[16] | |00000020|04 |0017FBB4|0096DCC4|GestiiServer.exe|0056DCC4|rtcSocketHttpSrvProv|TRtcSocketHttpServerProvider|InternalRead |477[5] | |00000020|04 |0017FBDC|0096D8A5|GestiiServer.exe|0056D8A5|rtcSocketHttpSrvProv|TRtcSocketHttpServerProvider|TriggerDataReceived |420[213] | |00000020|04 |0017FCA0|00968FA6|GestiiServer.exe|00568FA6|rtcSocketSrvProv |TRtcSocketServerProvider |wsOnDataReceived |1217[70] | |00000020|04 |0017FCD4|009583E0|GestiiServer.exe|005583E0|rtcSockBase |TRtcSocketBase |On_DataReceived |399[7] | |00000020|04 |0017FCE4|0095F9D4|GestiiServer.exe|0055F9D4|rtcWinSocket |TRtcWinSocket |DoRead |1624[8] | |00000020|04 |0017FCFC|0095F876|GestiiServer.exe|0055F876|rtcWinSocket |TRtcWinSocket |DoXRead |1580[1] | |00000020|04 |0017FD04|0095FE80|GestiiServer.exe|0055FE80|rtcWinSocket |TRtcWinSocket |DoMessage |1761[9] | |00000020|04 |0017FD28|00958139|GestiiServer.exe|00558139|rtcSockBase |TRtcSocketBase |Call_Message |278[18] | |00000020|04 |0017FD40|0096000B|GestiiServer.exe|0056000B|rtcWinSocket |TRtcWinSocket |WMASyncSelect |1780[10] | |00000020|04 |0017FD48|00960385|GestiiServer.exe|00560385|rtcWinSocket | |RtcSocketWindowProc |1913[45] | |00000020|04 |0017FDA8|00955EA0|GestiiServer.exe|00555EA0|rtcHWndPool | |RtcMainHWNDWindowProc |126[6] | |00000020|03 |0017FE18|76498989|USER32.dll |00018989|USER32 | |(possible GetWindowLongW+962) | | |00000020|03 |0017FE90|76498AB4|USER32.dll |00018AB4|USER32 | |(possible GetWindowLongW+1261)| | |00000020|03 |0017FEF4|76498B0B|USER32.dll |00018B0B|USER32 | |DispatchMessageW | | |00000020|03 |0017FF04|0062B9D7|GestiiServer.exe|0022B9D7|Vcl.Forms |TApplication |ProcessMessage | | |00000020|03 |0017FF20|0062BA1A|GestiiServer.exe|0022BA1A|Vcl.Forms |TApplication |HandleMessage | | |00000020|03 |0017FF44|0062BD59|GestiiServer.exe|0022BD59|Vcl.Forms |TApplication |Run | | |00000020|04 |0017FF74|0099C257|GestiiServer.exe|0059C257|GestiiServer | |Initialization |72[6] | |00000020|03 |0017FF8C|763FF26F|kernel32.dll |0008F26F|kernel32 | |BaseThreadInitThunk | | --------------------------------------------------------------------------------------------------------------------------------------------------------- Assembler Information: ------------------------------------------------------------------------------------------------ ; rtcInfo.PosEx (Line=8522 - Offset=104) ; -------------------------------------- 00870BC8 dec edx ; ; Line=8522 - Offset=105 ; ---------------------- 00870BC9 jnz rtcInfo.PosEx (Line=8517) ; ; Line=8522 - Offset=107 ; ---------------------- 00870BCB jmp rtcInfo.PosEx (Line=8550) ; ; Line=8524 - Offset=112 ; ---------------------- 00870BD0 mov dword ptr [ebp-$04], $FFFFFFFF ; ANSI: '...'; UNICODE: '...' ; ; Line=8525 - Offset=119 ; ---------------------- 00870BD7 mov eax, esi ; ; Line=8525 - Offset=121 ; ---------------------- 00870BD9 call System._DynArrayLength ; ; Line=8525 - Offset=126 ; ---------------------- 00870BDE mov edx, eax ; ; Line=8525 - Offset=128 ; ---------------------- 00870BE0 dec edx ; ; Line=8525 - Offset=129 ; ---------------------- 00870BE1 test edx, edx ; ; Line=8525 - Offset=131 ; ---------------------- 00870BE3 jle rtcInfo.PosEx (Line=8550) ; ; Line=8525 - Offset=137 ; ---------------------- 00870BE9 mov eax, $00000001 ; ANSI: '...'; UNICODE: '...' ; <-- EXCEPTION ; ; Line=8526 - Offset=142 ; ---------------------- 00870BEE movzx ecx, byte ptr [esi+eax-$01] ; ; Line=8526 - Offset=147 ; ---------------------- 00870BF3 cmp cl, byte ptr [ebx] ; ; Line=8526 - Offset=149 ; ---------------------- 00870BF5 jnz rtcInfo.PosEx (Line=8530) ; ; Line=8526 - Offset=151 ; ---------------------- 00870BF7 movzx ecx, byte ptr [esi+eax] ; ; Line=8526 - Offset=155 ; ---------------------- 00870BFB cmp cl, byte ptr [ebx+$02] ; ; Line=8526 - Offset=158 ; ---------------------- 00870BFE jnz rtcInfo.PosEx (Line=8530) ; ; Line=8528 - Offset=160 ; ---------------------- 00870C00 mov [ebp-$04], eax ; ; Line=8529 - Offset=163 ; ---------------------- 00870C03 jmp rtcInfo.PosEx (Line=8550) ; ; Line=8530 - Offset=168 ; ---------------------- 00870C08 inc eax ; ; Line=8531 - Offset=169 ; ---------------------- 00870C09 dec edx Registers: ----------------------------- EAX: ???? EDI: ???? EBX: ???? ESI: ???? ECX: ???? EBP: ???? EDX: ???? ESP: ???? EIP: ???? FLG: ???? EXP: 00870BE9 STK: 045FF7FC Stack: Memory Dump: ------------------ --------------------------------------------------------------------------- 045FF838: 00000000 00870BE9: B8 01 00 00 00 0F B6 4C 06 FF 3A 0B 75 11 0F B6 .......L..:.u... 045FF834: 00000000 00870BF9: 0C 06 3A 4B 02 75 08 89 45 FC E9 85 00 00 00 40 ..:K.u..E......@ 045FF830: 00000000 00870C09: 4A 75 E2 EB 7F C7 45 FC FF FF FF FF 8B C6 E8 E0 Ju....E......... 045FF82C: 00000000 00870C19: A3 B9 FF 8B D0 2B 55 F4 42 85 D2 7E 67 B8 01 00 .....+U.B..~g... 045FF828: 00000000 00870C29: 00 00 0F B6 4C 06 FF 3A 0B 75 55 0F B6 0C 06 3A ....L..:.uU....: 045FF824: 05EB8000 00870C39: 4B 02 75 4C 0F B6 4C 06 01 3A 4B 04 75 42 89 45 K.uL..L..:K.uB.E 045FF820: 00001488 00870C49: FC 8B 4D F4 49 83 E9 03 7C 30 41 89 4D F0 C7 45 ..M.I...|0A.M..E 045FF81C: 045FFB00 00870C59: F8 03 00 00 00 8B C8 03 4D F8 0F B6 4C 0E FF 8B ........M...L... 045FF818: 0064131F 00870C69: 7D F8 47 3A 4C 7B FE 74 09 C7 45 FC FF FF FF FF }.G:L{.t..E..... 045FF814: 045FFB0C 00870C79: EB 08 FF 45 F8 FF 4D F0 75 DB 83 7D FC 00 7F 04 ...E..M.u..}.... 045FF810: 045FFB00 00870C89: 40 4A 75 9E 8B 45 FC 5F 5E 5B 8B E5 5D C3 90 55 @Ju..E._^[..]..U 045FF80C: 00641305 00870C99: 8B EC 53 56 8B DA 8B F0 8B C3 8B D6 E8 5A 80 B9 ..SV.........Z.. 045FF808: 045FF814 00870CA9: FF 8B 13 B8 E0 0C 87 00 E8 96 93 B9 FF 8B F0 85 ................ 045FF804: 045FFB00 00870CB9: F6 7E 13 8B C3 E8 2D 82 B9 FF 0F B7 15 1A 55 AC .~....-.......U. 045FF800: 006412D1 00870CC9: 00 66 89 54 70 FE 5E 5B 5D C3 00 B0 04 02 00 FF .f.Tp.^[]....... 045FF7FC: 045FFB00 00870CD9: FF FF FF 01 00 00 00 2E 00 00 00 55 8B EC 83 C4 ...........U.... according to the assembler information it might be related with unicode or ansi but in all my functions i use normal strings and assign them with .asString Title: Re: Server eating up all cpu Post by: D.Tkalcec (RTC) on June 25, 2013, 07:58:14 PM Thank you for further investiragions. To me, the call stack you've just sent means that you have a serious case of corrupted memory. This either means that something you are using is overwriting memory it does not own, or the PC you are using has a faulty RAM unit.
Why? 1) The implementation of the PosEx function which is being called is very simple. It loops through all elements of a dynamic byte array (you can see the code in the rtcInfo unit) to scan for a substring, which is (in this case) CRLF (=13,10). The only reason for that function to end up in a very long loop, would be that the length element of a dynamic byte array being passed to the function was corrupt, and the Length() function returned a very large positive integer. 2) The location from which the PosEx function is being called in the rtcSocketHttpSrvProv unit could ONLY be reached if the FChunked private variable was set to TRUE, which would ONLY be the case if the Client is using Chunked Transfer encoding, or if the memory location of the FChunked variable was overwritten. Since RTC components do NOT implement data sending with Chunked Transfer encoding, it is impossible for the FChunked variable to be set to TRUE when using RTC components on the Client side. 3) Errors which you have posted previously from the INFO logs, indicate that data being deserialized was corrupt. If the error is ONLY happening on one machine, then I would recommend replacing all RAM units on that PC and see if the error goes away. If the error persists after replacing all RAM units on that PC, or if you can reproduce the error on another PC when running your Application with MultiThreaded=FALSE (in single-threaded), then the problem is either caused by some direct memory manipulation code (like the Move function), or by the use of pointers (either to memory or to objects). For example, the use of objects beyond their destruction (calling Free but not setting the variable to NIL), use of pointers after the memory they are pointing to was released, or by badly written direct memory manipulation code (writing beyond the end of a reserved memory block). PS. In Delphi 2009 and later (you areu sing Delphi XE2), the default String type is UnicodeString. If you are using "asString" to read and write String variables, only the lower 8 bits of each String character will be copied, ignoring the higher 8 bits. Best Regards, Danijel Tkalcec Title: Re: Server eating up all cpu Post by: Kevin Powick on June 25, 2013, 08:04:49 PM I'd really like to see your "cache" class and how it's accessing the database. Title: Re: Server eating up all cpu Post by: Cesar on June 25, 2013, 08:55:22 PM I could send it to you via email, where should i send it?
I'm already testing some of scenarios you gave me. One important thing is we're not using RTC components in the client side because our clients are mobile apps. Title: Re: Server eating up all cpu Post by: Kevin Powick on June 25, 2013, 09:06:27 PM One important thing is we're not using RTC components in the client side because our clients are mobile apps. Ok, but the issue is on the server side, right? Since you're using functions on the server, then I assume clients communicate via XML-RPC? Could you paste the class to Pastebin or TinyPaste? Title: Re: Server eating up all cpu Post by: D.Tkalcec (RTC) on June 25, 2013, 09:27:12 PM Since you've just said that you are NOT using RTC components on the Client side, then I have some questions about the Client:
1) How are you preparing requests to send remote function calls to the Server using the RTC format? From your INFO error log, I see that you are using the RTC format to deserialize data (and not XML-RPC). 2) Are you using encryption or compression with remote functions? If yes, then how do you compress and/or encrypt the data on the Client side? 3) Are you using Chunked Transfer encoding on the Client side to send data to the Server? If yes, how is this implemented? Best Regards, Danijel Tkalcec Title: Re: Server eating up all cpu Post by: Cesar on June 25, 2013, 09:39:31 PM there are the cache class TNewMemory
and 2 others that have all the db stuff basically i just get my "master connection" and my "agency connection" which are two different servers and pass it as parameter to whatever functions that need acces to it. https://gist.github.com/pacharrin/10b457afe3232e320ef9 Title: Re: Server eating up all cpu Post by: D.Tkalcec (RTC) on June 25, 2013, 10:05:54 PM Now that you've uploaded some of your units, in addition to the 3 questions above about your Client (see my post above), can you also upload the unit where the TMyConnection and TMyQuery classes are declared and implemented, as well as the unit where the "getiiConfig" class/object is declared and/or implementated? What I see so far, is that a lot of TMyConnection instances are being created, but I don't see what they contain or what they do. Also, I see that a "connection" variable is being passed on to a TMyQuery, but I don't see what type it is or what TMyQuery does.
Best Regards, Danijel Tkalcec Title: Re: Server eating up all cpu Post by: Cesar on June 25, 2013, 11:10:46 PM TMyQuery and TMyCOnnection are standard components to access mysql its a VCL widely used (http://www.devart.com/mydac/)
and gestii config is just a class that read an ini file on initialization and expose the variables to all the other classes, its never write again after the initialization so no multithread issues could happen. i could show you only the definition part but not the implementation because it contains sensitive data Code: type TGestiiConfig = class(TConfig) Title: Re: Server eating up all cpu Post by: Cesar on June 25, 2013, 11:22:06 PM my answers:
1) we have an inhouse implementation in java j2me and android that build the parameters in the RTC format, once the data is built, its send via http post. 2)nope, we are not compressing or encrypting 3)we use the standard http methods java provides, we are not specifically setting the chunked = true. to clarify this third point i just traced the traffic on my server and this a sample of one of my petitions with the resonse included. this is exactly how its traveling. Quote Host: gestii.com:8082 User-Agent: UNTRUSTED/1.0 3gpp-gba Content-Length: 320 X-Nokia-MusicShop-Version: 13.1109.31 X-Nokia-MusicShop-Bearer: GPRS/3G FC=location;RE=11;userid:S=4"2061";imei:S=15"354875047035164";lat:F=24.759599008246003;lon:F=-107.38657992405301;accu:F=14.550616264343262;bat:S=2"28";free:F=47472.0;total:F=1176576.0;location:RE=1;0:RE=5;cellid:S=7"1331274";lac:S=0"";mnc:S=14"050 (Iusacell)";mcc:S=3"334";count:I=11;ts:D=2013-6-25 17:19:18.0;log:S=0"";HTTP/1.1 200 OK CONTENT-LENGTH: 53 RE=3;code:I=;ts:D=2013-6-25 17:19:22.251;res:S=2"OK"; Since you've just said that you are NOT using RTC components on the Client side, then I have some questions about the Client: 1) How are you preparing requests to send remote function calls to the Server using the RTC format? From your INFO error log, I see that you are using the RTC format to deserialize data (and not XML-RPC). 2) Are you using encryption or compression with remote functions? If yes, then how do you compress and/or encrypt the data on the Client side? 3) Are you using Chunked Transfer encoding on the Client side to send data to the Server? If yes, how is this implemented? Best Regards, Danijel Tkalcec Title: Re: Server eating up all cpu Post by: Kevin Powick on June 26, 2013, 12:09:27 AM Cesar,
Just a thought. You say it only happens on the production server. Is there a chance that you are exhausting available database connections because the production server's MySQL database has a different configuration than your test server? I don't see you managing any type of connection pool. Even if the above was the case, I would expect more "normal" trapped errors to be seen, but I did notice the following. In the unit uGestiiDBUtils, you have the function getConnection(), where you define the connection, but never actually open the connection via something like a .Open method. Instead, the connection object created by that function is later passed as a parameter to a TMyQuery object. I'm not sure how MyDAC works, but does TMyQuery auto-open the defined connection? I guess it must or none of your queries would work. Also, though you do .Free your connections, I don't see that you ever explicitly .Close them first. Again, maybe MyDAC objects do that for you automatically? Even if closing is implicit in freeing, connections at MySQL may not immediately become available again until after some type of wait/timeout period on recently lost or closed connections. Even if I'm totally wrong on the above, it would be wise, IMO, to manage a database connection pool to recycle a small group of existing connections instead of constantly creating new ones. Title: Re: Server eating up all cpu Post by: Cesar on June 26, 2013, 12:40:15 AM The db connection management was one of my first guesses of the problem,
myDac automatically handles the connection pooling, (in the code i gave you i had it disabled because i was testing) But even with pooling disabled it never goes higher than 50 active connections. also MyDAc handles when you have exahusted your connections and raise an Error. And the destructor of the connection handles the closing. anyways I'm making the changes, to manually close them before freeing them. Title: Re: Server eating up all cpu Post by: Cesar on June 26, 2013, 02:02:30 AM I just made a new test with interesting results.
I logged every time i enter in the PosEx Function i discovered it doesn't get stuck in it, but instead, when the cpu spikes, it gets called again and again with the same parameters and length. now im running a new test logging the contents of InBuffer in the internalRead to see what exactly is sending. unfortunately, my users are already logging out and my traffic might not be enough to make another spike until tomorrow Title: Re: Server eating up all cpu Post by: D.Tkalcec (RTC) on June 26, 2013, 02:53:41 AM I've analyzed the code responsible for reading data with Chunked Transfer encoding and found a possible scenario which could result in an endless loop, calling InternalRead over and over again. I am working on an update now, will be releasing it shortly.
Best Regards, Danijel Tkalcec Title: Re: Server eating up all cpu Post by: Cesar on June 26, 2013, 03:14:47 AM Thanks Danijiel,
let me know if you need to test anything, i'll gladly be your guinea pig ;) Title: Re: Server eating up all cpu Post by: D.Tkalcec (RTC) on June 26, 2013, 03:24:28 AM I've uploaded the RTC SDK v6.15 to the "RTC SDK Downloads" area.
Please let me know if this update has resolved your issue with 100% CPU drain. Best Regards, Danijel Tkalcec Title: Re: Server eating up all cpu Post by: D.Tkalcec (RTC) on June 26, 2013, 09:37:21 PM Is my "guinea pig" dead? Or how should I interpret the long silence? ;)
Title: Re: Server eating up all cpu Post by: Cesar on June 27, 2013, 02:08:43 AM hehehe sorry,
i made the changes yesterday and the server has not spiked up again, it's almost 24 hours since the change so I'll say its officially working now. Thank you very much this bug was cracking my head for the last 3 weeks Title: Re: Server eating up all cpu Post by: D.Tkalcec (RTC) on June 27, 2013, 02:44:06 AM I'm glad you've decided to report this issue. Any RTC Server hosting content for mobile phones could have ended up with the same problem. I'm surprised it wasn't happening more often. Anyway ... I'm going to send an E-Mail now for everyone to download this update, because it's a bug I would rather not have in a production Server.
Best Regards, Danijel Tkalcec |