Title: Memory usage with many clients connected Post by: Aeter on July 03, 2018, 03:41:49 PM Hello,
our gateway application uses a rtcrecord to keep in memory connected clients. However, it seems there isn't a way to free up the memory when a client disconnects. The memory remains allocated, since the items in the list are not removed but only set to null. Is there another way to manage this? We would see the RAM memory usage that decrease when clients disconnect. procedure TRtcAbsRecord.SetNull(const index:RtcWideString; const Value: boolean); begin if Value then SetObject(index, nil, True); end; FValues.Objects[idx]:=nil; // do not call Delete(idx), or other fields Index positions would change Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on July 03, 2018, 04:05:35 PM The behavior of a "TRtcRecord" class (which you have noticed and reported here) is by design. Unless your Server has only a few users and very low traffic, you should NOT use a TRtcRecord class to keep track of your connected clients, but instead implement your own custom class for that purpose.
Title: Re: Memory usage with many clients connected Post by: Aeter on November 05, 2018, 09:56:50 AM Hello,
we've made a simple test using the sample applications provided with the Component. We used the Gateway Test and the App_Client. We noticed memory is allocated when the App_Client makes many connections. However, this memory is never released, even if the App_Client is disconnected and closed. If you make new connections, the memory increases again. There is no possibility, it seems, the memory usage returns to the initial state, when you open the Gateway Test app. How is it possible? We've prepared a simple video to show what happens. Can I send it by email ? Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 05, 2018, 01:23:49 PM You can upload your video to YouTube (or any other Service that provides free video uploads) and post the link here.
Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 09:59:02 AM Here is the link: https://streamable.com/s3fxn
As you can see, the private bytes allocated by the server application is never released. If you see also in the Windows task manager, the RAM used by the server never returns to the value when it was started, even if all clients are disconnected and sessions expired. How can we avoid this? The best scenario would be the RAM is completely freed when all clients are disconnected and there are no more connections active. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 10:02:18 AM 1. Which Delphi version are you using?
2. Are you using the default Memory Manager included with that Delphi version, or an alternative Memory Manager like FastMM4? 3. Do you know how Memory Management in Delphi works and when/how Operating System Memory is being allocated and released by the Delphi Memory Manager? 4. Are you aware of Memory Fragmentation and how it affects Memory Usage at the Operating System level when Memory is being managed by a Memory Manager (like it is the case in Delphi)? 5. Do you know how a Stack works in Delphi and how each Thread allocates Memory for its Stack when the Thread is created? 6. Do you know how a Thread Pool works and how a Thread Pool with active Threads, even if they are idle, affects Memory usage? RTC SDK does NOT directly allocate memory from the Operating System. Delphi Memory Manager does that. So ... if you want to know much Memory is being used by your Delphi Application, you should ask the Memory Manager you are using and NOT the Operating System. Here's more info about Memory Management in Delphi from the EMBT wiki: http://docwiki.embarcadero.com/RADStudio/Tokyo/en/Memory_Management If you are using an older Delphi version, make sure to use FastMM4 or some other Memory Manager, because the Memory Manager included with older Delphi versions (before it was replaced with FastMM) is poorly implemented and doesn't work well for Multi-Threaded Applications. In short, the old default Memory Manager in Delphi was very slow, caused a lot of memory fragmentation and resulted in memory usage to go higher and higher, until it was completely used up, so you would end up with a non-working Application after prolonged usage or a stress-test. Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 11:21:04 AM Everything is in "default" mode.
Tried with Delphi XE2 or Delphi Berlin, same behavior. Thanks for the points about Memory topics, however, I must be sure what is the cause of the continuosly increasing memory usage despite the absence of any active connection. If you say the problem is the thread pool, how to avoid this? In a real scenario, if I created a web server with RtcSdk, and no client is connected to the webserver, I expect the RAM memory I see in the task manager is less than 100 MB for example. Instead, simply using the sample application, after thousands of connections and thousands of disconnections, the RAM usage remains GBs. How to avoid this? I cannot believe it is an issue of the Delphi memory manager, otherwise this would be a serious limitation to create a real web server with Delphi and RTC without filling the RAM of any real server machine. Is it possible, with the sample application, to obtain this: 1. the gateway test app starts and only shows 50 MB memory usage 2. many connections are made by the test aplication: the gateway test app now uses 700 MB RAM 3. all connections are closed and freed up, all sessions are expired 4. the gateway test app now uses again only 50 MB RAM Isn't it reasonable? Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 12:23:22 PM If your Server has to serve thousands of Clients, then your expectations are unreasonable. If you are reaching a point where your 32-bit Server uses up all of its RAM, then you should consider going to 64-bit and using a PC with more available RAM.
Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 12:53:49 PM I'm sorry but this reply doesn't contain any answer to my question.
Let's assume we're talking only about 64 bit applications and OS. And let's assume I can create with RTC a web server that can allocate tens of GBs of RAM. That's not the point anyway. If the RTC component never frees up the RAM it uses, it isn't a scalable component. But I'm sure it is! It is a component to create an http web server, so isn't it reasonable to expect it can manage hundreds of thousands of connections without wasting tens of GBs of RAM? What about reproducing the issue on your side? I just would know if you can simply obtain this: 1. the gateway test app starts and only shows 50 MB memory usage 2. many connections are made by the test aplication: the gateway test app now uses 700 MB RAM 3. all connections are closed and freed up, all sessions are expired 4. the gateway test app now uses again only 50 MB RAM Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 12:56:25 PM You could try tinkering with socket buffer sizes (see SOCK_* variables in the "rtcSystem.pas" unit) and Thread Pool parameters (see RTC_THREAD_* variables in the "rtcThrPool.pas" unit) to see if you can find values that seriously reduce your Servers total Memory usage, without resulting in much higher CPU usage and/or reduced performance and/or reduced Network throughput.
Performance comes at a cost, so you need to decide what you want. Do you want a fast Server, or a Server that doesn't need much RAM? You can't have both. It's that simple. Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 01:13:55 PM The point is not "much RAM". I can also have a server with 128 GB of RAM.
But, after, let's say, 1 month of thousands of connections and disconnections, it couldn't be enough anyway, since every new connection increases the RAM usage, but every disconnection DOES NOT DECREASE the RAM usage. The question is: why the memory is never freed up? I'm sure you've the solution. You can simply reproduce it with the sample app. This would be a very important tip for all users, and will make the component absolutely scalable. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 01:34:10 PM Like all things in Delphi, when any part of RTC SDK requires Memory for anything, it uses the Delphi Memory Manager to allocate and release Memory, while the Memory Manager handles the rest. So ... what you see at the Operating System level is Memory that is currently RESERVED by the Delphi Memory Manager for use by your Application. This is NOT necessarily how much Memory is IN USE by components of your Application, but ONLY the amount of RAM currently RESERVED by the MEMORY MANAGER for current and possible future use by the Application.
Here's a simple test, completely unrelated to RTC: 1. Allocate 1.000 dynamic arrays of 64 KB in size. 2. Check Memory Usage in the Operating System. 3. Allocate 1 more dynamic array of 100 KB in size (separately). 3. Check Memory Usage in the Operating System. 4. Release the first 1.000 dynamic arrays you've allocated previously. 5. Check Memory Usage in the Operating System and let me know what you see. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 01:50:20 PM By the way ... I've already answered your question "how to reduce RTC memory usage". See my reply above ("rtcSystem.pas" and "rtcThrPool.pas" units), but ... keep in mind that reducing these parameters can have serious negative effects for your Server - like reduced performance, increased CPU usage and reduced network throughput. Going too low can also break your Server completely.
The values I've placed there are a result of my own stress-tests, which were mostly focused on finding the optimal values for the best possible performance, since the most important thing for a Server (IMO) is how fast it can handle all requests and respond, but these do NOT have to be THE BEST possible choices for ALL possible scenarios. For example, a slower CPU with more RAM and higher bandwidth might require increasing buffers to improve performance, while a faster CPU with less RAM and lower bandwidth might reach its peak with lower buffer sizes. So ... if you are bothered by Memory Usage of your RTC Server, feel free to try out different values. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 02:28:09 PM Here's the source code for a simple Memory Manager test (see my reply above) ...
Code: type TByteArr=array of byte; You can do the same with pointers using GetMem and FreeMem. It makes no difference. The problem is NOT that the code above is buggy, or that Delphi doesn't release memory used by dynamic arrays when you set their Length to zero (btw ... dynamic Strings are also dynamic Arrays), the "problem" is that the Memory Manager allocates Memory from the Operating System in LARGER blocks and releases these blocks ONLY if possible AND necessary. If you are NOT happy how the Memory Manager works in Delphi, you can write your own, or use one of the alternatives available out there. FastMM4 (now used with Delphi 2009 and above) was written by a developer who does NOT work for EMBT, and ... "Nexus DB" have their own Memory Manager, which is used with their Database Engine (it is heavily multi-threaded). Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 03:02:33 PM Here's the same with a simple pointer using GetMem/FreeMem ...
Code: var arr:array[0..1000] of pointer; NOTE: In this test, make sure to ONLY click the "Allocate" button ONCE and then click the appropriate "Release" button, because you would otherwise be creating a REAL MEMORY LEAK, since pointers to previously allocated memory blocks would be overwritten by GetMem by pointers to newly allocated blocks. This is NOT the case when using dynamic arrays in Delphi, because dynamic arrays have a state (length), which the "SetLength" function knows about and makes use of it when the array size is changing. Another thing that goes in favor of dynamic arrays vs pointers, is that dynamic arrays are auto-reference-counted, same as Ansi and Unicode Strings. Btw ... because of the two reasons above that go in favor for dynamic arrays, and because dynamic arrays are fully cross-platform, RTC SDK uses dynamic arrays for all its buffers and Delphi objects for the rest. Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 03:05:28 PM Hi,
the code you posted works exactly as expected. Also the second one. When memory is allocated filling up the array, the process uses 64 MB of RAM memory. When you deallocate the array setting the length to 0, the process returns to use just 3 MB of RAM. Shouldn't I see the same behavior when using RTC? I cannot find a reason why a gateway app with no active connection should use 5 GB or RAM in the task manager. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 03:22:52 PM Ok, poor example. Different Memory Managers are better optimized for different scenarios and the one you are using is better in handling larger blocks of the same size than the one I was using in my first tests. I'll try to come up with a better test, to show the actual behavior of a Memory Manager in a multi-threaded scenario, where a lot of different-size blocks are constantly allocated and released.
Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 03:23:33 PM Sure, just a moment
Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 03:27:49 PM Give me a few minutes. I'm working on a new test project.
Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 03:34:35 PM Here is the video: https://streamable.com/t14wr
Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 03:55:16 PM "a better test, to show the actual behavior of a Memory Manager in a multi-threaded scenario, where a lot of different-size blocks are constantly allocated and released."
Do you mean in the RTC scenario (or any web server scenario) there is no way to free up the arrays when all connections are closed? How is it possible? This would make the component unusable also for a medium traffic web server. For example, if every day there are 10.000 accesses to the webserver, so 10.000 connections, the websserver will allocate a certain amount of RAM. The next days, every new 10.000 users (while the old ones are disconnected and went away), the RAM will increase, and so on. After a week, any medium cloud server will have its RAM completely filled up. This would make the component unuseful for any production server, but I'm sure there must be a solution. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 04:09:54 PM No. I'm talking about Memory Fragmentation, which can result in the Memory Manager to be forced to keep larger Memory Blocks allocated, even when most of these Blocks are actually unused.
Anyway ... do you actually have a problem with a Server using up all of its RAM after it's been running for a long period of time, or what is the purpose of this thread? Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 04:56:55 PM Correct, our issue is that on our servers the gateway application increases its RAM usage indefinitely.
So we need periodically to restart the gateway application, creating a disservice, and anyway, this forces us to spend a lot of money for a lot of RAM... that should not be in use.... We really tried by ourselves to find the portion of the code that allocates (and not deallocates) the memory, but without success. An help to optimize this would be really really appreciated, and it can surely result in a great improvement for all users in our opinion. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 05:07:08 PM 1. What is the highest number of Clients your Gateway has to serve at the same time?
2. How much RAM is reserved by the Gateway process when it is under highest load? 3. What is the highest amount of RAM the Gateway has reserved, before it had to be restarted? Title: Re: Memory usage with many clients connected Post by: Aeter on November 06, 2018, 05:24:07 PM On some servers there can be for example 2000 connections per day, so 2000 sessions.
Day by day, the RAM increases for example up to 10 GB. So a server with 16 GB RAM cannot be enough considering it must also host a database and other processes. The problem is not the peak. If I know that I need 32 GB of RAM to handle a certain number of sessions, I can buy it. However, that peak cannot sum with the next one. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 06:25:43 PM How is the Gateway configured?
If you are using the default parameters, a Gateway with 2.000 Sessions would probably need about 1 GB of RAM for socket and internal connection buffers, plus 1 GB for streaming between Clients in 1:1 relations, but if your Clients are streaming data to groups of Clients, or if the Gateway is streaming "the same" data to multiple recipients, the amount of RAM required by the Gateway for output buffers could grow exponentially. Please note that I'm NOT taking about the RTC SDK here, but about the TRtcGateway component in particular. Because its job is to forward data received from any connected Client to any other connected Client or groups of Clients, it has much higher RAM requirements than any other component included in the RTC SDK. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 06, 2018, 08:28:32 PM This is NOT directly related to the 10 GB of RAM used by the Gateway hosting 2.000 clients, but ... the Memory usage you see in your short Gateway Test (first video) after opening and closing 5.000 connections is probably caused by RAM that remains allocated for the Stack used by RTC Worker Threads. This is normal, because a Thread can NOT work without a Stack. It also won't result in a Server using GBs or RAM, since a Thread only uses 1 MB of RAM by default for Stack (and can also be reduced with a compiler directive).
Anyway ... if you run the "Multi-Threaded Memory Manager Test" (see the source code below), you can reproduce the behavior you've seen in your short Gateway Test. Simply open a number of Threads (click the "Open 1" button multiple times), wait a while for them to allocate RAM (you will see the "Used ... KB RAM" value going up), then click the "Stop All" button and wait for the Threads to release the RAM used for dynamic arrays. You should see that, even after memory used for dynamic arrays has been released (the label should display "Using 0 KB RAM"), the process is still using several MB of RAM. The amount would depend on the number of Threads you leave open. In my test with 50 threads, it was roughly 50 MB. Memory usage is going to reach a low point again ONLY after you close all the Threads by clicking the "Close 1" button until "Open 0 Threads" and "Used 0 KB RAM" is displayed on the Form. Here's the source code of my relatively simple "Multi-Threaded Memory Manager test" ... PAS file ... Code: unit Unit1; DFM file ... Code: object Form1: TForm1 Title: Re: Memory usage with many clients connected Post by: Aeter on November 07, 2018, 10:31:52 AM I didn't change anything in the sample Gateway test app. So default configuration.
What I've noticed also using ProcessHacker, that shows also the numbero of threads of the process is that: 1. When creating multiple connections, a certain number of threads is created. I can see these threads in ProcessHacker. 2. When all connections are closed, the number of threads does not decrease. I still can see all threads in ProcessHacker, but the Gateway test app has no active connections 3. If I create again many connections, and the number of these connections are less or the same the previous ones, no new thread is created. But if I create more new connections, more threads are added, and never freed up. I see many memory blocks allocated of a fixed size, some of them are of 1280 KB and still contain connection information even if the connection is no more present. And at least a 1000 KB memory block for each thread. Is there a way to clean up these? And remove unused threads (and so the memory they allocated) when no connection is present? Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 07, 2018, 10:32:46 AM To check how the Gateway is configured, find assignments to the following global variables and calls to the following global functions (marked with bold in the code snippet below) in your Gateway Project ...
// global variables from the "rtcThrPool.pas" unit ... RTC_THREAD_POOL_OVERSIZE:=eThreads.Value; RTC_THREAD_POOL_LIMIT:=eThreads.Value; RTC_THREAD_POOL_MAX:=eThreads.Value div 2; // global functions from the "rtcConn.pas" unit ... rtcSetupReceivingParams(eInPack.Value,eInBuff.Value,eInRead.Value); rtcSetupSendingParams(eOutPack.Value,eOutBuff.Value,eOutWrite.Value); // Gateway component properties ... MyGate.ReceiveSpeedLimit:=eInSpeed.Value; MyGate.StreamSpeedLimit:=eOutSpeed.Value; if xIPv4.Checked then begin // IPv4 Server component properties ... Server4.MultiThreaded:=xMultiThreaded.Checked; Server4.ServerPort:=ePort.Text; Server4.Blocking:=xBlockingAPI.Checked; Server4.Listen(); end; if xIPv6.Checked then begin // IPv6 Server component properties ... Server6.MultiThreaded:=xMultiThreaded.Checked; Server6.ServerPort:=ePort.Text; Server6.Blocking:=xBlockingAPI.Checked; Server6.Listen(); end; Using these, you can configure how the Gateway is going to function. Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 07, 2018, 10:47:52 AM The default parameters which you see in the RTC Simple Gateway example Project were used in my stress-tests, where my goal was to find the optimal parameters to get the best possible network throughput for my specific test scenario, where my Server PC had 16 GB of RAM dedicated to the Gateway process and all the Clients were connected with fast GBit connections. If your Clients have lower bandwidth to the Server, or if your Server has limited RAM, you should NOT be using these same values for your production environment. Instead, you should run your own tests to find the optimal values for each one of your Servers.
Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 07, 2018, 11:07:04 AM About Threads ...
You can reduce the maximum number of Threads created by the RTC Thread Pool to reduce their Memory usage (less Threads = less Memory needed for Thread Stack), but ... even though you can configure the RTC Thread Pool to keep closing Threads when they are "not in use", I strongly advise AGAINST such practice, because this is going to impose the highest negative performance penalty on your Servers at times of the highest load and would effectively make the Thread Pool useless. It would also result in higher memory fragmentation, because the Memory block used for Stack would be moving up and down all the time. Memory used by Worker Threads when they are NOT actively working is negligible (about 1 MB for Stack), but the performance penalty you would have if you were closing them when they are "currently not used" would completely negate all the positive effects of using a Thread Pool. The whole point of a Thread Pool is to avoid performance bottlenecks caused by frequent opening and closing of Threads. The same goes for a Database Connection Pool. It would be useless if you would keep closing "currently unused" Database Connections. Title: Re: Memory usage with many clients connected Post by: Aeter on November 07, 2018, 11:53:01 AM I think basically the gateway works fine also for us with default parameters. I do not want to change them.
What I need to obtain is, for example: 6000 connections -> 250 threads 0 connections -> 0 threads Instead now is: 6000 connections -> 250 threads 0 connections -> still 250 threads Perhaps if we can free up threads, most of the memory allocated will be freed up too. Title: Re: Memory usage with many clients connected Post by: Aeter on November 07, 2018, 11:55:23 AM Ok, I understand. But, even if threads remain allocated, they should free up their information in memory about previous connections.
Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 07, 2018, 11:55:52 AM I'd really like to get this discussion to an end, because my time also costs money and I've already wasted one full day discussing a topic which is completely OFF-TOPIC here. If you find an actual memory leak and can send me a simple Project to reproduce it, I will be happy to take a closer look. But ... if your main problem is that you only have 16 GB RAM on your Server, but your Gateway Project has allocated 10 GB of RAM after a full day of serving 2.000 Clients, then my answer is: GET MORE RAM. Even my phone has 64 GB RAM!
Title: Re: Memory usage with many clients connected Post by: Aeter on November 07, 2018, 11:58:50 AM Anyway, it does not have sense to keep a threadpool of 1000 threads in memory even if there are no connections.
Title: Re: Memory usage with many clients connected Post by: D.Tkalcec (RTC) on November 07, 2018, 12:01:10 PM Read my posts. Either change Gateway parameters, or get more RAM.
|