Memory usage with many clients connected

[Aeter]

Hi,

the code you posted works exactly as expected. Also the second one.

When memory is allocated filling up the array, the process uses 64 MB of RAM memory.
When you deallocate the array setting the length to 0, the process returns to use just 3 MB of RAM.

Shouldn't I see the same behavior when using RTC?
I cannot find a reason why a gateway app with no active connection should use 5 GB or RAM in the task manager.

[D.Tkalcec (RTC)]

Ok, poor example. Different Memory Managers are better optimized for different scenarios and the one you are using is better in handling larger blocks of the same size than the one I was using in my first tests. I'll try to come up with a better test, to show the actual behavior of a Memory Manager in a multi-threaded scenario, where a lot of different-size blocks are constantly allocated and released.

[Aeter]

Sure, just a moment

[D.Tkalcec (RTC)]

Give me a few minutes. I'm working on a new test project.

[Aeter]

Here is the video: https://streamable.com/t14wr

[Aeter]

"a better test, to show the actual behavior of a Memory Manager in a multi-threaded scenario, where a lot of different-size blocks are constantly allocated and released."
Do you mean in the RTC scenario (or any web server scenario) there is no way to free up the arrays when all connections are closed?
How is it possible?

This would make the component unusable also for a medium traffic web server. For example, if every day there are 10.000 accesses to the webserver, so 10.000 connections, the websserver will allocate a certain amount of RAM. The next days, every new 10.000 users (while the old ones are disconnected and went away), the RAM will increase, and so on. After a week, any medium cloud server will have its RAM completely filled up.
This would make the component unuseful for any production server, but I'm sure there must be a solution.

[D.Tkalcec (RTC)]

No. I'm talking about Memory Fragmentation, which can result in the Memory Manager to be forced to keep larger Memory Blocks allocated, even when most of these Blocks are actually unused.

Anyway ... do you actually have a problem with a Server using up all of its RAM after it's been running for a long period of time, or what is the purpose of this thread?

[Aeter]

Correct, our issue is that on our servers the gateway application increases its RAM usage indefinitely.
So we need periodically to restart the gateway application, creating a disservice, and anyway, this forces us to spend a lot of money for a lot of RAM... that should not be in use....

We really tried by ourselves to find the portion of the code that allocates (and not deallocates) the memory, but without success.
An help to optimize this would be really really appreciated, and it can surely result in a great improvement for all users in our opinion.

[D.Tkalcec (RTC)]

1. What is the highest number of Clients your Gateway has to serve at the same time?

2. How much RAM is reserved by the Gateway process when it is under highest load?

3. What is the highest amount of RAM the Gateway has reserved, before it had to be restarted?

[Aeter]

On some servers there can be for example 2000 connections per day, so 2000 sessions.
Day by day, the RAM increases for example up to 10 GB. So a server with 16 GB RAM cannot be enough considering it must also host a database and other processes.
The problem is not the peak. If I know that I need 32 GB of RAM to handle a certain number of sessions, I can buy it. However, that peak cannot sum with the next one.

[D.Tkalcec (RTC)]

How is the Gateway configured?

If you are using the default parameters, a Gateway with 2.000 Sessions would probably need about 1 GB of RAM for socket and internal connection buffers, plus 1 GB for streaming between Clients in 1:1 relations, but if your Clients are streaming data to groups of Clients, or if the Gateway is streaming "the same" data to multiple recipients, the amount of RAM required by the Gateway for output buffers could grow exponentially.

Please note that I'm NOT taking about the RTC SDK here, but about the TRtcGateway component in particular. Because its job is to forward data received from any connected Client to any other connected Client or groups of Clients, it has much higher RAM requirements than any other component included in the RTC SDK.

[D.Tkalcec (RTC)]

This is NOT directly related to the 10 GB of RAM used by the Gateway hosting 2.000 clients, but ... the Memory usage you see in your short Gateway Test (first video) after opening and closing 5.000 connections is probably caused by RAM that remains allocated for the Stack used by RTC Worker Threads. This is normal, because a Thread can NOT work without a Stack. It also won't result in a Server using GBs or RAM, since a Thread only uses 1 MB of RAM by default for Stack (and can also be reduced with a compiler directive).

Anyway ... if you run the "Multi-Threaded Memory Manager Test" (see the source code below), you can reproduce the behavior you've seen in your short Gateway Test. Simply open a number of Threads (click the "Open 1" button multiple times), wait a while for them to allocate RAM (you will see the "Used ... KB RAM" value going up), then click the "Stop All" button and wait for the Threads to release the RAM used for dynamic arrays.

You should see that, even after memory used for dynamic arrays has been released (the label should display "Using 0 KB RAM"), the process is still using several MB of RAM. The amount would depend on the number of Threads you leave open. In my test with 50 threads, it was roughly 50 MB. Memory usage is going to reach a low point again ONLY after you close all the Threads by clicking the "Close 1" button until "Open 0 Threads" and "Used 0 KB RAM" is displayed on the Form.

Here's the source code of my relatively simple "Multi-Threaded Memory Manager test" ...

PAS file ...

Code:

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, ExtCtrls;

type
  TForm1 = class(TForm)
    Open: TButton;
    Close: TButton;
    lMemUse: TLabel;
    Timer1: TTimer;
    Start: TButton;
    Stop: TButton;
    lRun: TLabel;
    lWork: TLabel;
    procedure OpenClick(Sender: TObject);
    procedure CloseClick(Sender: TObject);
    procedure Timer1Timer(Sender: TObject);
    procedure StartClick(Sender: TObject);
    procedure StopClick(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

const
  TEST_SIZE=1*1000;       // 1.000 arrays per Thread
  TEST_MAX=10*1000*1000;  // up to 10 MB per Thread

type
  TByteArr=array of byte;
  TThr=class(TThread)
     public
      arr:array[0..TEST_SIZE] of TByteArr;
      memuse:longint;
      run,work:boolean;
      procedure Execute; override;
      end;

procedure TThr.Execute;
  var
    i,olen,nlen:integer;
  function doRun:boolean;
    begin
    result:=run;
    end;
  begin
  i:=0;memuse:=0;
  while doRun do
    begin
    if work or (memuse>0) then
      begin
      olen:=length(arr[i]);
      SetLength(arr[i],0);
      Dec(memuse,olen);
      if work then
        begin
        case random(6) of
          0: nlen:=random(100)+1;
          1: nlen:=random(1000)+100;
          2: nlen:=random(10000)+1000;
          3: nlen:=random(100000)+10000;
          4: nlen:=random(100000)+30000;
          else nlen:=0;
          end;
        end
      else
         nlen:=0;
     if (nlen>0) and (memuse<TEST_MAX) then
        begin
        if (nlen+memuse>TEST_MAX) then nlen:=TEST_MAX-memuse;
        SetLength(arr[i],nlen);
        Inc(memuse,nlen);
        end;
      end;
    if(i<TEST_SIZE) then
      Inc(i)
    else if Application.Terminated then
      break
    else
      begin
      i:=0;
      Sleep(0);
      end;
    end;
  for i:=0 to TEST_SIZE do Setlength(arr[i],0);
  end;

var
   thr:array of TThr;

procedure TForm1.OpenClick(Sender: TObject);
  var
    i:integer;
  begin
  randomize;
  i:=length(thr);
  SetLength(thr,i+1);
  thr[i]:=TThr.Create(true);
  thr[i].FreeOnTerminate:=true;
  thr[i].memuse:=0;
  thr[i].work:=True;
  thr[i].run:=True;
  thr[i].Suspended:=False;
  end;

procedure TForm1.CloseClick(Sender: TObject);
  var
    i:integer;
    mt:TThr;
  begin
  i:=length(thr);
  if i>0 then
    begin
    Dec(i);
    mt:=thr[i];
    thr[i]:=nil;
    mt.run:=False;
    SetLength(thr,i);
    end;
  end;

procedure TForm1.StartClick(Sender: TObject);
  var
    i:integer;
  begin
  for i:=0 to length(thr)-1 do thr[i].work:=True;
  end;

procedure TForm1.StopClick(Sender: TObject);
  var
    i:integer;
  begin
  for i:=0 to length(thr)-1 do thr[i].work:=False;
  end;

procedure TForm1.Timer1Timer(Sender: TObject);
  var
    i,sumuse,srun,swork:integer;
  begin
  sumuse:=0;srun:=0;swork:=0;
  for i:=0 to length(thr)-1 do
    begin
    Inc(sumuse,thr[i].memuse);
    if thr[i].run then Inc(srun);
    if thr[i].work then Inc(swork);
    end;
  lRun.Caption:='Open '+IntToStr(srun)+' Threads';
  lWork.Caption:='Working '+IntToStr(swork)+' Threads';
  lMemUse.Caption:='Using '+IntToStr(round(sumuse/1024))+' KB RAM';
  end;

end.

DFM file ...

Code:

object Form1: TForm1
  Left = 317
  Top = 193
  Caption = 'Form1'
  ClientHeight = 193
  ClientWidth = 205
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -13
  Font.Name = 'MS Sans Serif'
  Font.Style = []
  OldCreateOrder = False
  PixelsPerInch = 120
  TextHeight = 16
  object lMemUse: TLabel
    Left = 8
    Top = 160
    Width = 82
    Height = 16
    Caption = 'Using 0 Bytes'
  end
  object lRun: TLabel
    Left = 8
    Top = 112
    Width = 97
    Height = 16
    Caption = 'Open 0 Threads'
  end
  object lWork: TLabel
    Left = 8
    Top = 136
    Width = 114
    Height = 16
    Caption = 'Working 0 Threads'
  end
  object Open: TButton
    Left = 8
    Top = 8
    Width = 85
    Height = 41
    Caption = 'Open 1'
    TabOrder = 0
    OnClick = OpenClick
  end
  object Close: TButton
    Left = 99
    Top = 8
    Width = 89
    Height = 41
    Caption = 'Close 1'
    TabOrder = 1
    OnClick = CloseClick
  end
  object Start: TButton
    Left = 8
    Top = 52
    Width = 85
    Height = 41
    Caption = 'Start All'
    TabOrder = 2
    OnClick = StartClick
  end
  object Stop: TButton
    Left = 96
    Top = 52
    Width = 89
    Height = 41
    Caption = 'Stop All'
    TabOrder = 3
    OnClick = StopClick
  end
  object Timer1: TTimer
    OnTimer = Timer1Timer
    Left = 148
    Top = 104
  end
end

[Aeter]

I didn't change anything in the sample Gateway test app. So default configuration.

What I've noticed also using ProcessHacker, that shows also the numbero of threads of the process is that:

1. When creating multiple connections, a certain number of threads is created. I can see these threads in ProcessHacker.
2. When all connections are closed, the number of threads does not decrease. I still can see all threads in ProcessHacker, but the Gateway test app has no active connections
3. If I create again many connections, and the number of these connections are less or the same the previous ones, no new thread is created. But if I create more new connections, more threads are added, and never freed up.

I see many memory blocks allocated of a fixed size, some of them are of 1280 KB and still contain connection information even if the connection is no more present. And at least a 1000 KB memory block for each thread.

Is there a way to clean up these? And remove unused threads (and so the memory they allocated) when no connection is present?

[D.Tkalcec (RTC)]

To check how the Gateway is configured, find assignments to the following global variables and calls to the following global functions (marked with bold in the code snippet below) in your Gateway Project ...

    // global variables from the "rtcThrPool.pas" unit ...
    RTC_THREAD_POOL_OVERSIZE:=eThreads.Value;
    RTC_THREAD_POOL_LIMIT:=eThreads.Value;
    RTC_THREAD_POOL_MAX:=eThreads.Value div 2;

    // global functions from the "rtcConn.pas" unit ...
    rtcSetupReceivingParams(eInPack.Value,eInBuff.Value,eInRead.Value);
    rtcSetupSendingParams(eOutPack.Value,eOutBuff.Value,eOutWrite.Value);

    // Gateway component properties ...
    MyGate.ReceiveSpeedLimit:=eInSpeed.Value;
    MyGate.StreamSpeedLimit:=eOutSpeed.Value;

    if xIPv4.Checked then
      begin
      // IPv4 Server component properties ...
      Server4.MultiThreaded:=xMultiThreaded.Checked;
      Server4.ServerPort:=ePort.Text;
      Server4.Blocking:=xBlockingAPI.Checked;
      Server4.Listen();
      end;
    if xIPv6.Checked then
      begin
      // IPv6 Server component properties ...
      Server6.MultiThreaded:=xMultiThreaded.Checked;
      Server6.ServerPort:=ePort.Text;
      Server6.Blocking:=xBlockingAPI.Checked;
      Server6.Listen();
      end;

Using these, you can configure how the Gateway is going to function.

[D.Tkalcec (RTC)]

The default parameters which you see in the RTC Simple Gateway example Project were used in my stress-tests, where my goal was to find the optimal parameters to get the best possible network throughput for my specific test scenario, where my Server PC had 16 GB of RAM dedicated to the Gateway process and all the Clients were connected with fast GBit connections. If your Clients have lower bandwidth to the Server, or if your Server has limited RAM, you should NOT be using these same values for your production environment. Instead, you should run your own tests to find the optimal values for each one of your Servers.