Error with RtcDataServer.Read on Linux

[twinsoft]

Hi,

I think there is a problem with the read of the content body when source code is compiled for a linux platform. I have the following code on "onDataReceived" which runs perfectly on windows (and on linux for small contents):

Code:

procedure THttpSrv.DataProv_DataReceived(Sender: TRtcConnection);
var
 FJson: RtcWideString;
begin
 with TRtcDataServer(Sender) do
  try
   if Request.Complete then
    begin
      if Request.Method = 'GET' then
       Write('extOrder API: POST method should be used to access API')
      else
       begin
        FJson := Read; // [b]<-- here is the problem: different result on windows and linux[/b]
        if FJson <> '' then
         begin
          FJson := JSON_DecodeString(FJson);
          Log(msgDebug, Request.Host, FJson);
         end;
       end;
    end;
  except
   on e: Exception do
    Write(e.Message);
  end;
end;

when I compile the server for windows platform, I get the payload (through TRtcDataServer(Sender).Read) no matter how big the payload is. On linux, when the payload is small, everything is working as expected. But for large payload (i.e. large JSON's) I get the last part of the payload (I never get the beginning of the message). Have you got any idea why is this happening? I tried to find out if the read function has something like chunked receive (and the "read" function runs twice and overrides the first part) but I couldn't find something... (I'm waiting for the "Request.Complete" anyway). Looking in demos and examples, we only have cases how to "send" large files, not "get" ones from clients.

By the way, the "ContentIn" and "ContentLength" Variables has the same value on both Windows and Linux...

Thanks in advanced for any help,
Stefanos

[D.Tkalcec (RTC)]

I've never offered support for Linux, because a lot of things that worked flawlessly when compiled for any other platform, eventually crashed when compiled for Linux. If you absolutely HAVE TO target Linux, you might have better luck compiling your Project for Windows and running it on Linux under WINE (https://www.winehq.org/), than getting a reliable native Linux version. But even that is not guaranteed to work.

Anyway ... it must be more than 10 years since I've last used Linux. And even back then, it was only for experimental purposes, but never for anything serious, so ... I'm the wrong person to ask for help with anything Linux-related. If I need a reliable and easy-to-maintain Server, I use Windows.

Best Regards,
Danijel Tkalcec

[twinsoft]

Thank you Danijel for the quick reply.

I second that writing for Linux is sometimes quite challenging and you have right. By the way I managed to find a work around (I think you've mentioned it quite some time ago in an  old post of yours). I've created a global variable and did the "read" outside the "Request.Complete" merging the results to that variable. And inside the "Request.Complete" I used the content of that variable. It's a workaround that is working on both windows and linux OS. The previous code turned like the following:

Code:

procedure THttpSrv.DataProv_DataReceived(Sender: TRtcConnection);
var
 FJson: RtcWideString;
begin
 with TRtcDataServer(Sender) do
  try
   PayLoad := PayLoad + Read;  // <--- here I merge the "read" values to a global variable
   if Request.Complete then
    begin
      if Request.Method = 'GET' then
       Write('extOrder API: POST method should be used to access API')
      else
       begin
         //FJson := Read; // <-- here is the problem: different result on windows and linux. commented out
         FJson := PayLoad; // <-- we pass the content of the global variable to the local one
         PayLoad := ''; // <-- Empty variable for next request

        if FJson <> '' then
         begin
          FJson := JSON_DecodeString(FJson);
          Log(msgDebug, Request.Host, FJson);
         end;
       end;
    end;
  except
   on e: Exception do
    Write(e.Message);
  end;
end;

I'm not sure though, if it is thread-safe this kind of approach. What do you think?

Regards,
Stefanos

[D.Tkalcec (RTC)]

Even though you could be using a single global variable to store your request content in a Single-Threaded Application, this won't work in a Multi-Threaded environment. In a Multi-Threaded environment, you will need a separate storage space per Thread. This is one of the reasons why the Request object has its own private variable storage (the "Info" property), which exists once per Request and will be cleared after sending a Response. Since each Request is guaranteed to be handled from a single Thread at any time, you don't have to worry about concurrent access to Request content (thread-safe).

Here's a version that should work in a Multi-Threaded Server:

Code:

procedure THttpSrv.DataProv_DataReceived(Sender: TRtcConnection);
var
 FJson: RtcWideString;
begin
 with TRtcDataServer(Sender) do
  try
   Request.Info.asString['PayLoad'] := Request.Info.asString['PayLoad'] + Read;  // <--- store request content body
   if Request.Complete then
    begin
      if Request.Method = 'GET' then
       Write('extOrder API: POST method should be used to access API')
      else
       begin
         FJson := Request.Info.asString['PayLoad']; // <-- retrieve previously stored request content body
         Request.Info.asString['PayLoad'] := ''; // <-- clear your request content body storage
         if FJson <> '' then
         begin
          FJson := JSON_DecodeString(FJson);
          Log(msgDebug, Request.Host, FJson);
         end;
       end;
    end;
  except
   on e: Exception do
    Write(e.Message);
  end;
end;

I wouldn't recommend doing this with large files (100+ MB would definitely be too much), or with a Server hosting thousands of Clients, but you should be fine if your request content is limited to a few MB and your Server only has to handle a few Clients.

Best Regards,
Danijel Tkalcec

[twinsoft]

Thank you very much Danijel!

Yes this approach is way much better than using a global variable. In my scenario, I have a lot of concurrent requests but the payload should not be more than a dozen of KBs (surely less than a MB) so I think your solution would be quite optical for my case.

Thank you very much indeed!

Best Regards,
Stefanos

Edit: It is working flawlessly with concurrent connections in Multi-Threaded mode! Thanks again!

[D.Tkalcec (RTC)]

Normally, the request content body should have been buffered by RTC and a single call to Read after Request.Complete=TRUE should have been enough to give you a complete request content body back. Why that is NOT the case on Linux, even though it is handled by the same code on all platforms and works everywhere else, I have absolutely no idea. So ... make sure to test my suggested implementation, because ... who knows, maybe my suggestion won't work on Linux either, for whatever reason.

Out of curiosity ... which RTC SDK version are you using and what did you use to compile your Project for Linux?

Best Regards,
Danijel Tkalcec

[twinsoft]

Yes I've seen that the implementation is the same on all platforms that's why I was puzzled why there is a different outcome on linux...

I am using the v.9.50 version of RTC on RAD 10.3 Rio (Delphi). The application is an API server (console linux application) for CentOS.

By the way so far your solution is working very good (both on stability, integrity of data received, and speed).