Using TRtcTcpServer lost some data

[giancarlo]

Hi Danijel

I have a big problem, I have an application using TRtcTcpServer listening on port 15001 that receive the following data every 10 seconds:

6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>

But sometimes I lost some lines, I'm sure the client is sending the data, but I did not receive it.

the code I'm using to read the data is very simple, but maybe is not the correct:

Code:

procedure TForm1.rtcServerDataReceived(Sender: TRtcConnection);
Var sDato, sError: String;
  sFile: String;
begin
  mm1.Lines.Add(Sender.Read);
end;

My customer use a demo app named Hercules (HW Group), and do the same test, sending the same data and there is not data lost.

What I'm doing wrong?

 I Will apreciate your help to finish this project, thanks in advanced.

Giancarlo

[D.Tkalcec (RTC)]

1. "<CR><LF><LF>"? That's either two lines with <CR> being sent as part of data at the end of the 1st line, or there is a <CR> missing in the 2nd line, or you've added a 2nd <LF> by mistake. What do the Clients use to separate each line of data? <CR><LF> or <LF> or <CR><LF><LF> or something else?

2. Is your Server single-threaded or multi-theaded? (check MultiThreaded property) If your Server is multi-threaded, you should NOT access the GUI directly from the OnDataReceived event (Memo) without using the "Sync" method.

3. Are you doing anything to parse that content you receive to ensure that you have exactly one complete line of data ready for processing, before you forward that to the rest of your Application (where your lines are processed)?

The Sender.Read method gives you the current contents of your receiving buffer (raw and unmodified), but the code you have posted above assumes that you will get exactly one line of data with every call to Sender.Read.

Even though there is a chance that this will happen, it is NOT guaranteed to happen. You might also get only a part of the line in one OnDataReceived event and get the rest of that line in one or more upcoming OnDataReceived events. And if your Server is too busy processing data, you could also end up with multiple "lines" received in a single call to Sender.Read from the OnDataReceived event. All of these combinations are possible, so you have to parse the content received to ensure that you have the data you expect.

Here is an example of parsing the content received and extracting "lines" separated by <CR><LF> (#13#10):

Code:

procedure TForm1.rtcServerDataReceived(Sender: TRtcConnection);
var 
  sLine,myLine:String;
  sLoc:integer;
begin
  sLine:=Sender.Info.asString['line'] + Sender.Read; // Get the string we have stored + current receiving buffers
  repeat
    sLoc:=Pos(#13#10,sLine); // check for line delimiter <CR><LF>
    if sLoc>0 then // line delimiter found?
      begin
      myLine:=Copy(sLine,1,sLoc-1); // get the next line, without the line delimiter
      Delete(sLine,1,sLoc+1); // remove the line and delimiter from processing string

      { Now, "myLine" has exactly one complete line, without the <CR><LF> delimiter
         and "sLine" has the rest of the data (one line and <CR><LF> removed) }
      Memo1.Lines.Add(myLine); // Only as an example! Do NOT use this here is Server is Multi-Threaded!

      end;
    until sLoc<=0;
  Sender.Info.asString['line']:=sLine; // store whatever is left unprocessed
end;

Best Regards,
Danijel Tkalcec

[giancarlo]

Hi Danijel and thanks a lot!

1. "<CR><LF><LF>"? That's either two lines with <CR> being sent as part of data at the end of the 1st line, or there is a <CR> missing in the 2nd line, or you've added a 2nd <LF> by mistake. What do the Clients use to separate each line of data? <CR><LF> or <LF> or <CR><LF><LF> or something else?

a. Yes the clients are sending <CR><LF><LF>, is not a mistake, they are 2 Line Feed.

2. Is your Server single threaded or multi-threaded? (check MultiThreaded property) If your Server is multi-threaded, you should NOT access the GUI directly from the OnDataReceived event (Memo) without using the "Sync" method.


b. The production application is a Windows Services, and it is running MultiThreaded, it is not accessed GUI, there is not GUI, but we are missing some lines. That why I did the code I posted above, it is a test Windows Application and it is single-threaded just for test, but in the test environment, we are missing lines too.

3. Are you doing anything to parse that content you receive to ensure that you have exactly one complete line of data ready for processing, before you forward that to the rest of your Application (where your lines are processed)?

c. Not, I'm not parsing the content, we need the line exactly as it was sent​. In the code you send me, you are removing #13#10#10 it could be the problem, I will test with this code.

The Sender.Read method gives you the current contents of your receiving buffer (raw and unmodified), but the code you have posted above assumes that you will get exactly one line of data with every call to Sender.Read.

Even though there is a chance that this will happen, it is NOT guaranteed to happen. You might also get only a part of the line in one OnDataReceived event and get the rest of that line in one or more upcoming OnDataReceived events. And if your Server is too busy processing data, you could also end up with multiple "lines" received in a single call to Sender.Read from the OnDataReceived event. All of these combinations are possible, so you have to parse the content received to ensure that you have the data you expect.

Here is an example of parsing the content received and extracting "lines" separated by <CR><LF> (#13#10):

Code:

procedure TForm1.rtcServerDataReceived(Sender: TRtcConnection);
var 
  sLine,myLine:String;
  sLoc:integer;
begin
  sLine:=Sender.Info.asString['line'] + Sender.Read; // Get the string we have stored + current receiving buffers
  repeat
    sLoc:=Pos(#13#10,sLine); // check for line delimiter <CR><LF>
    if sLoc>0 then // line delimiter found?
      begin
      myLine:=Copy(sLine,1,sLoc-1); // get the next line, without the line delimiter
      Delete(sLine,1,sLoc+1); // remove the line and delimiter from processing string

      { Now, "myLine" has exactly one complete line, without the <CR><LF> delimiter
         and "sLine" has the rest of the data (one line and <CR><LF> removed) }
      Memo1.Lines.Add(myLine); // Only as an example! Do NOT use this here is Server is Multi-Threaded!

      end;
    until sLoc<=0;
  Sender.Info.asString['line']:=sLine; // store whatever is left unprocessed
end;

Best Regards,
Danijel Tkalcec
[/quote]

[D.Tkalcec (RTC)]

If you need a LINE exactly as it was sent, then you need to PARSE the raw data you receive from Sender.Read to make sure it does contain the LINE you need and nothing else. The purpose of the example parser I've posted above is to make sure that you have one complete LINE (assuming that #13#10 or <CR><LF> is your line separator) and NOT just a part of a line or multiple lines.

TCP/IP is a streaming protocol, which means that there are no implicit LINE or MESSAGE separators built in. You get a serie of raw bytes in the order they were sent, but it is your responsibility to parse that raw data to combine the bytes which belong together and separate the rest.

The OnDataReceived event will be triggered when there is more RAW DATA ready to be read and NOT when there is a new complete "LINE" ready for processing. You can not simply use the Sender.Read method and expect to get the next "LINE" from the Client. You get raw bytes.

For example, the "LINE" you've posted above has 68 bytes of "raw" data:

6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>

If a Client would send you 3 such LINES of data, the TCP/IP stream would look something like this:

6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>

Depending on the API used by the Client, the Network used to send that data (hubs, gateways, routers, proxies, ...), your Server configuration and the time it takes your Server to read each package of that raw data, using one line here to represent a single OnDataReceived event with one call to Sender.Read, the data you will receive on the Server could look like this ...

6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>

... or this ...

6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>
6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>
6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo
<CR><LF><LF>

... or this ...

6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>
6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo
<CR><LF><LF>6583750,067-232-1,1-CV,05-22-17 17:49:46,GIANV,gcarlo<CR><LF><LF>

... or this ...

6583750,
067-232-1,
1-CV,
05-22-17 17:49:46,
GIANV,
gcarlo<CR><LF><LF>
6583750,067-232-1,
1-CV,05-22-17 17:49:46,
GIANV,gcarlo
<CR><LF><LF>6583750,
067-232-1,1-CV,
05-22-17 17:49:46,
GIANV,gcarlo<CR><LF>
<LF>

... or any other combination of raw bytes sent by the Client, either split into multiple packages or combined.

This is why, when using raw TCP/IP, you HAVE TO parse the raw data you receive to extract the content you need. If that is NOT the problem, then how exactly does it look like when a LINE gets "lost"? What does the Client send to the Server and what do you receive on the Server? Are you checking Connect and Disconnect events on the Server?

If you really do "lose" a LINE, your problem might be that your connections are being dropped because the Network becomes over-flooded with packages and the Network either does NOT have enough bandwidth to handle all that data, or the Server is unable to process all Clients requests on time.

Anyway ... you should first update your Server to parse the content it receives to make sure the problem isn't caused by data fragmentation. Also, make sure to log clients connect and disconnect events, so you will know if you are losing TCP/IP packets while the connection is active (which should NOT happen if everything is working correctly) - or if you are losing connections (which would be possible if the Network or your Server can NOT handle the load).

Best Regards,
Danijel Tkalcec