Slow data transfer

[MickyJ]

We’re having a problem with data transfer performance. The RTC server (V6.38) is HTTP, using fmt_RTC, no encryption and no compression. We’re using RPC and not acting as a web server. All the RTC calls are quick and there’s no problem. The issue is when we are sending and receiving large amounts of data (1MByte) in a call.

For example, we transfer a file from the client to the server by reading a 1MB block of data from the file and putting it into an rtcByteArray. The server receives it and writes it to a file. We repeat this until all the file has been transferred. Functionally, it works without any problems.

However, the transfer speed is very poor. It seems to top out at about 14MB per second no matter the speed of the network. Even on a LAN it doesn't go any faster. Just to make sure there were no network issues we compared it with an FTP server and it was considerably faster.

We've tried changing the amount of data that is transferred in each call, e.g. 128K instead of 1MB, but after lots of testing 1MB was found to be optimal. Larger sizes were slightly faster but of course that would cause memory issues on the server.

We also ruled out the file system being the issue. Reading and writing of the files isn't the bottleneck.

Is there something obvious we may have missed? Should we be transferring files a different way? Is HTTP and/or the call format the issue? e.g. does it need to encode/decode the data?

Any and all suggestions are welcome. Thanks.

[D.Tkalcec (RTC)]

Did you mean 14 MByte/s or 14 MBit/s? If you mean 14 MByte/s, which is around 100 MBit/s, then I think it's pretty good for remote functions. Did you monitor CPU usage on both sides (Client and Server)? If you were using a single connection to make the tests, then your performance will also be limited by the speed of a single CPU core on the Client and the Server.

Anyway ...

Remote Functions were NOT designed for sending files. They were designed for sending objects and structured data. When using remote functions, the Client has to prepare the complete object in memory before sending, then serialize that object for transport into a single byte array, which is put into the transfer buffers and sent out in blocks large enough to fit into socket buffers. On the Server side, the complete request needs to arrive and be placed into receiving buffers, which are then transferred into a single continuous byte array and parsed to create objects in memory, which are sent to the Execute event for processing. Also, when you are sending a large file in multiple requests (or multiple remote function calls) instead of using a single request to do it in one "continuous" data stream, latency starts playing an important role. Send a PING from the Client to the Server and you will see how much time you lose with every single request you send to the Server, only because of latency.

For sending and receiving files, you should use plain HTTP requests and NOT remote functions. In other words, use the TRtcDataRequest component on the Client side and TRtcDataProvider component on the Server side. By utilizing these components, you have complete control of the sending and receiving process, on both sides. In a single request, you can send as much data as you want, completely eliminating the latency effect. You will still be splitting the file in smaller chunks when placing it into the transfer buffers and you will be receiving smaller chunks of different sizes on the Server, but this splitting process will only affect buffer handling, memory usage and CPU usage. It won't result in creating a separate request/response loop for each chunk, so there won't be performance penalties because of the latency effect.

Best Regards,
Danijel Tkalcec

[MickyJ]

Hi, thanks for the reply. For TRtcDataRequest and TRtcDataProvider, there isn't the same type of encryption as in TRtcClientModule and TRtcServerModule? If we didn't want to use SSL, is there any way we can use the same encryption as TRtcClientModule and TRtcServerModule? Thanks.

[D.Tkalcec (RTC)]

That is correct. TRtcDataRequest and TRtcDataProvider components do not provide automatic data encryption like TRtcClientModule and TRtcServerModule do, so you will have to implement the encryption protocol yourself if you do NOT want to use SSL encryption.

Encryption class used by TRtcClientModule and TRtcServerModule components (TRtcCrypt) is implemented in the rtcCrypt unit. It is fairly easy to use, with only a few methods and properties. For usage examples, check TRtcClientModule and TRtcServerModule component implementations.

Best Regards,
Danijel Tkalcec

[MickyJ]

Thanks again, I'll take a look.

[MickyJ]

My server is using a TRtcServerModule connected with a TRtcDataServerLink and is serving various RTC methods. Additionaly there is a TRtcDataProvider connected to the same TRtcDataServerLink as the TRtcServerModule for serving data files.

My client uses an TRtcClientModule with a TRtcHttpClient as a client and is calling various RTC methods from the above server. Additionaly there is a TRtcDataRequest that uses the same TRtcHttpClient as the TRtcClientModule for download data files.

TRtcServerModule and TRtcClientModule are using sessions for validating users which is working fine.

TRtcDataProvider and TRtcDataRequest do not manage any sessions themselves but I would like to share the same session info on the server side if possible. The TRtcDataRequest is using basically the same connection as the TRtcClientModule and TRtcProvider uses the same link as the TRtcSererModule but when the event fires on the TRtcDataProvider OnDataReceived(Sender: TRtcConnection) then TRtcDataServer(Sender).Session is nil.

Is it possible to share sessions this way? If not, how else can it be done?

Thanks

[D.Tkalcec (RTC)]

TRtcClientModule and TRtcServerModule component have automated Session management built into them and is enabled automatically if you use Encryption, or if you enable it with the AutoSessions property. For Sessions to work, TRtcServerModule will create a Session object and send it back to the TRtcClientModule component, after which the TRtcClientModule component will be sending it in each request to the TRtcServerModule component. Even though TRtcClientModule and TRtcDataRequest components are using the same TRtcHttpClient component, they do not share the same Session handling mechanism.

When using the same TRtcHttpClient component for TRtcClientModule and TRtcDataRequest components, to access the same Session data from the TRtcDataProvider component on the Server, you need to send the Session ID manually to the Server in each request when posting the request to the Server with the TRtcDataRequest component, then lock that Session ID for usage (check HaveSession and FindSession methods on the TRtcDataServer component).

If you need an example on using HaveSession and FindSession methods, HERE is a link to the related FAQ topic in the RTC archive. Since you want to use the Session created by the TRtcServerModule component, you will not be using the OpenSession method, but you will be sending the Session ID in the request from the Client and will be using the HaveSession method to check if a session exist and the FindSession method to lock the Session for usage (make Session data available through the Session property).

Best Regards,
Danijel Tkalcec

[MickyJ]

Thanks Danijel, that works great.