Request.Params.ItemCount / TRtcHttpValues.PrepareValues taking a very long time

[GeirBerg]

Hi!

We have encountered an issue with the PrepareValues parsing when accessing the Request.Params.ItemCount property.

Basically the client application (using an old Indy version in Delphi 2010) is trying to upload a batch of files to the server (using latest RTC 7.18 2016-Q1 in Delphi XE5).
The files are grouped together in a JSON object and are all base64 encoded. The batches contains as many files as can fit in about 10MB.
Usually this works just fine. However, today we encountered a set of files that triggers a timeout (30 seconds) on the client side.

After a lot of debugging and tracing the problem has been traced back to the code accessing Request.Params.ItemCount.

In this particular instance the internal parsing of the property (PrepareValues) is taking more than 80(!) seconds!
The CPU load is 100% on the core doing the parsing and I believe that FastMM may be the bottleneck. When PrepareValues has
completed I see that the FCnt property of FValues is 19886. To me it looks like the internal tree structure is allocating that many objects.
It may also be worth mentioning that I can see a temporary file is being used.

Now, whether this is due to a bug in the old Indy version on the client side or a bug in RTC I don't know.

As a temporary fix, we can skip this part of our code when using this particular upload functionality.
However, it would be nice to get this fixed so that it won't "blow up in our face" at some later point in time.

Regards.

[D.Tkalcec (RTC)]

To keep memory usage low while receiving the content and to make large file uploads possible, a temporary file is created by the "Request.Params.AddText" method if the content exceeds a pre-defined limit. This limit can be set for each request individually by using the "Request.Params.CacheSize" property before first use of the "Request.Params.AddText" method, or ... globally by changing the RTC_FORMPOST_CASHECIZE variable in the rtcInfo unit (assigning a different value on Application start).

Since the default cache size is 16KB, but your data is roughly 10MB in size and contains tens of thousands of individual fields or files, my guess would be that the use of a temporary file is the main reason for your performance issues. Why? Becase your field values or small files have to be extracted manually by performing tens of thousands of sequential search and read operations on a file.

The quickest solution to your performance problem (receiving a request encoded with FORM-POST containing a large number of field values or small files) is probably to disable the file caching mechanism of the Request.Params property by setting the Request.Params.CacheSize property to -1 before using the the Request.Params.AddText method. Naturally, this only works if your Server has enough memory to handle it.

Best Regards,
Danijel Tkalcec

[GeirBerg]

Hi Danijel,

Unfortunately this seems to have no effect on the speed at all. I have double checked that no temporary file is created. Also to clarify, we are not sending "tens of thousands of individual fields or files". I am now able to reproduce this problem with just a single base64 encoded file inside a simple JSON structure with a handful of fields. It seems that the size of the file(s) is the issue. When testing batch sizes of 1, 2, 5 and 10MB it is almost as if the performance degradation is exponential. Is it possible that the parser is getting confused by the content? As I mentioned earlier we don't really need to do this form-data parsing for this particular function as we know it does not contain any form-data we require (only a single large JSON structure). However, it would be nice to not have to maintain a 'blacklist' of functions in the framework.

For the record, this is how we use Request.Params to check for form-data. Maybe we're doing it wrong?

if Request.Method = 'POST' then
begin
  ...
  Request.Params.CacheSize := -1;  // Added to prevent temporary file use
  Request.Params.AddText(Body);
  ...
  Request.Params.ItemCount;  // Access any property to trigger PrepareValues parsing. If this is not done, then IsFile does not seem to (always) work!
  ...
  if Request.Params.IsFile('file') then
  begin
    ...
    FName := Request.Params['file'];
    ...
    Request.Params.GetFile('file', SStream);
    ...
  end;
end;

This code works fine when triggered by our web browser based clients uploading files, but not when triggered by this particular function in our Win32 (Indy) based client.

Regards.

[D.Tkalcec (RTC)]

To check what kind of content you are receiving in the request, you can use the "Request.ContentType" property. The "Request.Params" property is NOT a general-purpose content parser and should ONLY be used for parsing content in the MULTIPART/FORM-DATA format, or if the content is URL-Encoded using a single character as terminator (usually & or ;). In all other cases, you should NOT use the "Request.Params" property (or any of its sub-properties or methods) at all.

Since the parser does NOT seem to work correctly for content sent from your Indy client, chances are high that your Indy client is using some other format. If that is the case, then you should use something else to parse that content and NOT feed your data to the "Request.Params" property.

Unless you are already doing this, it would also be a good idea to use a separate URI to upload content from your Indy Client and use a separate TRtcDataProvider component (see the "Request.FileName" property) for accepting and processing that data. In other words, do NOT mix code used for processing data from a Web Browser with code used for processing data from your Indy client (or other sources, like 3rd-party Applications), to reduce chances of your Server making a wrong assumption and feeding the wrong parser with the data received.

Best Regards,
Danijel Tkalcec

[GeirBerg]

Thank you Danijel!

It looks like the only necessary change was to add a test for 'multipart/form-data' ContentType before performing the parsing.
Now everything seems to be working correctly!

Regards.

[D.Tkalcec (RTC)]

Thanks for your feedback. I've now released RTC SDK v7.19 (you can find it in the RTC SDK Downloads area), with these two modifications:

Bug in the "Request.Params.IsFile" property is now fixed. The property now works, even if you do not use the "Request.Params.ItemCount" property before. Please keep in mind that fixing the "Request.Params.IsFile" property also means that content parser will now be executed automatically if the "Request.Params.AddText" method was used to store any content, so ... please make sure to NOT use the "AddText" method at all if the content being received is NOT in the "MULTIPART/FORM-DATA" or "URL-Encoded" format.

Performance of the content parser in the "Request.Params" property when parsing "MULTIPART/FORM-DATA" content from a temporary file (when the cache limit is exceeded) should now be improved. Simply put, the old implementation was opening and closing the temporary file for every single file access, while the new implementation opens the file only once when the parser is initiated and closed it at the end, which avoids unnecessary open/close operations on the file, so there should be noticable improvement when processing content with a lot of input fields and/or files.

Best Regards,
Danijel Tkalcec