Does TrtcDataProvider.Read support Unicode

[WilliamY]

Hi,

I had the following codes in a TrtcDataProvider.OnDataReceived event:

var
  dta: string;
 ds: TRtcDataServer;
begin
   ds := Sender as TRtcDataServer;

   if ds.Request.Complete then
   begin
      dta := ds.Read;
      ProcessRequest(ds, dta);
   end;
end;

The browser post a json text to the server, including unicode strings(e.g. Chinese characters).  However,  Chinese characters weren't shown in dta, instead all non-unicode characters (see below)  like:

天大23


I'm wondering if its the browser/web server who didn't encode the json properly before sending to my server or its just a RealThinClient issue.  BTW, using Delphi 10.4 Sydney and lastest RealThinClient.


Regards,

William

[D.Tkalcec (RTC)]

All Read and Write methods in the RTC SDK work with raw binary data, which you need to encode and decode depending on the content you are sending or receiving. Each character in the String you get from the Read method, holds exactly one BYTE (8 bits) of the content you have received. And when you send a String using the Write method, the lower 8 bits of each character from your String will be sent, without any conversions taking place, ignoring higher bits (if present).

In short, the RTC SDK does NOT perform any conversions or make any modifications of the data you send or receive. When using Read, ReadEx, Write or WriteEx methods, you are always working with raw binary data. This has been the case since the first version of the RTC SDK and is still the case with the latest RTC SDK version today.

Here's a longer explanation:
https://rtcforum.teppi.net/index.php?topic=39.0

[WilliamY]

I'm sorry but the unicode world is really new to me,  Do you kindly let me know how to do the conversion to get  the proper Unicode strings after calling ds.Read or ds.ReadEx?  Thanks.

[D.Tkalcec (RTC)]

There is no simple answer to your question ... "How to do the conversion to get the proper Unicode strings?" ... because it depends on the content you are converting.

A Unicode String is just an in-memory data type, but there are many ways to convert that in-memory data into something that can be transported over binary transports like TCP/IP. And the method needed for converting that content into a Unicode Text after receiving it, depends on the method used by the sender to convert their Unicode Text into data that can be transported over TCP/IP.

If you receive an UTF-8 encoded String, you can decode it into a Unicode String with the Utf8Decode (takes a RtcString as parameter) or Utf8DecodeEx (takes RtcByteArray as a parameter) function from the "rtcSystem" unit. And if you need to convert a Unicode string into an UTF-8 encoded raw binary string, you can use the Utf8Encode (returns a RtcString) or Utf8EncodeEx (returns a RtcByteArray).

But ... if you are receiving content in any other format, then you will need to use a decoder that handles that format. Even though the RTC SDK does include a few functions for widely used formats like UTF-8 (Utf8Encode/Decode), URL (URL_Encode/Decode) and MIME (Mime_Encode/Decode), these are just some of the formats used (you won't find encoders and decoders for every possible data format in the RTC SDK), and ... it is your own responsibility to correctly encode the content before you send it out and/or decode the content you have received.

If you need more information about working with Unicode in Delphi, try searching for "Delphi Unicode" in www.google.com or www.duckduckgo.com or any other search engine.

[WilliamY]

Ah,  you know what,  I'm always confused about when should be calling UTF8Encode and when should be calling UTF8Decode, and I wrote:  data := Utf8Encode(ds.Read);  so never got right result. Changed to use Utf8Decode, everything is working as expect.  Thanks for your time for anwsering my questions. Much appreciated.

BTW,  I found that when calling ds.Write,  I don't need to do any conversion though.

[D.Tkalcec (RTC)]

When you are working with Unicode (your Text contains characters outside of the ANSI table), you should always Encode your Unicode Text before sending, because the Write method included in the RTC SDK does NOT apply any conversions for you. It might seem that a conversion happens implicitly, when you send some simple text between two machines that use the same code-page, but ... if you don't Encode your content properly before sending and Decode it properly after receiving, you might also end up with garbage (wrong content) when sending Unicode Text between machines that use different code-pages.

PS. Encoding and Decoding are general terms used when converting data from one format into another. And because JSON has its own format for Encoding Unicode Text inside JSON, using a JSON Encoder (or seriallizer) should make your Unicode String ready for transport, even without applying any other Encodings on top of that. But some JSON encoders do NOT handle Unicode Text conversions properly, in which case you could end up with something like an UTF-8 encoded JSON.

[WilliamY]

Thank you