ftMemo, ftWideMemo, ftWideString fields and encodings problem

[Max Terentiev]

Hi Danijel,

I have some problems with memo fields encoding:

My app uses 2 database connections: one for SQLite (using DevArt LiteDac) and second to MySQL using TADOConnection/TADOQuery.

Sqlite memo fields mapped as ftMemo, ADO memo fields mapped as ftWideMemo

I copy data to RtcDataSet using DelphiDataSetToRtc then send it to client as JSON.

On client side I show data in db-aware controls using virtual dataset (DevArt VirutalDAC) is this way:

VirutalField.Value:=RtcDataSet.Value[FieldName];

All works good except memo fields:

With ADO's ftWideMemo extra \0 characters added to string so "string" arrived as s\0t\0r\0i\0n\0g\0
With Sqlite ftMemo fields I have ? ? ? ? instead of russian characters (but strings are ok inside database I checked it with SQLite studio, something lost during transfer).

Sqlite ftString fields are ok with russian and english strings.
ADO ftWideString fields have problems with russian characters.

How to fix encodings problems ?

[D.Tkalcec (RTC)]

Even though functions from the "rtcDB" unit (which are provided out-of-the-box) for copying data to and from TDataSet descendants make things easier in the beginning and should work for most field types, these functions can NOT automatically recognize and translate all field types provided by all Database Access components. This is especially true for Memo and BLOB fields.

To correctly extract your data from a TDataSet descendant component and copy that data to a TRtcDataSet component, you should copy the "DelphiDataSetToRtc" function implementation from the "rtcDB" unit into one of your own units, give it a different name and modify your version to correctly copy all DB Field types you are using with Database Access components of your choice.

Also, while using the "Value" property on RTC Value objects should work for the most common data types, in case your data is stored in a special format or encoded in a certain way, you have to read the contents by using the same format used while storing it and decode it after reading - if necessary.

Best Regards,
Danijel Tkalcec

[Max Terentiev]

Hi Danijel,

One additional question: what encoding used in RTC for JSON ? Is its UTF-8 ?

What I mean: if I write:

SomeValue.AsString:=DelphiStringWithRussianChars;

Then get value JSON - is generated JSON will be UTF8 encoded ? Is RTC working this way:

Delphi string->RtcValue->UTF8->JSON
JSON->UTF8->RtcValue->Delphi string

?

[D.Tkalcec (RTC)]

1. To store and read Unicode Text using TRtcValue objects, always use "asText" (isType = rtc_Text), which handles Unicode Encoding and Decoding automatically for you.

2. JSON does NOT use UTF-8 Encoding, but has its own format for Encoding Unicode characters, which TRTCValue objects implement in "toJSON" and "fromJSON" methods - as described by the JSON standard:
http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf

3. "asString" is ONLY suitable for 8-bit content, because it only uses the lower 8 bits of each character, which is similar to using the AnsiString type in Delphi. It can be used for storing anything that does NOT require more than 8 bits per character for storage, like (for example) ANSI, URL-Encoded, MIME-Encoded or UTF8-Encoded content.

Best Regards,
Danijel Tkalcec

[Max Terentiev]

Ok, what encoding used for asText (isType = rtc_Text) ?

I ask because my server's json-rpc API will be open, so I must document in what encoding users should send their commands.

[D.Tkalcec (RTC)]

When serializing and deserializing Strings stored in "asText" (isType = rtc_Text) to and from XML-RPC (to/fromXMLRPC methods) as well as RTC (to/fromCode methods) formats, UTF-8 Encoding and Decoding will be used internally by TRtcValue objects serialization and deserialization methods.

But ... this does NOT apply to JSON and JSON-RPC, since JSON has its own format for storing Strings, which is described in the JSON standard (ECMA 404 - "String" on Page 4 - see link below). TRtcValue objects use the JSON standard for "escaping" Unicode Strings (as described in ECMA 404) when serializing "asText" (isType = rtc_Text) to JSON, and follow the same rules when deserializing JSON back into TRtcValue objects. There is no other Encoding or Decoding performed on Strings by TRtcValue objects when using JSON:
http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf

In a nutshell, if you use JSON or JSON-RPC, there is no need to specify "encoding" for Strings, becuause it is already specified by the JSON standard (unless you add your own Encoding on top of that standard).

If you want to see the exact implementation used for "encoding" and "decoding" Strings to and from JSON when using "asText" with TRtcValue objects, check the "JSON_EncodeString" and "JSON_DecodeString" functions from the "rtcInfo.pas" unit ("Lib" folder in the RealThinClient SDK).

Best Regards,
Danijel Tkalcec

[Max Terentiev]

It's clear now, thanks Danijel !

Very nice support !