Returning data in JSON format

[garyk1968]

Hi

I am using RTC to develop a lightweight HTTP server which can process REST style requests, query some data and return that data in JSON format. Now if I query using a browser I can get it to work but I need to consume this data on an iphone.

I can get my iphone code to retrieve JSON data from any number of webservices out on the web but not from my RTC webserver so it must be doing something different hence no resultset.

I am using RtcHTTPServer and TRtcDataProvider doing the querying and conversion of the data into JSON myself and simply doing a Write(<jsonstring>) but there must be something else that needs setting perhaps?

Any help most appreciated

Gary

[D.Tkalcec (RTC)]

If your app is working with browsers, my guess would be that you aren't setting some HTTP header values which are required by phones. For example, the Response['Content-Type'] is one of the things most browsers will try to automatically detect, but I doubt you can expect the same from apps running on a mobile device.

There are also other HTTP header values which the app running on a mobile device might expect from a Server. If you use the File_Client demo to request any file from a WebServer running Apache, you will see that the Server is returning values for 'SERVER', 'LAST-MODIFIED', 'ACCEPT-RANGES', 'DATE' and 'ETAG' in its HTTP header, all of which can be set by using the TRtcDataServer(Sender).Response property in the RTC SDK before using WriteHeader or Write to send out the actual content.

Here is an example of HTTP header sent from an Apache Web Server:
DATE: Wed, 10 Feb 2010 16:07:38 GMT
SERVER: Apache/2.2.14 (Unix)
LAST-MODIFIED: Wed, 02 Dec 2009 10:07:07 GMT
ETAG: "573d40b-5810-479bc0981c8ca"
ACCEPT-RANGES: bytes
CONTENT-LENGTH: 22544
CONTENT-TYPE: text/html

You can also use the File_Client demo to send a request to a JSON Server which is working with your iPhone to see what the Server is sending out. The problem could also be in your formatting of the JSON content (sent out with the Write method).

I hope this helps.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

I've taken a short look at the JSON format (Java Script Object Notation) definition and from what I see, it is a rather complex format. And this means that some format elements you are using in your response might either be missing or badly implemented on the implementation running on your iPhone, while those used by other JSON Servers are implemented correctly on the iPhone app responsible for receiving the data.

In other words, if you can't get your JSON response to be translated correctly on the iPhone, it could be due to a number of reasons. Bad data formatting on your side, which might be tolerated by some more complex implementations and thus work from Web Browsers running on PCs (for example, JSON classes you are using to prepare the data before sending might be doing something wrong). Or, you are using some format elements which aren't implemented on the iPhone app, or ... it's simply a fact of missing HTTP headers which are expected by the iPhone app.

Anyway ... if you do find out what is wrong (either in your code or in the RTC SDK), please let us know.

Best Regards,
Danijel Tkalcec

[garyk1968]

Hi Danijel,

Thanks for your response. Yes I thought it might be related to headers, I am now setting the status to 200 OK and the content type to application/json but it still doesnt work. There is a method NSString stringWithContentsOfURL which should return the JSON result and store it in a string, this is before parsing but the string is always nil, Ive been using jsonlint.com to validate my json anyway but I think there is something wrong in the response/request side of things.

I'll carry on and let you know what I find!

Regards

Gary

[garyk1968]

Hi Danijel,

Just as a update I cant get the file_client project to work with any known json services (which I do know work)

The error I always get is EWinSockException 'WSocketResolveHost: Cannot convert host address <website address>'

For example this URL will return JSON;

http://ws.geonames.org/citiesJSON?north=44.1&south=-9.9&east=-22.4&west=55.2&lang=de

but either putting the whole line in the server address field or splitting out the url into the server name and the parameters into the query parameters also returns an error, any ideas?

Many thanks


Gary

[D.Tkalcec (RTC)]

Works just fine here. You are probably splitting the parameters wrong.

Here is what you should enter in the File_Client demo to get the data from the URL you have provided ...

Server Port: 80
Server Address: ws.geonames.org
FileName: citiesJSON
Query parameters: north=44.1&south=-9.9&east=-22.4&west=55.2&lang=de

When I enter the above parameters in File_Client demo, I get this RESPONSE header ...

Status: 200 OK
DATE: Wed, 10 Feb 2010 19:52:49 GMT
SERVER: Apache/2.2.10 (Linux/SUSE)
CACHE-CONTROL: no-cache
ACCESS-CONTROL-ALLOW-ORIGIN: *
VARY: User-Agent
TRANSFER-ENCODING: chunked
CONTENT-TYPE: application/json;charset=UTF-8

And this content body ...

{"geonames":[{"fcodeName":"capital of a political entity","countrycode":"MX","fcl":"P","fclName":"city, village,...","name":"Mexico City","wikipedia":"de.wikipedia.org/wiki/Mexiko-Stadt","lng":-99.1386111,"fcode":"PPLC","geonameId":3530597,"lat":19.4341667,"population":11285654},{"fcodeName":"capital of a political entity","countrycode":"PH","fcl":"P","fclName":"city, village,...","name":"Manila","wikipedia":"","lng":120.9822222,"fcode":"PPLC","geonameId":1701668,"lat":14.6041667,"population":10444527},{"fcodeName":"capital of a political entity","countrycode":"BD","fcl":"P","fclName":"city, village,...","name":"Dhaka","wikipedia":"de.wikipedia.org/wiki/Dhaka","lng":90.4086111,"fcode":"PPLC","geonameId":1185241,"lat":23.7230556,"population":10356500},{"fcodeName":"capital of a political entity","countrycode":"KR","fcl":"P","fclName":"city, village,...","name":"Seoul","wikipedia":"de.wikipedia.org/wiki/Seoul","lng":126.9997222,"fcode":"PPLC","geonameId":1835848,"lat":37.5663889,"population":10349312},{"fcodeName":"capital of a political entity","countrycode":"ID","fcl":"P","fclName":"city, village,...","name":"Jakarta","wikipedia":"de.wikipedia.org/wiki/Jakarta","lng":106.8294444,"fcode":"PPLC","geonameId":1642911,"lat":-6.1744444,"population":8540121},{"fcodeName":"capital of a political entity","countrycode":"JP","fcl":"P","fclName":"city, village,...","name":"Tokyo","wikipedia":"de.wikipedia.org/wiki/Tokio","lng":139.691677093506,"fcode":"PPLC","geonameId":1850147,"lat":35.6895265930799,"population":8336599},{"fcodeName":"capital of a political entity","countrycode":"TW","fcl":"P","fclName":"city, village,...","name":"Taipei","wikipedia":"de.wikipedia.org/wiki/Taipeh","lng":121.525,"fcode":"PPLC","geonameId":1668341,"lat":25.0391667,"population":7871900},{"fcodeName":"capital of a political entity","countrycode":"CN","fcl":"P","fclName":"city, village,...","name":"Beijing","wikipedia":"de.wikipedia.org/wiki/Peking","lng":116.397228240967,"fcode":"PPLC","geonameId":1816670,"lat":39.9074977414405,"population":7480601},{"fcodeName":"capital of a political entity","countrycode":"CO","fcl":"P","fclName":"city, village,...","name":"Bogotá","wikipedia":"de.wikipedia.org/wiki/Bogot%C3%A1","lng":-74.0833333,"fcode":"PPLC","geonameId":3688689,"lat":4.6,"population":7102602},{"fcodeName":"capital of a political entity","countrycode":"HK","fcl":"P","fclName":"city, village,...","name":"Hong Kong","wikipedia":"","lng":114.15007352829,"fcode":"PPLC","geonameId":1819729,"lat":22.2840136009625,"population":7012738}]}


Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Btw ... here is the code you would use in Delphi to prepare a TRtcHttpClient and TRtcDataRequest components if you wanted to send the above example request to the above example Server ...

with RtcHttpClient1 do
  begin
  ServerPort:='80';
  ServerAddr:='ws.geonames.org';
  AutoConnect:=True;
  end;
with RtcDataRequest1 do
  begin
  Client := RtcHttpClient1;
  Request.Host:='ws.geonames.org';
  Request.Method:='GET';
  Request.FileName:='/citiesJSON';
  Request.Query.Text:='north=44.1&south=-9.9&east=-22.4&west=55.2&lang=de';
  Post;
  end;

Best Regards,
Danijel Tkalcec

[garyk1968]

Hi Danijel,

Thanks for your assistance it is most appreciated, I still cant get a valid result from file_client, I will try it on another machine in the morning but I get a 404 error as in the screenshot below



Best regards


Gary

[D.Tkalcec (RTC)]

Looks to me like you are connecting to a different physical Server. The Server you are connecting to is reporting "Apache/2.2.14 (Unix)" in the SERVER header, while the one I am connecting to reports "Apache/2.2.10 (Linux/SUSE)" at the same place, so these can't be the same Servers.

If you are going through a Proxy, the problem could be that the Proxy you are using is incorrectly handling the request. Or, the problem could be with your ISPs DNS Server, in which case the IP address resolution is mismatched.

The domain name "ws.geonames.org" is translated to the IP address 188.40.33.19 by my DNS Server. This is what the new File_Client shows as IP address and is also what I get when I do a PING ws.geonames.org using CMD on Windows.

Try doing a PING ws.geonames.org in command prompt (cmd.exe on Windows XP and later) and see if you get a different IP than me. If you do, try using the IP address 188.40.33.19 in the Server Address field of the File_Client demo and make sure to correct the Host Name after you've modified the Server Address, because the Host Name has to be "ws.geonames.org".

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Since you also have a problem connecting to a JSON Server running on the Internet, I am starting to think that your problem is not even the JSON output of your Server, but that your problem is caused by your firewall or wrong entry in your Internet Service Providers DNS Server.

Are you sure that your iPhone is getting through to your Server, opens a connection on your Server and sends a request?

You can easily test if you Server is accessible from your iPhone by using a Web Browser (Safari?) from your iPhone, regardless of the JSON implementation. Simply drop another rtcDataProvider component on the Server, link it to the RtcHttpServer component and implement its OnCheckRequest and OnDataReceived events as shown in the Quick Start Server lesson 1 to get a simple Server output which can be tested with any WebServer, then run the Server and use your Servers IP address in the WebBrowser on your iPhone.

You can also test your JSON code by placing a breakpoint in the OnCheckRequest event of your Server if you can run the Server in Delphi debugger. Or, if the Server is running on a remote PC where you do not have Delphi installed, you can use the rtcLog unit, call StartLog from your Servers initialization section and use the Log function from the OnCheckRequest event of your rtcDataProvider component to write what requests you are getting in. You can also add logging to the event where you are writing the result out, to see what your Server is sending out.

Also, regardless of your firewall and router settings, you should be able to connect to your Server from your iPhone directly by using your Servers IP address inside LAN. For this to work, you need WLAN and have to set up your iPhone to use WLAN for internet access. This would allow you to focus on getting your Server-side code working with the code used on your iPhone first.

Once you get this working, it is simply a matter of correctly setting up your firewalls to make your Server accessible over the Internet. If a Server works inside LAN, it has to work over the Internet - provided you set up your firewall/router correctly.

You can always test your implementation locally by using a Web Browser. If a HTTP Server is working over "localhost", provided your Server is accessible over the Internet (firewalls set up correctly), your Server-side code will work over the Internet just like it does locally. And once you have made sure that your iPhone client is able to connect to your RTC Server and gets a response from the Server, it's only a matter of correctly using JSON on both sides (Server and Client). If you are manually generating JSON data, the problem could be your code preparing the JSON response. But it could also be the JSON client implementation you are using on the iPhone.

Anyway ... please keep me posted on your progress and let me know if you need more info or help.

Best Regards,
Danijel Tkalcec

[garyk1968]

Hi Danijel,

Its not a firewall issue (I dont think) as I am echoing the requests/query params to a memo field on my server and I can see the requests coming in ok, its just not sending anything back. I'm going to play around with the flie client demo because I can use that to query my server and at least see the response headers and will let you know. Rather than use my own server I will also amend the demo one, I'm not sure if I need to hook up the data provider events to the server events and using the demo server should at least ensure everything is setup correctly

Best Regards

Gary

[D.Tkalcec (RTC)]

1) Is your rtcDataProvider component linked to your RtcHttpServer component (through its "Server" property)?

2) Have you implemented the "OnCheckRequest" event on your rtcDataProvider component and are you calling "Accept" in that event to accept the request which you want to process?

3) Have you implemented the "OnDataReceived" event on your rtcDataProvider component to wait until the complete request has been received from the client (Request.Complete=TRUE) before you start processing the request and sending a response out?

4) What do you mean by "not sending anything back"? If you use the Write method from the Sender:TRtcConnection parameter in the OnDataReceived event, data will be sent out. Is it possible you are calling the Write() procedure from Delphi instead of using the Write() method from the Sender:TRtcConnection component?

Can you post the code of all the events you have implemented on your rtcDataProvider component, so I can see if there is anything wrong there? I am still making only wild guesses and will not be able to help you unless I know what you are doing.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

I am officially confused. In your very first port, when you have started the topic, you wrote <quote> ... if I query using a browser I can get it to work ... </quote> and in your last post you wrote <quote> ... I am echoing the requests/query params to a memo field on my server and I can see the requests coming in ok, its just not sending anything back.</quote>

Can you please clarify:

A) Is your Server really working with a Web Browser? In other words, if you enter something like http://localhost/myURI?x=myparams in the Address field of the Web Browser (I do not know what you are doing, so the URL is only an example), does the Web Browser get a response from the Server which you can save to a file and the file you receive contains the JSON data?

B) When you send a request from your iPhone, does the request get accepted in your OnCheckRequest event and does your Server send a response out the same way it does when the request comes in from a Web Browser?

If you can use a memo to print out request parameters, you can also use it to print out the data you are about to send out with the Write() method. As suggested, you can also use the rtcLog unit with its StartLog and Log methods to add more logging to your code.

If your iPhone is sending the same request to your Server as you Web Browser, and your Server works with the Web Browser by sending out JSON data, then your Server will also send the same response to your iPhone (provided the Server gets the request from the iPhone). There is no code in the RTC SDK which would discriminate one browser while working with another one. Unless you explicitly write something in your own code to NOT accept a request coming from one client, or to send out a different response to a client (depending on the request headers or IP or whatever).

I've just used Safari on my iPhone to browse through web pages hosted by a RTC Server running on my PC. I see no reason why this should not work. If your response is correctly formatted (you've said that you have validated your JSON data with some online app) and your Server is sending the response out to a Web Browser, and you say that your Server is getting requests from your iPhone, but the iPhone app is not receiving a response, then the problem can only be somewhere in the JSON client on the iPhone.

Or am I missing something?

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Is it possible that the client app you are using on your iPhone is working ONLY with HTTP/1.0 and does NOT support the new HTTP/1.1 standard?

The current implementation in the rtcHttpServer is always sending a HTTP/1.1 response out, even if it received the request using HTTP/1.0. All the clients I have tested so far, even those working only with HTTP/1.0, have not had a problem with this. But if the implementation on your iPhone is very lazily written, it could be waiting for the HTTP/1.0 text in the response.

Best Regards,
Danijel Tkalcec

[garyk1968]

I am officially confused. In your very first port, when you have started the topic, you wrote <quote> ... if I query using a browser I can get it to work ... </quote> and in your last post you wrote <quote> ... I am echoing the requests/query params to a memo field on my server and I can see the requests coming in ok, its just not sending anything back.</quote>

Sure and the last thing I want to do is confuse you!

Can you please clarify:

A) Is your Server really working with a Web Browser? In other words, if you enter something like http://localhost/myURI?x=myparams in the Address field of the Web Browser (I do not know what you are doing, so the URL is only an example), does the Web Browser get a response from the Server which you can save to a file and the file you receive contains the JSON data?

Exactly that, I can enter the URL into a browser and it did show the JSON in the browser. Now I have used Response.ContentType and set it to 'application/json' it now prompts me for a filename, I save the file and in that file is json.

B) When you send a request from your iPhone, does the request get accepted in your OnCheckRequest event and does your Server send a response out the same way it does when the request comes in from a Web Browser?

In my OnCheckRequest I am simply writing to a memo field the FilePath.Value[0] and the Query.Value['index'] so when the iphone sends the request the url is http://<server ip>/get/customers?index=A I see two lines in the memo field 'get' and 'A'

If you can use a memo to print out request parameters, you can also use it to print out the data you are about to send out with the Write() method. As suggested, you can also use the rtcLog unit with its StartLog and Log methods to add more logging to your code.

Well I have used your file_client demo and it returns the JSON as well as 200, OK and content-type:application/json from my server but I will add rtcLog as well.

If your iPhone is sending the same request to your Server as you Web Browser, and your Server works with the Web Browser by sending out JSON data, then your Server will also send the same response to your iPhone (provided the Server gets the request from the iPhone). There is no code in the RTC SDK which would discriminate one browser while working with another one. Unless you explicitly write something in your own code to NOT accept a request coming from one client, or to send out a different response to a client (depending on the request headers or IP or whatever).

The only thing I haven't tried is putting the url into safari on the mac I will do that. I run windows in virtual machine on the mac so have been using that to build/test the server from within the virtual machine to itself (i.e. localhost) as well as to another windows box but like I say I can see the requests coming in from the iphone anyway.

I've just used Safari on my iPhone to browse through web pages hosted by a RTC Server running on my PC. I see no reason why this should not work. If your response is correctly formatted (you've said that you have validated your JSON data with some online app) and your Server is sending the response out to a Web Browser, and you say that your Server is getting requests from your iPhone, but the iPhone app is not receiving a response, then the problem can only be somewhere in the JSON client on the iPhone.

Or am I missing something?

Well its not getting as far as the JSON parsing. In objective C strings are objects (NSString) and you simply create an instance of NSString with the initWithContentsOfURL method which creates your instance, goes off to the URL you specify and returns your result, be it HTML, XML or JSON it doesn't care. Once your NSString has the data then you do whatever parsing you need to do. And this is where it makes no sense, calling the webservice I showed you above it works but with RTC it doesn't. I'm not blaming RTC at all, its likely to be me but I'm darned if I can find out why! I will go through the steps you mentioned in you previous post and check all settings/event handlers from the Delphi side and as you say browse on the device, I'm actually running this on the iphone simulator not the device but it shouldn't make any difference.

Thanks for your patience and help!

Gary