TRtcHttpClient and authentication data in HTTP header

[xacom]

Hi,

I've got an application that uses TRtcHttpClient and TRtcDataRequest to push message notifications to IP handsets.
The dev environment is CBuilder 6 (yes very old and I'm in the process of upgrading to Embarcadero XE, but we've not found the time to do it yet).
The version of RTC installed is v42010Q3.

This program has worked well for a number of years. However, we've now added support for a new IP handset and I'm finding that the authentication on these new handsets fail after a while (5 to 24 hours).
This is odd, and when I look at the HTTP header with Wireshark, I can see that the authentication data that triggers a reply of "HTTP/1.1 401 Unauthorized" is identical to what previously resulted in a "HTTP/1.1 200 OK" reply.

However, once I restart the program, authentication gets re-negotiated, and different authentication data can be seen in the HTTP header. This authentication data works with the handset for a while, but then after 5 to 24 hours), I start seeing "HTTP/1.1 401 Unauthorized" and the cycle continues.

So... It seems that the authentication data expires (which according to Wiki, is what is supposed to happen with 'nonce' data in a HTTP header), and no re-negotiation occurs until the program is stopped and restarted. To illustrate, in Wireshark I see the following sequence:

(Note: username has been replaced with "****", but in both cases the username is identical and correct)

1) Authentication details in header are accepted:
POST /push HTTP/1.1
Content-Type: text/xml
Content-Length: 99
Host: 10.87.98.111
Cache-Control: no-cache
Authorization: Digest username="****",realm="PUSH Authentication",nonce="137284180",uri="/push",algorithm=MD5,response="6361997b0f697b6d295bacf9e4f1d64c"

2) Authentication details get rejected:
POST /push HTTP/1.1
Content-Type: text/xml
Content-Length: 99
Host: 10.87.98.111
Cache-Control: no-cache
Authorization: Digest username="****",realm="PUSH Authentication",nonce="137284180",uri="/push",algorithm=MD5,response="6361997b0f697b6d295bacf9e4f1d64c"

3) After program restart Authentication gets re-negotiated, nonce and response values change, and the authentication is accepted:
POST /push HTTP/1.1
Content-Type: text/xml
Content-Length: 99
Host: 10.87.98.111
Cache-Control: no-cache
Connection: Keep-Alive
Authorization: Digest username="****",realm="PUSH Authentication",nonce="137289529",uri="/push",algorithm=MD5,response="7e663bdcffb3d191afcbd248befb1395"

So... If I list the likely causes/scenarios for this behaviour, I can think of the following:
a) That my code needs to tell RTC to re-negotiate when it gets an unexpected 'Unauthorized' response. If so, then how?
b) That my code is not using RTC correctly.
c) That the communication between RTC and the phone is not ending nicely, and thus RTC thinks that some kind of 'session' is still open and therefore presumes that the authentication data it used last time is still okay.
d) A bug with RTC that has probably been fixed in a later version.

Can anybody help with some suggestions?

With regard to scenario d), I've just upgraded my dev environment to use a later version of RTC - v602_2012Q3, and hopefully it will fix this issue, but I figured that I'd post this anyway, as time is against me (and I'll need to wait at least half a day to see if it fixes the issue), and even if the later version does fix things, this post might still be interesting for readers.

To help readers evaluate whether scenario's a) and b) might be the cause (which I suspect is the most likely cause), I've included a summary of how the program is using TRtcHttpClient and TRtcDataRequest.

Summary of RTC usage within the program

The components are created at run-time within a thread. To allow messages to be concurrently sent to multiple phones, the threads are managed within a thread pool, with each thread having it's own RTC client and data component. Once a thread is created, they are initialised as follows:

Code:

  PushClient = new TRtcHttpClient(NULL) ;
  DataReq    = new TRtcDataRequest(NULL);
  
  Client->ServerAddr = "";
  Client->ServerPort = "80"; 
  Client->MultiThreaded = false;
  Client->AutoConnect = true;
  Client->UseProxy = false;
  Client->Blocking = true;
  Client->UseSSL = false; 
  Client->UseWinHTTP = false;
  Client->ReconnectOn->ConnectLost = false;
  Client->ReconnectOn->ConnectError = false;
  Client->ReconnectOn->ConnectFail = false;
  Client->UserLogin->UserName = "";
  Client->UserLogin->UserPassword = "";
  Client->UserLogin->CertStoreType = certAny; 

  Client->Timeout->AfterConnect = 0;
  Client->Timeout->AfterConnecting = 0;
  Client->Timeout->AfterDataIn = 0;
  Client->Timeout->AfterDataLost = 0;
  Client->Timeout->AfterDataOut = 0;
  Client->Timeout->AfterDataReceived = 0;
  Client->Timeout->AfterDataSend = 0;
  Client->Timeout->AfterDataSent = 0;

  Client->TimeoutsOfAPI->ConnectTimeout = 3;
  Client->TimeoutsOfAPI->ReceiveTimeout = 3;
  Client->TimeoutsOfAPI->ResolveTimeout = 3;
  Client->TimeoutsOfAPI->ResponseTimeout = 3;
  Client->TimeoutsOfAPI->SendTimeout = 0;

  Client->Request->Method = "POST";

  Client->OnBeginRequest = RtcClientBeginRequest;
  Client->OnConnect = RtcClientConnect;
  Client->OnConnectError = RtcClientConnectError;
  Client->OnConnectFail = RtcClientConnectFail;
  Client->OnConnecting = RtcClientConnecting;
  Client->OnConnectLost = RtcClientConnectLost;
  Client->OnDisconnect = RtcClientDisconnect;
  Client->OnDisconnecting = RtcClientDisconnecting;
  Client->OnException = RtcClientException;
  Client->OnInvalidResponse = RtcClientInvalidResponse;
  Client->OnReconnect = RtcClientReconnect;
  Client->OnRepostCheck = RtcClientRepostCheck;
  Client->OnResponseAbort = RtcClientResponseAbort;
  Client->OnResponseData = RtcClientResponseData;
  Client->OnResponseDone = RtcClientResponseDone;
  Client->OnResponseReject = RtcClientResponseReject;
  Client->OnSessionClose = RtcClientSessionClose;
  Client->OnSessionOpen = RtcClientSessionOpen;

  DataReq->Client = Client;
  DataReq->AutoRepost = 0;
  DataReq->HyperThreading = false;
  DataReq->OnBeginRequest = RtcDataRequestBeginRequest;
  DataReq->OnDataReceived = RtcDataRequestDataReceived;

When It's time to send a message, the following occurs:

Code:

  PushClient->ServerAddr = PhoneIP; 
  PushClient->ServerPort = PhonePort; 
  PushClient->UserLogin->UserName = UserName;
  PushClient->UserLogin->UserPassword = Password;

  DataReq->Request->FileName = "/push";
  DataReq->Request->Method = "POST" ;
  DataReq->Request->ContentType = "text/xml"; 
  PushClient->UserLogin->CertStoreType = certAny;   
  
  PushClient->Connect(true, true);
  DataReq->Post();
  
  /***   We then wait for a reply                        ****/
  /*     Which gets processed by RtcDataRequestDataReceived */
  
  PushClient->SkipRequests();
  PushClient->Disconnect();

The contents of RtcDataRequestBeginRequest is as follows:

Code:

void __fastcall TPushThread::RtcDataRequestBeginRequest(TRtcConnection *Sender)
{
  TRtcDataClient * cl ;

  cl = (TRtcDataClient *) Sender;

  // Write the XML POST Content
  cl->Write(XML_Data);  
}

Can anybody help with some suggestions?

[D.Tkalcec (RTC)]

You are setting the UserLogin->UserName and ->UserPassword parameters, which are ONLY used with the WinInet API for authentication with a (Proxy) Server. The RTC SDK does NOT generate any kind of Authentication headers for you, but simply forwards your UserLogin parameters to the WinInet API before Connect, and the WinInet API handles the generation of Authentication headers - as it sees fit.

Because Authentication has to happen before any data is being sent, it is most likely done in the "Connect" procedure. Unless the WinInet API is smart enough to re-negotiate Authentication headers after receiving a 401 Unauthorized respose, you should force re-negotiation manually by closing and opening the connection (using Disconnect and Connect).

If closing and opening the connection doesn't solve your problem, then I can would suggest removing UserName and UserPassword parameters from the UserLogin property and setting all required HTTP Request headers manually in the OnBeginRequest event, before sending any data with Write or WriteEx.

You can set the Authorization HTTP Header through the (TRtcDataClient *) Sender -> Request -> Value["Authorization"] property in C++ and any other HTTP request headers by replacing "Authorization" with the apropriate HTTP header name (as seen in plain HTTP).

Best Regards,
Danijel Tkalcec

[xacom]

Many thanks Danijel,

You put me on a road that ended up being a very interesting journey. I've still got some experimenting to do and there are still a few things that I don't fully understand, but I've finally managed get my code to handle the Digest authentication instead of relying on wininet. Coding the logic to parse the 'unauthorized' reply, pick out the nonce, realm, algorithm and qop values was a bit annoying, but using md5 and the various algorithms to calculate the correct response value was fun.

I did find however, that my attempts to follow you suggestion of using (TRtcDataClient *) Sender -> Request -> Value["Authorization"] didn't produce the expected result. i.e. it added a
  "Authorization=Digest username=.... etc"
entry to the header and not a
  "Authorization: Digest username=.... etc"
entry.

I also tried using
  cl->Request->Params->AddText("Cache-Control: no-cache\r\n");
  cl->Request->Params->AddText("Connection: Keep-Alive\r\n");
  cl->Request->Params->AddText("Authorization: " + BuildAuthenticationResponse() + "\r\n\r\n");
And this happily added the desired authorization data to the headers, but it got appended after the auto-generated header details in such a way that the authorization data got ignored by the handset. i.e. A Wireshark trace showed an empty line between the auto-generated header details and the data that I appended:

POST /push HTTP/1.1
HOST: 10.67.18.13
CONTENT-TYPE: text/xml
CONTENT-LENGTH: 295

Cache-Control: no-cache
Connection: Keep-Alive
Authorization: Digest username="****",realm="PUSH Authentication",nonce="94668507",uri="/push",algorithm=MD5,response="74a48d3c12188e878da47c3f485c2a24"

..etc..

In the end, I found that using the (TRtcDataClient *) Sender->Request->HeaderText property within the TRtcDataRequest OnBeginRequest event handler produced better results for me.

The example below shows what I did to clear the header data and redo the header data from scratch.

Code:

  TRtcDataClient* cl = (TRtcDataClient*) Sender;

  cl->Request->HeaderText =
        m_method + " " + m_file_name + " HTTP/1.1\r\n" +
        "Host: " + cl->ServerAddr + "\r\n" +
        "Content-Type: " + m_content_type + "\r\n" +
        "Content-Length: " + AnsiString(content.Length()) + "\r\n" +
        "Cache-Control: no-cache\r\n" +
        "Connection: Keep-Alive\r\n" +
        "Authorization: " + BuildAuthenticationResponse() + "\r\n\r\n";

  cl->WriteHeader(true);

To go hand-in-hand with the above code, I also had to use the TRtcDataRequest OnDataReceived event handler as the spot where the 'Unauthorized' reply got handled and that if necessary, where the retry got triggered from. eg.

Code:

TRtcDataClient * cl = (TRtcDataClient *)Sender;

if (cl->Response->StatusCode == 401)
{
     if (m_auth_attempt == 0)
     {
        ++m_auth_attempt;
        ClientLog(">>> Responding to authentication challenge");
        if (ParseAuthenticationChallenge(cl->Response->HeaderText))
        {
           // Retry
           SendNotify();
        }
     }
     else
     {
        ClientLog(">>> Failed to Authenticate");
     }
}

I trust that the way in which I've used RTC above is not flawed and that it's not going to come back and bite me in the future?

Regards,

Justin

[D.Tkalcec (RTC)]

Why would using Request->Value in C++ work differently than using Request.Value[] or Request[] in Delphi and add "Authentication=" to the HTTP header instead of "Authentication:" is beyond me. But I'm glad you've found the HeaderText property, which allows you to work with the complete HTTP headers in a single String.

I don't see much of your code here, so I can't say if it might get back and bite you later, but if you are normally preparing a new request if you get a 401 response, and posting that request as you would with any other request, then I don't see a problem. In most cases, if the implementation works today, it should continue working later - unless something else changes.

Best Regards,
Danijel Tkalcec