Access violation in rtutils.dll

[Ryan]

Hi All,

Has anybody had any issues with hit and miss connectivity when using WinInet? I have a very odd one which I have been struggling with for a while. The customer has a Windows 2008 R2 Remote Desktop Host and is trying to run our software on there. On the test server I have set-up it works fine. The client gets intermittent connectivity i.e. it will fail the 1st time. Work the 2nd and 3rd times and fails on the 4th attempt.

It always falls over in TRtcWInetHttpClientProvider.SendHeaderOutEx when calling HttpSendRequest(hRequest, Addr(sndHeader[0]), length(sndHeader), Addr(s[0]), length(s)). The header data is always the same "CONTENT-LENGTH: 61" and the sndHeader length is 20 with the length of s being 61.

The code calling this is as follows:


    with Self.ModuleFunctions do begin
      Prepare('GetTranslationData');
      try
        Param.asBoolean['ReleasedOnly'] := (not IsParamSet('AllLanguages'));
        Param.asString['LangCode'] := NewLangCode;
        Screen.Cursor := crQueryActive;
        FuncResult := Execute(False);
        if FuncResult.isType = rtc_Record then begin
          ResultRec := FuncResult.asRecord;
          if ResultRec.asBoolean['Success'] then begin
            TranslationHelper.RefreshLanguageInfo(ResultRec.asDataSet['LangCodesDS'],
                                                  ResultRec.asDataSet['StringTableDS']);
          end
          else begin
            RTCDataModule.ParseErrorInfo(ResultRec.asString['ErrorInfo'], 'Update Translation Data - (Execute)');
            Application.Terminate;
          end;
          Screen.Cursor := crDefault;
        end
        else if FuncResult.isType = rtc_Exception then begin
          RTCDataModule.ShowDatabaseError('Update Translation Data', 'Execute', FuncResult.asException);
          Application.Terminate;
        end;
        FreeAndNil(FuncResult);
      except
        on E: Exception do begin
          Screen.Cursor := crDefault;
          raise;
        end;
      end;
    end;


Any further ideas for trouble shooting would be appreciated.

Many thanks for any help.

Take care,
Ryan

[D.Tkalcec (RTC)]

When using the WinInet or WinHTTP API, TRtcHttpClients sending requests with content body to the same URI using different content lengths has often resulted in an API error and a disconnect, but this has been fixed in RTC SDK 6.18 and should no longer be an issue in the latest version. Are you having these problems with the latest RTC SDK version?

Best Regards,
Danijel Tkalcec

[Ryan]

Hi Danijel,

Yes I'm using 6.23.

Just to make it clear because I don't think I did in my original post. This is a start-up function which is run before the main form is shown. So the first time he runs the program he gets the error. Then he doesn't for runs 2 & 3 but gets the error again on the 4th time. The project is in Delphi XE3 for Win32 only.

I wondered about dumping the byte data for sndHeader and s to to ensure everything was exactly the same for each run.

Take care,
Ryan

[D.Tkalcec (RTC)]

Does the RtcHttpClient component used in this start-up function have MultiThreaded property set to True or False? Are the components created in code? If yes, can you post the code used to create them?

Best Regards,
Danijel Tkalcec

[Ryan]

HI Danijel,

The RtcHttpClient component is on a data module which is the first item created in application start-up.


    Application.Initialize;
    Application.MainFormOnTaskbar := True;
    Application.Title := 'Prime 4';
    Application.CreateForm(TRTCDataModule, RTCDataModule);
    Application.CreateForm(TPrimeMain, PrimeMain);
    Application.Run;


The MultiThreaded property is false and the call to the get GetTranslationData is done from the OnCreate event on the main form.

[D.Tkalcec (RTC)]

The code in question is being executed from the main thread with no background thread activity, all the conditions are identical every time it is being run, but it throws an AV 50% of the time. If that is correct, then there must be some kind of a memory problem. Either with physical RAM in the machine being used, or in the code executed before the AV happens. Since this happens at the very beginning of the application, there shouldn't be too many places that need re-checking. Often cause of AVs is reuse of variables which have been freed, but not NIL-ed. They can result in memory (now occupied by other variables/objects) to get overwritten.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Have you tried creating a separate Application which would only call that one function and display the result, without doing anything else?

Also ... do you know if the Application is working normally when the startup-up function doesn't throwing the AV, or are there problems happening afterwards?

Best Regards,
Danijel Tkalcec

[Ryan]

Hi Danijel,

I hope you have had a good weekend. Sorry for not getting back to you on Friday. I have to pick my daughter up from day care around 5pm.

To be honest I think there is a problem with his server / set-up as this is the only customer I have with the problem. In answer to your questions I have sent several test programs but this customer isn't very forthcoming with information about their set-up. My first program identified that WinInet was returning a host lookup error (WinInet code 12007). This happened even when using the IP address to make the connection. After several more test programs and running though the STRACE program I finally got a Windows Socket Error code, 10022. He then volunteered the information we has running the program from a network share. I turns out there are some potential problems when applications use windows sockets and is run from a network share. This can affect Windows 7, Windows 2008 R2 servers. It is down to a change in the Windows Filtering Platform (WFP). This link  http://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/3076a9cd-57a0-418d-8de1-07adc3b486bb/socket-fails-with-error-10022-when-application-is-run-from-certain-network-shares-on-vista-and?forum=wsk gives more details on that problem.

We then went back to my Prime program run from the local drive and that is when the random crashes started up again. The last test program I created didn't fail but wasn't a exact copy of the Prime start-up code. This one allowed the main form to appear with a few test buttons on it. Clicking a test button would create the data module and perform the function call. He was able to run this 30 times without failure.

I have got him to run Prime though STRACE but no output is given when rtutils.dll throws the exception. This leads me to believe the exception is occurring before the socket is opened.

Over the weekend I have created a test program which copies the Prime start-up as closely as it can. I've asked him to run it this morning and let me know. I've also created a version of Prime which sends the output buffer to debug view as a hex string. I just wanted to see if the output buffer is being corrupted in some way.

I'm not saying the fault I have found is a problem with the RTC library. I just happens to be the last function call before rtutils.dll throws the exception. I just wondered if anybody else had experienced any similar problems as I am running out of ideas to try.

Many thanks for your help so far.

Take care,
Ryan

[D.Tkalcec (RTC)]

If none of the components from the DataModule are being used by the main Form, try creating the DataModule from the OnCreate event of the main Form, then call the function directly afterwards. The user would then have the same experience as if the DataModule was created before the main Form, but the creation order would be reverse and that might make a difference.

Maybe something on the main Form messes up the memory and overwrites portions occupied by objects on the DataModule, which then results in the AV.

Best Regards,
Danijel Tkalcec

[Ryan]

Hi all,

I just though I would follow up with a work around for this problem in case anybody searching the support forum has the same issue.

Just to re-cap:

• The production system is a Windows 2008 R2 Remote Desktop Host Server.
• My program only fails on this server with no predictable pattern
• The program works on the test environment without issue
• The program works on other remote desktop host servers that this customer has serving other sites without issue
• No other user / customer is reporting a similar problems. (About 1000 users spread over 60 customers)

After taking Danijel suggestion I moved the data module creation to the OnFormCreate event of the main form and comment out everything else. Unfortunately I still got the hit a miss exception in rtutils.dll. So I modified a copy of TRtcWInetHttpClientProvider to dump convert all of the parameters used for the WinInet calles to HEX strings and sent them to the debugger for display. This revealed that the data lengths and HEX strings were identical when comparing a successful run to a failed run.

I then on further to investigate potential memory errors using FastMM. This revealed nothing. Dumping the call stacks before calling  HttpSendRequest revealed they were identical. Using madExcept (http://madshi.net/) I was able to get a call stack listing when rtutills.dll produced the access violation. It showed the following stack:


00000008   0.08527645   [13320] Prime 4 - Access violation at address 70D31ED6 in module 'rtutils.dll'. Write of address 70D31ED6   
00000009   0.62387568   [13320] Prime 4 - 70d31ed6 rtutils.dll                                    
00000010   0.62402391   [13320] Prime 4 - 70d323f1 rtutils.dll                                 TraceRegisterExA   
00000011   0.62416679   [13320] Prime 4 - 6fb55d8c RASAPI32.dll                                RasEnumEntriesW   
00000012   0.62430614   [13320] Prime 4 - 75946ce4 wininet.dll                                 #101   
00000013   0.62445301   [13320] Prime 4 - 75995891 wininet.dll                                 HttpSendRequestA   
00000014   0.62462616   [13320] Prime 4 - 009a20fe Prime4current.exe rtcWInetHttpCliProv   951 TRtcWInetHttpClientProvider.SendHeaderOutEx   


Having had some help from Madshi it turns out that stack tracing into system DLLs can be a bit hit and miss, especially when matching up function names.

In the end I managed to persuade the user to add my EXE file to the data execution prevention (DEP) exception list for the server. As soon as he did this the program worked without any issues. The moment the program was removed from the exception list it failed again. Having set my Windows 7 development machine to use the same DEP setting the fault doesn't occur. The program also works correctly on all our Windows 2008 R2 servers and the test remote desktop host I set-up at the start of this problem. All of which have the same settings for DEP as the customers remote desktop server does.

Given this information I found a Windows API function called SetProcessDEPPolicy http://msdn.microsoft.com/en-us/library/bb736299%28v=vs.85%29.aspx. Depending on the parameter passed into this function it will ask for the process to be included or excluded from the DEP monitoring. The Windows OS needs to have DEP configured to either use "Opt In" or "Opt Out" for the function call to work. The "Opt In" setting is the default setting for desktop systems and the "Opt Out" setting is the default setting for server systems. If the user has changed the configuration to be "Always Off" or "Always On" then this function call will not work.

Here is the function I have created to toggle the DEP protection for my program. It is a modified version of the code show in post 2 of this thread http://stackoverflow.com/questions/8066266/how-can-i-enable-dep-nx-and-aslr-on-a-delphi-2006-or-earlier-executable


procedure ChangeDEPPolicy(Enabled: Boolean);
const
  PROCESS_DEP_DISABLE                     : DWORD=$00000000;
  PROCESS_DEP_ENABLE                      : DWORD=$00000001;
  PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION : DWORD=$00000002;
var
  SetProcessDEPPolicy: function(dwFlags: DWORD): BOOL; stdcall;
begin
  SetProcessDEPPolicy := GetProcAddress(GetModuleHandle(kernel32), 'SetProcessDEPPolicy');
  if Assigned(SetProcessDEPPolicy) then begin
    if Enabled then begin
      if not SetProcessDEPPolicy(PROCESS_DEP_ENABLE) then begin
        OutputDebugString('Unable to opt in to the DEP protection.');
      end;
    end
    else begin
      if not SetProcessDEPPolicy(PROCESS_DEP_DISABLE) then begin
        OutputDebugString('Unable to opt out of the DEP protection.');
      end;
    end;
  end;
end;


Making this the first function call in my program has switched off the DEP checking for my program and it operates on the customers server without any further problems. I know this hasn't got to the real cause of the problem. If anybody has any tips for getting to the cause of problems caused by DEP protection in Windows please let me know.

I hope this information might be of help to somebody if they have a similar problem.

Take care,
Ryan

[D.Tkalcec (RTC)]

Thank you for your detailed explanation of the problem and for a work-around.

Best Regards,
Danijel Tkalcec