RTC Forums

Hi Danijel, I've a Gate Server with 50 clients connected.
One of these was disconnect every 30 seconds.
What should I check?

The most probable cause for this is poor connection between the Gateway and the Gate Client, but it could also be a number of other things.

To help analyze the problem, please answer these questions first:

1. Which RTC SDK version did you use to compile the Gateway and that specific Gate Client? If you were using some older RTC SDK version, did you try recompiling the Gateway and Gate Clients with the latest RTC SDK version (v7.33) to see if that fixes the problem?

2. Which Delphi version did you use to compile the Gateway and that specific Gate Client?

3. What target platform did you compile the Gateway to and what is the target platform of that specific Gate Client? (Win32, Win64, MacOSX, iOS32, iOS64 or Android)

4. What kind of a connection is the problematic Gate Client using to conencting to the Gateway? Is it inside the same LAN as the Gateway, or over the Internet?

5. If that Gate Client is connecting to the Gateway over the Internet, is the Gate Client using a fast fixed-line internet connection or a mobile connection?

6. How much network bandwidth does your Gateway have available for handling all the Gate Clients?

7. What network/internet bandwidth do your "working" Gate Clients have and what bandwidth does your problematic Gate Client have?

8. Is that problematic Gate Client beihind a firewall or behind a Proxy or is some Anti-Virus software running on the Client machine? If that Gate Client is running on Window, which Windows version? If it is one of the newer Windows versions, is the "Windows Defender" active?

9. How is the TRtcHttpGateClient component configured on the problematic Gate Client? Please, list all the components properties (except GateAddr and GatePort).

Best Regards,
Danijel Tkalcec

1. RTC 7.33
2. XE10
3. XE10
4. Over internet
5. Fast fixed-line
6. Gateway is on a dedicated server with 200Mb/s bandwidth
7. I'm not sure on this!
8. On this client is only installed AVG antivirus.
9.

The problem is only on a single client, also on this client (and all others) I use RtcModule that connect on other server with a callback connection a it never disconnect.


This is my log:

I found some issue:
Into GateBeforeUserLogin event I check UserAuth and UserInfo parameters and for this client are empty.

Most Anti-Virus software is buffering all outgoing data and forwarding the request ONLY after the complete request content is sent by the Client. To make the Client work in that scenario, you have to disable Clients streaming for outgloing data by setting the StreamBlockSizeOut property to 6 (PING packet size). If you use a higher value, your Client will NOT be able to send PING messages to the Gateway and the Gateway will disconnect the Client because of inactivity on the incoming streaming.

Even though this is less common, some Anti-Virus software and Proxy Servers could also be buffering incoming data and forwarding the response to the Client ONLY after the complete responce has been received from the Gateway, making it impossoble for the Gateway to send periodical PING packages through an open conection to the Client, in whcih case Client will be the one closing the conneciton because of inactivity. If your CLient is behind such a Proxy or Anti-Virus software, you will also have to disable the incoming stream on the Client by setting the StreamBlockSizeIn property to 6.

Let me know if changing either of these parameters makes a difference for that Client.

Best Regards,
Danijel Tkalcec

As for the "UserInfo" and "UserAuth" parameters being empty on the Gateway, implement the AfterLoggedIn event on the TRtcHttpGateClient component to write down the following values:

Client.GateUserAuth
Client.GateUserInfo
Client.MyUID
Client.MyIP

To check if these values match for the Client, in addition to the info you are already logging on the Gateway inside the BeforeUserLogin event, write down these parameters:

TRtcDataServer(Sender).Request.URI
TRtcDataServer(Sender).Request.HeaderText
TRtcDataServer(Sender).PeerAddr

Let me know what comes out on both ends.

Best Regards,
Danijel Tkalec

I try with StreamBlock beacuse the UseInfo is needed to ahndle before login to check if client have valid information to be logged in, and if I use AfterLogin this information is blank into onBeforeLogin on the Gate.

I've tested by changing StreamBlockSizeOut but it doesn't work.

I need this values
Client.GateUserAuth
Client.GateUserInfo
not empty into BeforeUserLogin (Gate Server) because I check this values to check a valid login.

If I implement the AfterLoggedIn this values became empty into GateBeforeUserLogin event.

StreamBlockSize properties only define streaming properties.
They have nothing to do with UserInfo and UserAuth parameters.

1. Did you change GateCli:TRtcHttpGateClient component to use this?
StreamBlockSizeOut := 6;
StreamBlockSizeIn := 6

2. Did you write AfterLoggedIn event for GateCli:TRtcHttpGateClient component to log this?
Log('UserAuth='+Client.GateUserAuth);
Log('UserInfo='+Client.GateUserInfo);
Log('UID='+IntToStr(Client.MyUID));
Log('IP='+Client.MyIP);

This event should NOT make any changes to the component. It should ONLY log current property values.

3. Did you write the OnBeforeLogin event on the Gateway to log this?
Log('URI='+TRtcDataServer(Sender).Request.URI);
Log('Header='+TRtcDataServer(Sender).Request.HeaderText);
Log('Addr='+TRtcDataServer(Sender).PeerAddr);

Best Regards,
Danijel Tkalcec

1. Oh :) yes I know...

2. I populate Client.GateUserInfo on client like:

and check this value into GateBeforeUserLogin event on server.
Is this the correct method or not?

This is the log:

2016-08-29 17:52:01.278; Addr=95.231.151.x
2016-08-29 17:52:01.292; URI=/$l
2016-08-29 17:52:01.302; Header=CONTENT-LENGTH: 48


I've do that because into your demos I see that:

 { User is logging in. Check "UserAuth" if Login is allowed and raise an exception if not.
    Because we ONLY want to accept users with UserAuth "MyChatClient",
    we will raise an exception for all other Clients.

    If you want to accept other Clients,
    the lines below have to be commented out ... }

Your Gateway log shows that the Gateway has received 48 bytes of concent in the login request, which probably means that the Gate Client has sent 44 bytes of UserInfo and/or UserAuth. But since "UserInfo" and "UserAuth" parameters are both empty in the BeforeUserLogin event on the Gateway, my guess is that the GateClient is using the wrong encryption key to encrypt that data before sending.

There are 2 parts of the encryption code used here.

First is the GatePrimaryKey, which has to be 100% identical on the Gateway and GateClient components.
The second part is calculated by the Gateway based on the users IP Address and the Forwarded-for HTTP header value.

If you are 100% sure that the GatePrimaryKey value set on the GateClient is identical to the one used by your Gateway,
then you should check the IP address generated by the Gateway for that Client.
To do this, enter your Gateways address with "/$p" in a Web Browser on the Client PC, like this:
http://<your_gateway_address>/$p

Try it several times, using at least 2 different Browser windows to see if the Address chaanges and
check if it is the same address logged by the Gateway inside the GateBeforeUserLogin event for that Client.

Best Regards,
Danijel Tkalcec

I've tested it and the GatePrimaryKey is obviously same :)
However I've tested also the Client IP with /$p pararm into the browser and also the IP address is same :(

I think, I have found the problem.

Are you using the TRtcGateway component with a TRtcHttpServer component?

If you are, can you check the MultiThreaded property on the TRtcHttpServer component?

Best Regards,
Danijel Tkalcec

Yes it's TRUE, this is my HTTPServer

I've check some logs and I see the "UserAuth" and "UserInfo" is not always empty, sometime are correct...but it gone offline and notify GateUserNotReady.

There was a bug in RTC SDK v7.32 and v7.33 which made the TRtcGateway component unusable with a single-threaded Server.
Please, download the latest update (RTC SDK v7.34 - just released) and let me know if it solves your problem as well.

Also, to make sure that Anti-Virus software isn't causing your problems, unless you have the option to entirely disable the Anti-Virus software and see if that fixes your issue, you should set StreamBlockSizeOut:=6 and StreamBlockSizeIn:=6 on the GateCli:TRtcHttpGateClient component to at least avoid problems caused by the Anti-Virus software buffering requests and/or responses for too long.

Best Regards,
Danijel Tkalcec

Sorry but nothing to do.
It always disconnect after about 80seconds.

Than you for your feedback.

I've found and fixed more bugs related to the way StreamBlockSizeIn and StreamBlockSizeOut properties were implemented, which resulted in premature disconnects and/or flooding in my tests. Please, download RTC SDK v7.35 (released now) and let me know if this fixes your problems.

Make sure to set StreamBlockSizeIn:=1 and StreamBlockSizeOut:=1 on the TRtcHttpGateClient component used by the problematic Gate Client, to avoid issues with Anti-Virus software buffering requests/responses or closing an idle connection after 30 seconds.

Should you still experience disconnect issues when sending large amounts of data, use the new functions "SetupLowestReceivingSpeed" and "SetupLowestSendingSpeed" on the Gateway and on Gate Clients to configure the lowest speed available to each connection. This tells the RTC SDK to use smaller input and/or output buffers to minimize problems of dropped TCP/IP packets, which can also results in disconnects. Receiving and Sending speed is configured per connection and should be done on both sides (Gateway and Gate Client).

Check the "Updates_Latest.txt" file for more info.

Best Regards,
Danijel Tkalcec

For now I've recompiled only the server with 735 and now that client work perfect after 10minutes...
I've found now other 3 clients that sometimes gone down.
Tomorrow I recompile and release the client and test it.
I will keep the situation under control and let you know if there was any news.

Thank you so much.

Regards

Thanks again for your feedback.

I've been running a number of tests inside LAN and over the Internet in the last 24 hours and did not experience any more unexplained disconnects with Gateway and Gate Client components after fixing StreamBlockSizeIn and StreamBlockSizeOut properties, but ... if you do find any more issues, please let me know.

PS. I've released RTC SDK 7.36 now as the next stable quarterly update. Compared to v7.35 which you are using now, there were only a few minor changes, but I would still recommend updating - just to stay on the safe side.

Best Regards,
Danijel Tkalcec

Now in Debug when I start the Gate I got an exception 'NOT Found @205(InternalResetOutput)'...what is this?

That is normal and expected.

It means that you have restarted the Gateway without closing Gate Clients, which is fine, but unless the Gateway was unavailable for several minutes and Gate Clients have logged out,  these Gate Clients will try to relogin with their old UserID, which can not work for obviois reasons.

The Gateway will then raise an internal exception and respond to the Gate Client that the UserID was not found, after which the Gate Client will log out and log in with a request to receive a new UserID.

By the way ...

If you pause the Gateway process while debugging for several seconds, Gate Clients will start closing their connections because of a missing PING from the Gateway. If you need to debug your own Gateway code, make sure to add Gateway exceptions to the ignore list, so they do not pause the Gateway. Also make sure to be quick when the process is paused inside your code, or take into account that Gate Clients will start logging out while you are looking at the code.

To debug the Gateway code or your own code used inside the Gateway, it is better to log what the gateway is doing instrad of using the debugger. Only use the debugger when you have identified a problem and know exactly where you have to place a breakpoint.

Best Regards,
Danijel Tkalcec

Hi Danijel,

I'm having the same type of issue when trying to connect to a gateway via a hardware firewall, even though this firewall is currently set to allow all traffic. The client connects to the HttpServer without problems but the UserAuth and UserInfo are not received on the gateway. When I test without the firewall everything works fine...

- I have installed version 7.3.6
- The StreamBlockSizeIn/Out are set to 1
- The GatePrimaryKeys match
- The HttpServer is multithreaded
- All Timeout values on the server are set to -1

Do you have any idea what might be causing this?

Best regards,
Pieter

Compile your Gateway Project with RTC_DEBUG and DEBUG_RTCGATE compiler defines declared in Project options, make sure there are no files in the LOG sub-folder where you place the Gateway exe before starting the Gateway, leave the Gateway running until that problematic Client tries to log in, then stop the Gateway and wait for all connections to close, then close the Gateway process.

There should now be a LOG folder with several files with the log extension. Please, send me all the files from the LOG folder (ZIP compressed) by EMail to:
(http://www.realthinclient.com/supportmail.gif)

Best Regards,
Danijel Tkalcec

Hi Danijel,

I have added the compiler defines to both the server and the client project options and performed a rebuild.
When I run them they produce only one log file each: Server.exe.DEBUG.log and Client.exe.DEBUG.log.

I wonder if this is correct since you said that several log files should be produced.
In the log files I don't see any specific reference to gateway communication.

Just so you know, I'm compiling with Delphi 10.1 Berlin 64 bit.

Best regards,
Pieter

My mistake. The 2nd compiler define for the Gateway is RTCGATE_DEBUG and not DEBUG_RTCGATE.

I also need to know the full URI and HTTP headers received on the Gateway from that problematic Client. See my replies to ISIGest (above) for details.

Best Regards,
Danijel Tkalcec

Thank you. I will send you the debug logs and header information.

Best regards,
Pieter

Hi Danijel,

I was just wondering, did you receive the log files in good order?

Thank you and best regards,
Pieter

We have been moving to a new appartment since last thursday and I did not have time to check my E-Mails yet. I will check later today or tomorrow and get back to you then.

Best Regards,
Danijel Tkalcec

Thanks again for reporting this issue and for sending me the LOG files.

After spending the last 4 hours staring at the log files you've sent me and checking the Gateway Login procedure over-and-over-again, wondering how that could be possible, it finally hit me. I forgot to check "if Sender.Request.Complete" in the Login data providers OnDataReceived event before reading the request content body and processing user login.

Everything was working fine if "GateUserInfo" and "GateUserAuth" properties were empty on the Gate Client, or if the complete Login request was received before the 1st call to Login data provider's OnDataReceived event, but ... if the request was received in more than one TCP/IP packet and the OnDataReceived event was called more than once for the same Login request, the Login procedure would be executed more than once for the same Client, creating a new User ID with each call and triggering the BeforeUserLogin event with a new UserID and empty "UserLogin" and "UserInfo" parameters for each call.

Anyway ... this bug should be fixed in the latest RTC SDK update (v7.38 - just released).
Please download it, try it out and let me know if everything works now.

PS. You only need to rebuild and restart the Gateway/Server.

Best Regards,
Danijel Tkalcec

Hi Danijel,

Thank you very much for the update! I will run a test tomorrow and let you know how it went.

Best regards,
Pieter

Ok. I'm looking forward to your feedback.

Best Regards,
Danijel Tkalcec

Hi Danijel,

I have installed version 7.38 and have rebuilt the server.
I then started the server from Delphi and stopped it after a few seconds.

When the server is stopped it gives this error:
First chance exception at $000000000088B91B. Exception class $C0000005 with message 'c0000005 ACCESS_VIOLATION'.

The execution halts at FCS.Acquire:
procedure TRtcBaseSockServerProvider.Enter;
  begin
  FCS.Acquire;
  end;

The resulting ERROR.log file contains one line:
2016-09-08 13:36:38.604; TRtcSocketClientThread.Destroy Par.RemoveThread Exception! EAccessViolation: Access violation at address 000000000088B91B in module 'Lobby.exe'. Read of address 0000000000000000

This did not happen when I was using versions up to 7.36.
I have sent you the log files in case you want to review them.

Thank you and best regards,
Pieter

Did you stop the Server listener and disconnect the Client before shutting down the Application, as explained in this FAQ topic (https://rtcforum.teppi.net/index.php?topic=95.0)?

If you are using the TRtcHttpGateway component, set "Active:=False" to close the integrated Server listener.
If you are using the TRtcHttpGateClient component, set "AutoLogin:=False" and "Active:=False" to disconnect Clients connections to the Gateway Server.

Best Regards,
Danijel Tkalcec

Hi Danijel,

I just tested logging in with the client and it works great.
I will verify those few things regarding the error.

Thank you and best regards,
Pieter

Hi Danijel,

To stop the server I am using this code:

Server: TRtcHttpServer
Gate: TRtcGateway
GateSrv: TRtcHttpGateClient

procedure TfrmMain.btnStopSrvClick(Sender: TObject);
begin
   if Server.StopListenNow() = wait_OK then begin
      ... some database stuff...
   end;
end;

procedure TfrmMain.ServerListenStop(Sender: TRtcConnection);
begin
   if not Sender.inMainThread then
      Sender.Sync(ServerListenStop)
   else begin
      Gate.RemoveGroup(SrvID, LGroupID, False, False);
      GateSrv.Active := False;
   end;
end;

GateSrv.AutoLogin:=False (this never changes).

I never saw that error before and the code for stopping the server has not been changed.

Best regards,
Pieter

By the way, this happens without a client logging in.
I am only starting the server and then stopping it again.

I've found a few things that I've changed in v7.37 while working on TRtcMessageClient and TRtcMessageServer components, which could be causing AVs on the TRtcHttpServer component shutdown. Please, download RTC SDK v7.39 (just released) and let me know how it goes.

Best Regards,
Danijel Tkalcec

I have installed version 7.40 and rebuilt the server but I'm sorry to say that the error has persisted.
The situation is still the same as mentioned in THIS topic (https://rtcforum.teppi.net/index.php?topic=1366.msg4252#msg4252.).

Best regards,
Pieter

Please, download RTC SDK v7.41 (just released) and let me know if this works for you.

Best Regards,
Danijel Tkalcec

Thank you for fixing this Danijel!
It works perfectly now  :)

Best regards,
Pieter

Thanks again for your feedback.

Best Regards,
Danijel Tkalcec