Firewall thinks RTC data stream is a flood attack

[Paul Breneman]

A client is using RTC (Pro) along with the CopyCat database replication components ( www.dbReplication.com ).  The application has been working well but when it was recently installed on a new server the firewall would prevent operation.  They've added an exception to the firewall to temporarily allow things to work but we are looking for a better solution.

The firewall is Microsoft's Forefront Threat Management Gateway (commonly referred to as "TMG") in Standard version, standalone (not in an array).

Is there any RTC setting we could change to make the firewall happy?

[D.Tkalcec (RTC)]

As far as I know, it is expected of you to configure the Firewall on the Server PC to allow incoming connections to Ports where you are running trusted (Server) applications. My guess is that the old Server PC has already been properly configured and installing the new Server PC simply required the same attention - which you've done by explicitly allowing communication on the Port where your Server app is listening.

In other words, at least as far as I know, except for configuring the Firewall on the Server PC to explicitly allow incoming connections to your Server Application, there is nothing you can do to make a Firewall "happy". The RTC SDK uses HTTP/S for all communication which makes it proxy-friendly, but it won't allow you to silently go through firewalls or proxy servers if they aren't configured to allow communication through the Port you have chosen.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

By the way ...

Do you know if CopyCat clients are continuously polling data from the Server?

Also, if data is being continuously polled by Clients, do you know if the Server is using RTC Delayed Calls to reduce bandwidth usage, or if requests are being fired from Clients one after another at short time intervals which could mean that Clients are flooding the Server and the Firewall is complaining about that?

Best Regards,
Danijel Tkalcec

[Paul Breneman]

Thanks for your thoughts Danijel.

This is the first use of the RTC/CopyCat replication program at this company so they don't have any previous firewalls configured already for the replication program.

They did open up the port as things worked for about one minute before the firewall shut it down.

This problem occurred with only a single test program (only a single client).  The replication program doesn't constantly poll the server but it was sending a lot of data to the server through a single connection.  And the firewall interpreted all this data to be a flood attack and shut it down after about one minute.

[jonneve]

Hello Paul and Danijel,

The CopyCat / RTC transport certainly does make a multitude of RTC requests in a rather short period of time, as there is a request for starting the transaction, another for preparing each query, another for executing each query, another for committing, etc. And since there are a good number of queries that need to be executed, there are many -- small -- requests, so perhaps that could be an explanation.

Paul, if I understand correctly, the firewall is letting the connection through, but only for 1 minute, and then it cuts it off because it thinks it's a flood attack, is that right? So it's not just a matter of opening the port... In that case, what did they deactivate in order to make it work?

So, if I understand correctly, maybe the RTC delayed calls could be the answer. But what does it do exactly? How can you group up requests, since we need the responses to each request in order to make the following ones (CopyCat replicates with a very synchronous logic, not asynchronously with multiple threads)?

Failing that, it looks to me like a Firewall configuration problem : isn't there some way to tell the server to allow requests coming from that client's IP address? Or set up a VPN? Because CopyCat does inevitably perform a great number of HTTP requests, and I don't see what could slow them down other than deliberately reducing the bandwidth...

Regards,
Jonathan Neve.

[D.Tkalcec (RTC)]

Hi Jonathan and thanks for joining the conversation,

1) RTC Delayed Calls can be used to reduce bandwidth in case the Client needs to continually poll the Server for data, but the Server does not always have something to return (for example, another Client needs to send the data first). In these cases, the Server could prepare a TRtcDelayedCall object to put the call to "sleep" until data arrives. But ... if your requests usually have a response ready immediately, there is no point in using RTC Delayed Calls.

2) You can pack any number of remote function calls into a single request and also use results from Server-side calls as parameters for other Server-side calls (nested remote function calls). This allows you to use the result prepared by FunctionA as a parameter in FunctionB, for example. There is no limit to how complex the call can be or how deeply you nest your calls, but you need to write your Server-side functions so that they can work together in this way. But ... if your application or components require serial communication flow in which the Client needs every response and where you can't prepare data for a new remote function call before you have the result of the last remote call, I'm afraid there is no simple way to reduce the number or request/response loops.

Best Regards,
Danijel Tkalcec

[D.Tkalcec (RTC)]

Hi Paul,

from your reply, it sounds to me like Microsoft wrote that Firewall to ring alarm bells when there is high traffic on specific ports, even if they are open. If that is the case and there is a lot of data which needs to go through the connection, I am afraid there is nothing anyone can do to make the firewall "happy". The only solution is to tell the Firewall that it is OK for the Client to send so much data. If the Firewall can't live with that, the only "solution" is to replace the Firewall.

Best Regards,
Danijel Tkalcec

[Paul Breneman]

Thanks Danijel and Jonathan for your thoughts.  You've helped to clarify what the problem most likely is:

http://technet.microsoft.com/en-us/library/dd441028.aspx
Setting flood mitigation connection limits
Maximum HTTP requests per minute per IP address
600 (custom: 6,000)

Hopefully they can increase this value to more than 600 for all outside IP addresses.  Since there will be multiple clients it won't work if they can only make an exception for a single IP address.

Thanks again for your help!

[PhilH]

Hello,

I am the one working with the actual replication attempts in question.

Here is our experience. The CopyCat replication works well when the client (remote) computer's IP address is specified as an exception to the firewall rule.

We then removed this IP address from the exception list and set the
“Maximum HTTP requests per minute per IP address” to 6000.

At this setting, the replication process was cut off after between 45 and 60 seconds (in 2 different tests)

Following are various settings in the firewall. Might we need to also adjust any of these?

1.       Maximum TCP connect requests per minute per IP address
2.       Maximum concurrent TCP connections per IP address
3.       Maximum half-open TCP connections
4.       Maximum HTTP requests per minute per IP address
5.       Maximum new non-TCP sessions per second per rule
6.       Maximum concurrent UDP sessions per IP address
7.       Specify how many denied packets trigger an alert

Thanks for your help,
Phil Horst

[D.Tkalcec (RTC)]

If increasing the maximum request count per minute to 6000 did not make the firewall happy, judging by the CopyCat explanation Jonathan wrote, I'm pretty confident that your clients will get into a situation where they will need to execute more than 100 requests per second and that is most likely what triggers the firewalls alarms.

I don't know that firewall, but if this is the case, then none of the other available options will make things any better and your best option is to simply add all clients IP addresses to your Firewalls exclusion list on the Server.

Best Regards,
Danijel Tkalcec