Features of repelling DDoS attacks and the history of an attack on one large bank

[Man]

Previously, DDoS attacks could be successfully repelled on the side of the attacked company's data center. Quick, smart actions by the admin and good filtering hardware guaranteed sufficient protection from attacks. Today, botnet services are rapidly becoming cheaper, and DDoS is becoming available almost in small businesses.

About half of the attacks are directed at online stores or commercial sites of companies in the spirit of "taking down a competitor", almost always attack media sites, especially after "hot" publications, much more often than it seems, they hit government services. In Russia, the main targets are banks, retail, media and tender platforms. One of the large Russian banks, for example, periodically blocks traffic from China - attacks from there come with enviable regularity, one of the last was more than 100 Gbps.

Accordingly, when the attack exceeds, say, 10 Gbps, repelling it on your side becomes problematic due to the banal clogging of the channel. This is exactly the moment when you need to switch to the data cleaning center so that all the "bad" traffic is filtered out somewhere near the main channels, and does not go to you. Now I will tell you how this works for one of our vendors of security tools - Arbor, monitoring about 90 Tbit/sec (45% of the world's Internet traffic).

Attack scenario​

First, the attacker selects targets within the infrastructure. Most of the "stupid" attacks are on HTTP, there are many attacks on DNS, but the main "smart" attacks are usually aimed at previously explored nodes within the target's infrastructure. Often, DDoS aims not only and not so much to deny services, but to be able to carry a more serious threat "under the general noise" through the DMZ. This is often achieved by "overloading" perimeter protection systems, such as firewalls, IPS/IDS and the like, based on session tracking (stateful inspection). Therefore, colleagues believe that if a device has a session table, it should be considered part of the infrastructure that needs protection.

Main attack points: The attack reflection scheme is implemented as follows:[/B]

1. On the side of the protected company there is a device that "closes" the network. As soon as an attack begins, the device must recognize it as an anomaly using one of the mechanics, for example, built-in countermeasures, abnormal behavior of the traffic source, or compliance with a known attack profile (signature from the database).
2. If the device copes with the attack, the work continues in a relatively normal mode: legitimate traffic is allowed through, illegitimate traffic is cut. There are several "alarm levels" that differ in the complexity of the attack and the possible loss of legitimate traffic.
3. If the communication channel starts to get clogged, then the traffic is automatically redirected using the standard Cloud Signaling protocol. First, everything comes to the site of a large provider (this could be Rostelecom, Orange, Transtelecom, Akado), where the data is cleaned by Peakflow SP equipment. The already cleaned traffic goes to the end client. At the same time, the client has an understanding of everything that is happening - he can quickly go to the personal account of the telecom operator and see the current cleaning status, what countermeasures are working, what is the effectiveness of attack suppression, and so on. The client device also shows the effectiveness of the cleaning currently taking place and a list of blocked hosts. If desired, you can easily dump traffic in pcap format from both devices for subsequent "debriefing".

A reality that is not usually talked about​

1. Loss of legitimate traffic.
Combating a DDOS attack means dropping illegitimate packets and letting legitimate traffic through, and there is sometimes a very fine line between them. Many vendors like to write in their brochures that these losses are close to zero. In my experience, they can easily reach 2% — this means that, theoretically, the same director who went on a business trip will not be able to get into the corporate network from a hotel or a conference. It is very important here that the system supports the ability to resolve specific connections or protocols "on the fly". For a number of vendors, the settings are practically hardcoded into the firmware of the hardware from the moment the attack begins, and changing them is extremely problematic. Arbor has a completely different approach. Firstly, there are three "alarm modes" for managing the false-positive level — from "cut everything that is definitely not ours" to "there is an opportunity to dig into more detail". Secondly, there is a convenient search for blocked hosts and the ability to unblock traffic for a specific host, protocol or country in one click. It should be noted that the reality of the presence of false positives in the fight against DDoS is recognized by all significant market players. Alexander Lyamin (Qrator) once noted: "anyone who says that he has zero false positives is a charlatan".

2. The ability to use a clogged communication channel to signal an attack.
How can a client request an attack from a provider if the channel between them is clogged due to DDoS? Strictly speaking, even with close to 100% utilization of the communication channel in the direction from the provider to the client, there is a very high chance that the request to clear data will reach the provider. For this, Cloud Signaling works on top of UDP, and in addition, the protocol does not require a response from the provider. Thus, it is enough to have at least a little capacity in the direction from the client to the operator. To be on the safe side, it is recommended to organize a separate channel for exchanging Cloud Signaling messages. However, a backup channel is usually created, plus the alarm occurs at a threshold of about 70-80% of the channel.

3. Time to switch to the data cleaning center.
The delay in redirecting traffic to the cleaning center of the DDoS protection service provider can take units or even tens of minutes. This is mainly due to the redirection mechanism - based on DNS records or BGP announcements, as well as the fact whether the decision to redirect is made manually or automatically. In any case, if the attack is suppressed in the "cloud", then it will not be possible to avoid a delay. At a minimum, it is several minutes. Therefore, we adhere to the concept of multi-level protection, when large attacks on communication channels are suppressed by the operator, and slow, barely noticeable application-level attacks are suppressed by equipment installed at the customer's site.

4. Use of SSL certificates.
Analyzing SSL/TLS traffic for application-level attacks is quite problematic — you need to decrypt every packet. Here you face a difficult dilemma: either share your certificates with the service provider, or accept the risk that an attack at the HTTPS level may be missed. Vendors try to find solutions without opening packets (which does not always work well), or use a special SSL/TLS traffic decryption module built into the client device:

In this case, your certificates are loaded into a device under your control, and the delays for additional packet processing are milliseconds.

5. Manual signature generation.
Most signatures are generated by DDoS protection solution vendors. However, situations periodically arise when resources are exposed to new types of attacks.

Depending on the vendor, there are two possible scenarios for action in such a situation: either request a new signature from the manufacturer, or create a signature yourself. In the first case, you will most likely have to pay extra for the service, and also remain under attack for a considerable time waiting for the problem to be solved.

In the second case, the principle of "saving the drowning is the drowning man's own doing", or their partners, works. Here, the functionality of the equipment becomes critically important, allowing you to quickly identify, intercept and analyze malicious traffic. And the ability to automatically generate a signature based on the information received becomes a salvation in a critical situation. For example, the ability to go to packet capture, select the desired bit pattern, and in a couple of clicks create a signature that blocks such a sequence.

Advantage of scale​

Arbor has a very interesting "feature" that allows them to use their market share very effectively. The fact is that their equipment is used by all TIER-I telecom operators and providers and by most TIER-II operators.

Arbor's position in the market of DDoS protection equipment in the Carrier, Enterprise, Mobile segments is the first (Infonetics Research for December 2013) - 65% of the entire market.

About 90 Tbps of information passes through the ATLAS system. As soon as signs of an attack appear somewhere, the devices begin to transmit data about what is happening via their own communication network, and the information quickly spreads throughout the world. For example, if a small provider is "taken down", its hardware signals via a special protocol to a higher level. If the higher operator has an agreement with the lower one, then the signature is accepted and distributed to all subnets of the large provider.

ATLAS honeypots are located on the main nodes of the global Internet to detect and classify attacks, botnet activity and various malicious software. The information is sent to the ATLAS data center, where it is combined with data received from the Arbor Peakflow installation and other data. The system automatically analyzes hundreds of thousands of code elements, and allows the team of engineers to promptly update signatures for the company's clients around the world.

Connection Features​

It is worth saying a few more words about the hardware that is installed directly in your data center or on the site. There are many realities associated with it that are not always talked about.

1. Setup and configuration.
As a rule, the device needs time to profile traffic and evaluate network behavior. But some devices are supplied in "combat" mode to work from the first second after connection - there are already ready-made templates, plus the device receives data from the "cloud" of its vendor. On the one hand, this is good in terms of repelling an ongoing attack, on the other - if you are used to fine-tuning everything in your infrastructure, you will have to partially rely on the vendor's experience here.

2. The security hardware itself can become a point of failure.
Therefore, firstly, at serious sites it is duplicated or several devices are assembled into a cluster (accordingly, control devices are needed). And secondly, such devices have hardware bypasses that allow you to switch interfaces at the physical level directly to traffic in case of various emergency situations - power outages, software failures, etc.

3. Protective hardware can also become a target for an attack.
Especially if it is a device with a state table (session table), for example, it combines the functionality of protection against DDOS, IPS, a firewall, etc.). Each time a new session is opened on such devices, the device allocates memory to track the session, fills the log, and so on - and the smarter the device, the more work is done. Many sessions - high utilization of CPU and memory. Therefore, when choosing a solution, it is worth paying attention to this aspect.

4. Usually there is quite a lot of junk traffic on the network, and in the case of large organizations, there are also regular peaks of activity that can pass for a "stupid" DDoS.
It is clear that everything will be profiled by geography, by the time of use of services, and so on, but at the first stage it is important to understand that connecting the device to the network will not kill the services. For this, a mirror connection mode is used: the device is put into the network on bypass and receives mirror traffic, showing what it would reject and what it would pass. This allows you to make an assessment before problems arise. I know customers where DDoS protection has been in this mode for quite a long time. Arbor believes that building a baseline is only one of the methods for combating DDoS, but it is much better to use specialized countermeasures. Colleagues say the same about signatures - no matter how large the sensor network is and how much traffic is analyzed in it, you cannot rely only on signatures. Complex threats require a combination of different protection mechanisms.

5. At the level of traffic exchange between providers, a rather unpleasant situation sometimes occurs when a subnet transmits an attack signature, and the upstream provider says: "Yeah, interesting, but we won't clean it up."
The principle is very simple - while the channel copes, all DDoS traffic is billed. It is not always interesting for the upstream provider to spend money and clean it up when it can simply be shipped downstream, and at the expense of the victim. Such situations are unpleasant for service providers, but do not affect users of DDoS protection services in any way, since the operators fulfill their obligations in full.

Reports​

One of the important things is to quickly understand what is happening. Let's look at the reports using two attacks on a large Russian bank as an example. Feel how the admins have added gray hair in the comic below.

Attack capacity of about 50 Gbit/sec​

This attack started with DNS amplification. Traffic recorded on the service provider's Arbor Peakflow TMS: up to 7.15 Gbps, 1.1 Mpackets/s. Taking into account filtering of most of the attack on FlowSpec, the total attack traffic is estimated at 50 Gbps. The attack was continued by HTTP flood.

Having recorded an abnormal increase in traffic, the client device automatically requested help from the service provider in suppressing the attack.

The protection system installed at the operator transmitted information about the progress of the attack suppression to Pravail APS in real time.

When the DNS amplification attack subsided, a large amount of HTTP flood appeared.

In addition to HTTP flood, TCP SYN flood was also present.

As a result, most of the DNS amplification attack was successfully suppressed using Flowspec, the rest was suppressed by the protection system, and the HTTP and TCP SYN flood was dropped on Pravail APS.

Attack capacity of about 125 Gbps​

A few minutes at the beginning — HTTP flood. A surge in the number of foreign hosts is visible (red graph). About 1000 hosts per domain of "Large Russian Bank". Main countries: USA (672), Germany (141), UK (82), Italy (30), Netherlands (24), France (14).

The next surprise is NTP amplification — up to 125 Gbps.

After a failed NTP amplification — SYN flood with spoofed IPs — up to 300 Mbps

Source