BLACK HAT conference. Survival lessons in a 300 Gbps DDOS attack. Part 1
My name is Matthew Prins, I am one of the co-founders and CEO of CloudFlare. Do not think that I will be as interesting as the director of the NSA, but at least I will try to be less controversial. In fact, I am a private law professor, so I read the Constitution and can start a long discussion on this topic. Someday we will discuss this in great detail, hopefully I can tell you the story of the lawsuits that we are currently conducting against the United States government to dispute certain things that are happening, since this is perfect rubbish.
I will tell you a less interesting, but dramatic story that took place from March 18 to March 25, when one of our clients underwent the strongest DDoS attack. I will not talk much about CloudFlare, if you want to learn about our service, just go to the site. But when our client came under attack, it was an event from the line of those journalists like to write about, and this attack was covered by the New York Times for 4 days in a row. It was a really powerful attack, so I want to talk about it.
So, there is a non-profit organization called Spamhaus, which tracks spammers and their activities. I know these guys for a long time, since 2006-2007, when an alleged spammer from Illinois sued them. Then the founder of this organization, Steve Linford, called me and asked if I could speak for them as a free attorney. I replied that I used to be a private lawyer, which means that I taught students in the class, but I never spoke in court. I very quickly found a real lawyer for them, who soon helped them to "resolve" the situation, so I know that Spamhaus is a very decent organization, they act quite openly.
Our clients are many organizations, but they don’t like to talk openly about their problems, but the guys from Spamhaus are not so closed, and when they were attacked, they called us and said: “Hey, we can tell you our story”! So this is a story about the attack on Spamhaus and about what happened behind the scenes when they were "brought down".
I broke this story into 3 parts. The first is the nature of the attack, what it is and who is behind it. If you want to know more about this, then come to DefCon, where I am going to speak next week.
The second part is what needs to be done to stop the attack, how to create a network that is really capable of it. This is not a story about great intelligence or unbelievable software, it’s really the story of designing a smart network.
The last part is what you can do on your own networks to stop this type of attack. I promise that the outcome of the speech will not be campaigning to use the CloudFlare service as the most protected, instead I will tell you about the advanced things, having done that, you will become less vulnerable to such DDoS attacks.
So, on Monday, March 18, we received a call from one of the Spamhaus volunteers, who informed us about the attack. This is how their website looked like at that time - the attackers brought it to the offline state, effectively disconnecting the Internet.
Later, the media wrote that the attack was made by someone Fenn, who lives in a bunker somewhere in the Netherlands. This guy was not the real initiator of this action, he was probably just the one who was willing to give an interview to the New York Times. Rather, the real mastermind of the attack is the 15-year-old teenager from London, who is now under arrest, but the press does not make a fuss about it.
In an article dated 23.03.2013, the New York Times newspaper wrote that the attack was carried out by the Danish company Cyberbunker in response to the fact that Spamhaus was blacklisted for spreading spam. The headquarters of Cyberbunker was indeed located in an abandoned NATO bunker (note the translator).
Most likely, the motive of the action was the fact that Spamhaus declared someone a spammer. This organization does a lot of good things, but it happens that they pinch someone’s tail, and in this case, the offended side responds, in this case it organizes the attack. Spamhaus is technically savvy enough to repel many different types of attacks, but in this case, a massive group attack hit their DNS infrastructure and simply knocked the site out of the network. It could happen to any organization, and they wanted to stop it. They knew that we were helping to prevent such attacks, so they turned to us.
So, from March 18 to March 21, what we call the "tantalizing attacks" took place. Such attacks are our bread, weekly we have several such attacks. So that you have an idea of their scale, I will say that last week we recorded 162 attacks on our clients, and we rather successfully stopped them. These DoS attacks occur at an intensity of 10 to 80 Gigabits per second. For some organizations, such as financial ones, attacks that occur with such intensity can be dangerous. So, last year, some financial institutions suffered from DoS attacks of 60-80 Gbit / s, but I repeat - these are fairly common attacks, familiar to us, and there is nothing in them that deserves a mention at the BlackHat conference.
Another thing to note is that the DoS attacks are not all the same, the size of the attack, that is, the number of packets attacking the site, is important. What we will talk about today refers to attacks of level 3, that is, attacks against the network layer of the OSI model. This is pure volume attack. But it is not the worst, much worse than the attack of the 7th, application level, which is not so voluminous, but it creates much more problems for specific applications. But today I am not going to discuss attacks that require special software.
The same applies to Level 4 DoS attacks, when an attacker uses a small amount of Syn-packets, causing significant damage to the victim, because his operating system is not able to cope with these requests.
In our case, the attack is based solely on a large volume, which tries to destroy you at level 3, knock you out of the network and overload the port with traffic. On Wednesday, March 20, one of the ports of our network, or rather, a collection of ports, on one of which the Spamhaus site sat, was attacked.
This slide shows the proxy cache, the blue line is traffic coming from our network, and the green area is incoming traffic. The high peak at the end of the chart is an attack that fell entirely on Spamhaus and brought down our network. Our network can easily withstand attacks of 75 Gbit / s intensity, because it is designed to “digest” such a volume of traffic.
However, during the attack on March 20, the “ceiling” of DoS intensity was reached - 100 Gbit / s. This looked like an exceptional case, because if you buy or create a basic router, then contact someone like Cisco or Juniper, and the coolest and largest “box” you can buy from them has a port with a capacity of 100 Gig. Theoretically, you can use several 100-gigabit ports and pass them through one interface, but the ceiling of DoS attacks usually does not exceed the specified amount.
Soon after, Tom phoned me, one of our network engineers, and said that the intensity of the attack had increased. She can push herself beyond the boundary values, so I asked how much she grew up, thinking that we are talking about a slight excess. The next slide shows the intensity of the new attacks that took place on March 24-25.
The graph shows the universal coordinated time of UTC, I live in San Francisco, we had another 23 at this time, so you can imagine what my dinner was on March 23. They called me and said that the attack exceeded the ceiling of 100 Gigabits, but we did not think that very soon we would be able to see something like that! Suddenly, 4 calls were received from the provider of one of our incoming networks and network engineers that we now have traffic with an intensity of more than 300 Gbit / s, which reaches the limit of our network's capabilities.
The graph shows that throughout March 24-25, attack rates dropped to 90 Gbit / s and increased to 300 Gbit / s, causing a denial of service to the incoming traffic of one of our providers, which has completely exhausted network bandwidth.
The question is how? How did a hacker manage to do this if it is known that the ceiling of the equipment is 100 gigabits? How did this guy from London, this 15-year-old, manage to organize such an unprecedentedly intense attack?
Many people, including the media, expressed the wrong opinion that a botnet is needed for such an attack. But in this particular case, there was no real traffic from the botnet network. Other people said that it was Anonymous, because there are many people in this community who are coordinated and can work together to launch attacks of a certain kind. No, it is not. The attack was carried out by 3, possibly 4 people, one of whom was a real technical inspiration, but very limited in terms of resources and performers.
Many people felt that outstanding technical skills were required to carry out such an attack. I say that I saw the code that provoked this attack, which is about 50 lines in C, not including the headers that are usually included in such an exploit, so this is quite a bit of information.
In just a second, I will show you how you can launch such an attack. What is really needed for this is the list of open clients DNS, or DNS resolvers, later I will tell you what it is. The second thing you need is a few servers, or just low-powered computers running on networks that allow you to replace the original IP.
These are the basic components for attacking, and if you have these two things, as it turns out, even in very small quantities, then you can launch a very, very large attack. So far, nothing that weakens such attacks has changed, and this is a bit creepy.
Let's talk about open DNS resolvers. DNS is something like small blank pages for the Internet, it is a caching system that creates the ability to respond to requests that were sent to the server.
An improperly configured DNS resolver, also called an open DNS resolver, answers questions no matter who asks it. This “slutty” DNS resolver answers everyone, no matter who asks. It works without any restrictions. It has no restrictions on the number of requests, or on who to receive requests from.
You may ask: “What about Google? After all, he uses an open DNS resolver! ” However, organizations such as Google are doing a lot to limit the number of requests addressed to resolvers, in order to make it difficult to launch attacks, they limit the size of the answers that are provided on requests. Just last week, the guys from Bind announced a new extension that allows them to set a speed limit for requests, this is exactly what we lacked for a long time. Because what scares you most is that authoritative, open DNS servers are increasingly being used for attacks.
When carrying out any attack, the hacker seeks to strengthen it in any way possible, for example, using a botnet. If I have very few of my own resources, I want to use them to create additional resources. At the BlackHat conference, it was said that DDoS is a perfectly asymmetric attack, because one little person can create huge problems, and open DNS resolvers clearly demonstrate this.
This slide shows the key command to launch an attack through a Windows terminal, or console.
This command was launched at 2 am from an open DNS resolver 63.217.84.76, which worked on the PCCW network. If you send this request, the server will respond to you.
digANY is just a DNS query that initiates a search for any domain names NS, isc. org means that they wanted to show you every possible type of DNS record, so their DNS file is huge, followed by the resolver IP address and the flag edns = 0, which says: “Give me everything, including dns.sec and many others flag types, which you may not get by default. The notcp flag is an important element, because it orders the resolver not to use TCP for data exchange, but to send them all over UDP, and bufsize sets the buffer size to 4096, that is, to make the buffer as large as possible to send very “fat” packets.
So, you run this cute little query of 64 bytes in size and get back what is now running on the screen. That is, on a 64 byte request, you get a response of 3363 bytes in size.
So, if you manage to find an open DNS resolver, you send it 64 bytes and receive a response of more than 3300 bytes in size. That's what I mean when I say that you can use tiny resources to create a big problem. In this case, we get an attack gain of about x50.
However, the gain does not work by default, and if I send a small request to get a big answer, I just send a DDoS attack on myself. Therefore, the second mandatory component for such an attack is a network that allows the source IP address to be spoofed.
That is, I try to pretend that the DNS request was sent from an IP address from which it was not sent. I send a message to the DNS resolver, and since UDP is a “burn and forget” protocol without a handshake, after receiving the request, it simply sends the answer to someone it considers as the sender.
Well-established networks do not allow this, they do not allow packets to come from IP addresses that do not belong to this network — these are BCP38 recommendations. The majority of Internet providers follow these guidelines when creating networks, however there are networks that ignore this rule.
So, if we have two necessary ingredients, then since UDP does not use a handshake, you can easily change the IP address of the source of the request. For the attack on Spamhaus, the specific DNS shown on the slide was used, and the confidence that the connection is made via UDP, which does not require any handshake. Then, the responses of all of these open DNS resolvers located around the world are sent back to yours or any other target network.
The secondary effects of these DDoS attacks affect customer support, because hundreds of thousands of open DNS servers are involved in this attack. Their users track the path to an attacker address 190.93.243.93, from which tons of packets emanate from their infrastructure, and begins to flood you with angry messages like “you don’t know what you are doing, you don’t know how to run a normal network, your computers hacked, how dare you attack me! ”and the like. And you have to be very polite to explain to them that in fact they are attacking you, and they themselves suffer because they have not correctly configured their DNS resolver, the answers to the queries come back, and all this ultimately affects you. themselves.
Actually, this is an opportunity to teach people to “clean” their networks, and this is the positive aspect of such attacks, because people do not realize that this happens all the time, due to the incorrect configuration of open DNS servers, only on a small scale. And only when the intensity of attacks increases significantly, they begin to pay attention to it.
I think all of you are familiar with such old types of attacks, like Smurf, where you use the transmission of ICMP broadcast requests to routers via routers, and use source spoofing, causing computers to respond to the victim of your spoof by hammering communication channels.
On the positive side, the routing industry has learned to deal very effectively with this attack. They are still found, but it is quite difficult to launch the Smurf attack. However, in general, attacks by reflecting DNS responses are very easy to organize.
How common are the components of such attacks? The next slide shows the Open Resolver Project, this is quite a revolutionary product. Many of those present here argued for a long time whether to publish on the network a list of all open DNS resolvers running on the Internet. The fact is that this list can be used both by the “good guys” so that they can point clients to incorrectly configured networks, or by the “bad guys” who can use it to organize DoS attacks.
It is not by chance that this project was launched on Monday, which followed the weekend, during which we endured all this hell. Thus, the largest Internet providers of the first level were able to make sure that the “bad guys” have this list for a long time and therefore it is necessary to start cleaning it, getting rid of open servers. Thanks to this site, you can enter a range of your IP addresses and check whether there are open DNS resolvers on your network. Thus, if the PCCW did a similar check, they could clear the network before they had problems. We attracted the media to cover the role of this project, so that all providers know about this list, because the number of open DNS resolvers used on the Internet is 28 million. I would be happy to inform you that today this number is 30, 40 or 50 million, however, unfortunately, at the time of launch it was only 21 million, so we are clearly moving in the wrong direction.
The next thing I want to mention is the completely abnormal percentage of networks that allow spoofing. The slide shows the MIT project that allows you to track the spoofing networks. It was difficult to check them remotely, but tests show that almost 25% of the Internet allows spoofing.
In 2013, there can be no excuse for not using BCP 38 for border routers. We must not allow packets that are not born on your network and that come from unknown IP addresses to increase the load on the routers. Any router released after 2002 makes it very easy to enable such a function, so do not use it - just laziness!
The good news is that mostly large botnet networks do not support this kind of spoofing. Global networks are mostly closed for spoofing, so a catastrophic scenario is possible if hackers succeed in spoofing, for example, China Telecom's Chinese network. В этом случае можно было бы использовать ботнет для генерации трафика на DNS-серверы для осуществления потенциальной атаки огромного размера. Вероятно, присутствующим легко понять, как трудно избавиться от всех открытых резолверов. Но вы можете реализовать в собственных сетях возможность игнорировать чужие пакеты, чтобы не подвергнуться подобной атаке.
Давайте рассмотрим два конкретных инцидента, имевших место при атаке на Spamhaus. Первый был 24 марта и заключался в атаке интенсивность около 309 Гбит / с на протяжении 28 минут. Паника началась со звонка Тома, который сказал, что у нас атака более 300 Гбит / с. Я спросил, работает ли наша сеть, и Том сказал, что мы пока держимся, мы можем обеспечить Spamhous и других клиентов. И хотя наблюдаются небольшие потери пакетов и задержки в передаче трафика, но в целом сеть держится. Тогда я сказал, чего же вы так испугались? Они ответили – потому что атакующие используют всего лишь 0,1% открытых резолверов, которые существуют в мировом интернете! Но если они смогут использовать хотя бы 1%, у нас будет не 300-гигабитная, а 3-х терабитная атака. Если они будут использовать 10% открытых DNS-резолверов, мы получим 30-терабитную атаку. Я думаю, что сегодня мы могли бы выдержать до 3 терабит трафика, но в марте у нас не было такой возможности, мы не могли победить 30-ти терабитную атаку и я не знаю никого, кто смог бы её выдержать. Это было очень серьезным вызовом работоспособности сети.
Когда мы стали разбираться, откуда исходит атака и просмотрели кучу сетевых провайдеров первого уровня, то обнаружили всего 3 сети, которые позволяли спуфинг адресов, и возможно, их обслуживало от 5 до 7 скомпрометированных серверов. Один был в Сан-Хосе, другой находился в Миннесоте и третий в Германии. Было задействовано всего 3 сети, и эти скромные ресурсы привели к относительной большой атаке.
Следующий пик пришёлся на 25 марта, когда был зафиксирован трафик со скоростью 287 Гбит/с на протяжении 72 минут. Использовалось практически такое же количество открытых резолверов – 31154, и те же три спуфинговые сети. Ещё раз повторюсь – атакующий задействовал очень ограниченные ресурсы.
Отмечу, что у нас был всего 1 атакующий с одним ноутбуком, у нас есть скриншот, который я покажу позже. Он контролировал от 5 до 7 скомпрометированных серверов, 3 сети, позволяющие подмену оригинальных IP-адресов, которые генерировали трафик интенсивностью около 9 Гбит/с, направленный на 0,1% открытых DNS-резолверов. И всё это привело к возникновению 300 Гбит/с DDoS-трафика.
Многие люди считают, что это была самая большая атака в истории сетей, не могу сказать, правда ли это, но она действительно была самой большой из всего того, что мы видели.
Это история о великом организаторе, которому удалось проделать то, о чем раньше даже не догадывались. Мы все знали проблеме открытых резолверов, это не было каким-то умным новым взломом, порчей сетевых протоколов, атакой фрагментации пакетов и так далее, это выглядело так, как будто пещерный человек просто долбит огромной дубиной тебе по башке.
Повторю снова – то, что произошло, было невероятно эффективной атакой и на Земле очень мало сетей, которые способны ей противостоять. Опять же, мы очень гордимся тем фактом, что мы сделали тогда и что мы продолжаем делать каждый день, и тем, что продолжаем инвестировать в нашу сеть, чтобы сделать её больше.
Немного расскажу о том, что мы сделали для того, чтобы иметь возможность сопротивляться атаке, что мы предприняли, чтобы её остановить. Некоторые из наших мер применимы в общем случае, некоторые являются уникальными. На этом слайде вы видите, что мы предоставляем услуги по всему миру, имея 23 точки подключения к сервису CloudFlare, а к концу года их количество увеличится до 50. Так что на сегодня у нас имеется развитая сеть X-класса.
Мы изначально подумали о целесообразности нашей сети, но то, что было заложено в неё в начале, на сегодня превратилось в очень мощный инструмент. Мы используем протокол под названием Anycast для передачи пакетов методом BGP через множество поставщиков DNS, которые также используют Anycast. Фактически мы используем его для HTTP-трафика, как наиболее стабильного. Скажу, что это большая головная боль – обеспечить нормальную работу такой сети, как наша.
Если вы не знакомы с протоколом Anycast, то вот что он собой представляет. В обычной схеме сети имеется один источник и один пункт назначения, а в Anycast один источник связан с несколькими пунктами назначения, при этом для доставки пакетов адресату выбирается кратчайший путь. Таким образом, если у нас имеется 23 дата-центра по всему миру, то все они имеют один и тот же IP-адрес, например, адрес Spamhous. Поэтому, когда мы имеет дело с атакой интенсивностью 300 Гбит / с, она распределяется между всеми дата-центрами таким образом, что на каждый дата-центр приходится порядка 13 Гбит / с трафика.
Проблема состоит в том, что атакующий тоже может заняться математическими подсчётами и очень быстро перенаправить атаку прямо на край нашей сети. Это немного дурацкая затея, так как в результате атакующий может получить эффект, обратный усилению. То есть вместо того, чтобы усилить атаку увеличением её поверхности, он вынужден сократить эту поверхность и уменьшить интенсивность атаки.
29:00 мин
BLACK HAT conference. Survival lessons in a 300 Gbps DDOS attack. Part 2
Спасибо, что остаётесь с нами. Вам нравятся наши статьи? Хотите видеть больше интересных материалов? Поддержите нас оформив заказ или порекомендовав знакомым, 30% скидка для пользователей Хабра на уникальный аналог entry-level серверов, который был придуман нами для Вас: Вся правда о VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps от $20 или как правильно делить сервер? (доступны варианты с RAID1 и RAID10, до 24 ядер и до 40GB DDR4).
VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps до весны бесплатно при оплате на срок от полугода, заказать можно тут .
Dell R730xd в 2 раза дешевле? Только у нас 2 х Intel Dodeca-Core Xeon E5-2650v4 128GB DDR4 6x480GB SSD 1Gbps 100 ТВ от $249 в Нидерландах и США! Читайте о том Как построить инфраструктуру корп. класса c применением серверов Dell R730xd Е5-2650 v4 стоимостью 9000 евро за копейки?
I will tell you a less interesting, but dramatic story that took place from March 18 to March 25, when one of our clients underwent the strongest DDoS attack. I will not talk much about CloudFlare, if you want to learn about our service, just go to the site. But when our client came under attack, it was an event from the line of those journalists like to write about, and this attack was covered by the New York Times for 4 days in a row. It was a really powerful attack, so I want to talk about it.
So, there is a non-profit organization called Spamhaus, which tracks spammers and their activities. I know these guys for a long time, since 2006-2007, when an alleged spammer from Illinois sued them. Then the founder of this organization, Steve Linford, called me and asked if I could speak for them as a free attorney. I replied that I used to be a private lawyer, which means that I taught students in the class, but I never spoke in court. I very quickly found a real lawyer for them, who soon helped them to "resolve" the situation, so I know that Spamhaus is a very decent organization, they act quite openly.
Our clients are many organizations, but they don’t like to talk openly about their problems, but the guys from Spamhaus are not so closed, and when they were attacked, they called us and said: “Hey, we can tell you our story”! So this is a story about the attack on Spamhaus and about what happened behind the scenes when they were "brought down".
I broke this story into 3 parts. The first is the nature of the attack, what it is and who is behind it. If you want to know more about this, then come to DefCon, where I am going to speak next week.
The second part is what needs to be done to stop the attack, how to create a network that is really capable of it. This is not a story about great intelligence or unbelievable software, it’s really the story of designing a smart network.
The last part is what you can do on your own networks to stop this type of attack. I promise that the outcome of the speech will not be campaigning to use the CloudFlare service as the most protected, instead I will tell you about the advanced things, having done that, you will become less vulnerable to such DDoS attacks.
So, on Monday, March 18, we received a call from one of the Spamhaus volunteers, who informed us about the attack. This is how their website looked like at that time - the attackers brought it to the offline state, effectively disconnecting the Internet.
Later, the media wrote that the attack was made by someone Fenn, who lives in a bunker somewhere in the Netherlands. This guy was not the real initiator of this action, he was probably just the one who was willing to give an interview to the New York Times. Rather, the real mastermind of the attack is the 15-year-old teenager from London, who is now under arrest, but the press does not make a fuss about it.
In an article dated 23.03.2013, the New York Times newspaper wrote that the attack was carried out by the Danish company Cyberbunker in response to the fact that Spamhaus was blacklisted for spreading spam. The headquarters of Cyberbunker was indeed located in an abandoned NATO bunker (note the translator).
Most likely, the motive of the action was the fact that Spamhaus declared someone a spammer. This organization does a lot of good things, but it happens that they pinch someone’s tail, and in this case, the offended side responds, in this case it organizes the attack. Spamhaus is technically savvy enough to repel many different types of attacks, but in this case, a massive group attack hit their DNS infrastructure and simply knocked the site out of the network. It could happen to any organization, and they wanted to stop it. They knew that we were helping to prevent such attacks, so they turned to us.
So, from March 18 to March 21, what we call the "tantalizing attacks" took place. Such attacks are our bread, weekly we have several such attacks. So that you have an idea of their scale, I will say that last week we recorded 162 attacks on our clients, and we rather successfully stopped them. These DoS attacks occur at an intensity of 10 to 80 Gigabits per second. For some organizations, such as financial ones, attacks that occur with such intensity can be dangerous. So, last year, some financial institutions suffered from DoS attacks of 60-80 Gbit / s, but I repeat - these are fairly common attacks, familiar to us, and there is nothing in them that deserves a mention at the BlackHat conference.
Another thing to note is that the DoS attacks are not all the same, the size of the attack, that is, the number of packets attacking the site, is important. What we will talk about today refers to attacks of level 3, that is, attacks against the network layer of the OSI model. This is pure volume attack. But it is not the worst, much worse than the attack of the 7th, application level, which is not so voluminous, but it creates much more problems for specific applications. But today I am not going to discuss attacks that require special software.
The same applies to Level 4 DoS attacks, when an attacker uses a small amount of Syn-packets, causing significant damage to the victim, because his operating system is not able to cope with these requests.
In our case, the attack is based solely on a large volume, which tries to destroy you at level 3, knock you out of the network and overload the port with traffic. On Wednesday, March 20, one of the ports of our network, or rather, a collection of ports, on one of which the Spamhaus site sat, was attacked.
This slide shows the proxy cache, the blue line is traffic coming from our network, and the green area is incoming traffic. The high peak at the end of the chart is an attack that fell entirely on Spamhaus and brought down our network. Our network can easily withstand attacks of 75 Gbit / s intensity, because it is designed to “digest” such a volume of traffic.
However, during the attack on March 20, the “ceiling” of DoS intensity was reached - 100 Gbit / s. This looked like an exceptional case, because if you buy or create a basic router, then contact someone like Cisco or Juniper, and the coolest and largest “box” you can buy from them has a port with a capacity of 100 Gig. Theoretically, you can use several 100-gigabit ports and pass them through one interface, but the ceiling of DoS attacks usually does not exceed the specified amount.
Soon after, Tom phoned me, one of our network engineers, and said that the intensity of the attack had increased. She can push herself beyond the boundary values, so I asked how much she grew up, thinking that we are talking about a slight excess. The next slide shows the intensity of the new attacks that took place on March 24-25.
The graph shows the universal coordinated time of UTC, I live in San Francisco, we had another 23 at this time, so you can imagine what my dinner was on March 23. They called me and said that the attack exceeded the ceiling of 100 Gigabits, but we did not think that very soon we would be able to see something like that! Suddenly, 4 calls were received from the provider of one of our incoming networks and network engineers that we now have traffic with an intensity of more than 300 Gbit / s, which reaches the limit of our network's capabilities.
The graph shows that throughout March 24-25, attack rates dropped to 90 Gbit / s and increased to 300 Gbit / s, causing a denial of service to the incoming traffic of one of our providers, which has completely exhausted network bandwidth.
The question is how? How did a hacker manage to do this if it is known that the ceiling of the equipment is 100 gigabits? How did this guy from London, this 15-year-old, manage to organize such an unprecedentedly intense attack?
Many people, including the media, expressed the wrong opinion that a botnet is needed for such an attack. But in this particular case, there was no real traffic from the botnet network. Other people said that it was Anonymous, because there are many people in this community who are coordinated and can work together to launch attacks of a certain kind. No, it is not. The attack was carried out by 3, possibly 4 people, one of whom was a real technical inspiration, but very limited in terms of resources and performers.
Many people felt that outstanding technical skills were required to carry out such an attack. I say that I saw the code that provoked this attack, which is about 50 lines in C, not including the headers that are usually included in such an exploit, so this is quite a bit of information.
In just a second, I will show you how you can launch such an attack. What is really needed for this is the list of open clients DNS, or DNS resolvers, later I will tell you what it is. The second thing you need is a few servers, or just low-powered computers running on networks that allow you to replace the original IP.
These are the basic components for attacking, and if you have these two things, as it turns out, even in very small quantities, then you can launch a very, very large attack. So far, nothing that weakens such attacks has changed, and this is a bit creepy.
Let's talk about open DNS resolvers. DNS is something like small blank pages for the Internet, it is a caching system that creates the ability to respond to requests that were sent to the server.
An improperly configured DNS resolver, also called an open DNS resolver, answers questions no matter who asks it. This “slutty” DNS resolver answers everyone, no matter who asks. It works without any restrictions. It has no restrictions on the number of requests, or on who to receive requests from.
You may ask: “What about Google? After all, he uses an open DNS resolver! ” However, organizations such as Google are doing a lot to limit the number of requests addressed to resolvers, in order to make it difficult to launch attacks, they limit the size of the answers that are provided on requests. Just last week, the guys from Bind announced a new extension that allows them to set a speed limit for requests, this is exactly what we lacked for a long time. Because what scares you most is that authoritative, open DNS servers are increasingly being used for attacks.
When carrying out any attack, the hacker seeks to strengthen it in any way possible, for example, using a botnet. If I have very few of my own resources, I want to use them to create additional resources. At the BlackHat conference, it was said that DDoS is a perfectly asymmetric attack, because one little person can create huge problems, and open DNS resolvers clearly demonstrate this.
This slide shows the key command to launch an attack through a Windows terminal, or console.
This command was launched at 2 am from an open DNS resolver 63.217.84.76, which worked on the PCCW network. If you send this request, the server will respond to you.
digANY is just a DNS query that initiates a search for any domain names NS, isc. org means that they wanted to show you every possible type of DNS record, so their DNS file is huge, followed by the resolver IP address and the flag edns = 0, which says: “Give me everything, including dns.sec and many others flag types, which you may not get by default. The notcp flag is an important element, because it orders the resolver not to use TCP for data exchange, but to send them all over UDP, and bufsize sets the buffer size to 4096, that is, to make the buffer as large as possible to send very “fat” packets.
So, you run this cute little query of 64 bytes in size and get back what is now running on the screen. That is, on a 64 byte request, you get a response of 3363 bytes in size.
So, if you manage to find an open DNS resolver, you send it 64 bytes and receive a response of more than 3300 bytes in size. That's what I mean when I say that you can use tiny resources to create a big problem. In this case, we get an attack gain of about x50.
However, the gain does not work by default, and if I send a small request to get a big answer, I just send a DDoS attack on myself. Therefore, the second mandatory component for such an attack is a network that allows the source IP address to be spoofed.
That is, I try to pretend that the DNS request was sent from an IP address from which it was not sent. I send a message to the DNS resolver, and since UDP is a “burn and forget” protocol without a handshake, after receiving the request, it simply sends the answer to someone it considers as the sender.
Well-established networks do not allow this, they do not allow packets to come from IP addresses that do not belong to this network — these are BCP38 recommendations. The majority of Internet providers follow these guidelines when creating networks, however there are networks that ignore this rule.
So, if we have two necessary ingredients, then since UDP does not use a handshake, you can easily change the IP address of the source of the request. For the attack on Spamhaus, the specific DNS shown on the slide was used, and the confidence that the connection is made via UDP, which does not require any handshake. Then, the responses of all of these open DNS resolvers located around the world are sent back to yours or any other target network.
The secondary effects of these DDoS attacks affect customer support, because hundreds of thousands of open DNS servers are involved in this attack. Their users track the path to an attacker address 190.93.243.93, from which tons of packets emanate from their infrastructure, and begins to flood you with angry messages like “you don’t know what you are doing, you don’t know how to run a normal network, your computers hacked, how dare you attack me! ”and the like. And you have to be very polite to explain to them that in fact they are attacking you, and they themselves suffer because they have not correctly configured their DNS resolver, the answers to the queries come back, and all this ultimately affects you. themselves.
Actually, this is an opportunity to teach people to “clean” their networks, and this is the positive aspect of such attacks, because people do not realize that this happens all the time, due to the incorrect configuration of open DNS servers, only on a small scale. And only when the intensity of attacks increases significantly, they begin to pay attention to it.
I think all of you are familiar with such old types of attacks, like Smurf, where you use the transmission of ICMP broadcast requests to routers via routers, and use source spoofing, causing computers to respond to the victim of your spoof by hammering communication channels.
On the positive side, the routing industry has learned to deal very effectively with this attack. They are still found, but it is quite difficult to launch the Smurf attack. However, in general, attacks by reflecting DNS responses are very easy to organize.
How common are the components of such attacks? The next slide shows the Open Resolver Project, this is quite a revolutionary product. Many of those present here argued for a long time whether to publish on the network a list of all open DNS resolvers running on the Internet. The fact is that this list can be used both by the “good guys” so that they can point clients to incorrectly configured networks, or by the “bad guys” who can use it to organize DoS attacks.
It is not by chance that this project was launched on Monday, which followed the weekend, during which we endured all this hell. Thus, the largest Internet providers of the first level were able to make sure that the “bad guys” have this list for a long time and therefore it is necessary to start cleaning it, getting rid of open servers. Thanks to this site, you can enter a range of your IP addresses and check whether there are open DNS resolvers on your network. Thus, if the PCCW did a similar check, they could clear the network before they had problems. We attracted the media to cover the role of this project, so that all providers know about this list, because the number of open DNS resolvers used on the Internet is 28 million. I would be happy to inform you that today this number is 30, 40 or 50 million, however, unfortunately, at the time of launch it was only 21 million, so we are clearly moving in the wrong direction.
The next thing I want to mention is the completely abnormal percentage of networks that allow spoofing. The slide shows the MIT project that allows you to track the spoofing networks. It was difficult to check them remotely, but tests show that almost 25% of the Internet allows spoofing.
In 2013, there can be no excuse for not using BCP 38 for border routers. We must not allow packets that are not born on your network and that come from unknown IP addresses to increase the load on the routers. Any router released after 2002 makes it very easy to enable such a function, so do not use it - just laziness!
The good news is that mostly large botnet networks do not support this kind of spoofing. Global networks are mostly closed for spoofing, so a catastrophic scenario is possible if hackers succeed in spoofing, for example, China Telecom's Chinese network. В этом случае можно было бы использовать ботнет для генерации трафика на DNS-серверы для осуществления потенциальной атаки огромного размера. Вероятно, присутствующим легко понять, как трудно избавиться от всех открытых резолверов. Но вы можете реализовать в собственных сетях возможность игнорировать чужие пакеты, чтобы не подвергнуться подобной атаке.
Давайте рассмотрим два конкретных инцидента, имевших место при атаке на Spamhaus. Первый был 24 марта и заключался в атаке интенсивность около 309 Гбит / с на протяжении 28 минут. Паника началась со звонка Тома, который сказал, что у нас атака более 300 Гбит / с. Я спросил, работает ли наша сеть, и Том сказал, что мы пока держимся, мы можем обеспечить Spamhous и других клиентов. И хотя наблюдаются небольшие потери пакетов и задержки в передаче трафика, но в целом сеть держится. Тогда я сказал, чего же вы так испугались? Они ответили – потому что атакующие используют всего лишь 0,1% открытых резолверов, которые существуют в мировом интернете! Но если они смогут использовать хотя бы 1%, у нас будет не 300-гигабитная, а 3-х терабитная атака. Если они будут использовать 10% открытых DNS-резолверов, мы получим 30-терабитную атаку. Я думаю, что сегодня мы могли бы выдержать до 3 терабит трафика, но в марте у нас не было такой возможности, мы не могли победить 30-ти терабитную атаку и я не знаю никого, кто смог бы её выдержать. Это было очень серьезным вызовом работоспособности сети.
Когда мы стали разбираться, откуда исходит атака и просмотрели кучу сетевых провайдеров первого уровня, то обнаружили всего 3 сети, которые позволяли спуфинг адресов, и возможно, их обслуживало от 5 до 7 скомпрометированных серверов. Один был в Сан-Хосе, другой находился в Миннесоте и третий в Германии. Было задействовано всего 3 сети, и эти скромные ресурсы привели к относительной большой атаке.
Следующий пик пришёлся на 25 марта, когда был зафиксирован трафик со скоростью 287 Гбит/с на протяжении 72 минут. Использовалось практически такое же количество открытых резолверов – 31154, и те же три спуфинговые сети. Ещё раз повторюсь – атакующий задействовал очень ограниченные ресурсы.
Отмечу, что у нас был всего 1 атакующий с одним ноутбуком, у нас есть скриншот, который я покажу позже. Он контролировал от 5 до 7 скомпрометированных серверов, 3 сети, позволяющие подмену оригинальных IP-адресов, которые генерировали трафик интенсивностью около 9 Гбит/с, направленный на 0,1% открытых DNS-резолверов. И всё это привело к возникновению 300 Гбит/с DDoS-трафика.
Многие люди считают, что это была самая большая атака в истории сетей, не могу сказать, правда ли это, но она действительно была самой большой из всего того, что мы видели.
Это история о великом организаторе, которому удалось проделать то, о чем раньше даже не догадывались. Мы все знали проблеме открытых резолверов, это не было каким-то умным новым взломом, порчей сетевых протоколов, атакой фрагментации пакетов и так далее, это выглядело так, как будто пещерный человек просто долбит огромной дубиной тебе по башке.
Повторю снова – то, что произошло, было невероятно эффективной атакой и на Земле очень мало сетей, которые способны ей противостоять. Опять же, мы очень гордимся тем фактом, что мы сделали тогда и что мы продолжаем делать каждый день, и тем, что продолжаем инвестировать в нашу сеть, чтобы сделать её больше.
Немного расскажу о том, что мы сделали для того, чтобы иметь возможность сопротивляться атаке, что мы предприняли, чтобы её остановить. Некоторые из наших мер применимы в общем случае, некоторые являются уникальными. На этом слайде вы видите, что мы предоставляем услуги по всему миру, имея 23 точки подключения к сервису CloudFlare, а к концу года их количество увеличится до 50. Так что на сегодня у нас имеется развитая сеть X-класса.
Мы изначально подумали о целесообразности нашей сети, но то, что было заложено в неё в начале, на сегодня превратилось в очень мощный инструмент. Мы используем протокол под названием Anycast для передачи пакетов методом BGP через множество поставщиков DNS, которые также используют Anycast. Фактически мы используем его для HTTP-трафика, как наиболее стабильного. Скажу, что это большая головная боль – обеспечить нормальную работу такой сети, как наша.
Если вы не знакомы с протоколом Anycast, то вот что он собой представляет. В обычной схеме сети имеется один источник и один пункт назначения, а в Anycast один источник связан с несколькими пунктами назначения, при этом для доставки пакетов адресату выбирается кратчайший путь. Таким образом, если у нас имеется 23 дата-центра по всему миру, то все они имеют один и тот же IP-адрес, например, адрес Spamhous. Поэтому, когда мы имеет дело с атакой интенсивностью 300 Гбит / с, она распределяется между всеми дата-центрами таким образом, что на каждый дата-центр приходится порядка 13 Гбит / с трафика.
Проблема состоит в том, что атакующий тоже может заняться математическими подсчётами и очень быстро перенаправить атаку прямо на край нашей сети. Это немного дурацкая затея, так как в результате атакующий может получить эффект, обратный усилению. То есть вместо того, чтобы усилить атаку увеличением её поверхности, он вынужден сократить эту поверхность и уменьшить интенсивность атаки.
29:00 мин
BLACK HAT conference. Survival lessons in a 300 Gbps DDOS attack. Part 2
Спасибо, что остаётесь с нами. Вам нравятся наши статьи? Хотите видеть больше интересных материалов? Поддержите нас оформив заказ или порекомендовав знакомым, 30% скидка для пользователей Хабра на уникальный аналог entry-level серверов, который был придуман нами для Вас: Вся правда о VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps от $20 или как правильно делить сервер? (доступны варианты с RAID1 и RAID10, до 24 ядер и до 40GB DDR4).
VPS (KVM) E5-2650 v4 (6 Cores) 10GB DDR4 240GB SSD 1Gbps до весны бесплатно при оплате на срок от полугода, заказать можно тут .
Dell R730xd в 2 раза дешевле? Только у нас 2 х Intel Dodeca-Core Xeon E5-2650v4 128GB DDR4 6x480GB SSD 1Gbps 100 ТВ от $249 в Нидерландах и США! Читайте о том Как построить инфраструктуру корп. класса c применением серверов Dell R730xd Е5-2650 v4 стоимостью 9000 евро за копейки?
Source: https://habr.com/ru/post/439706/