Recently I was asked to assist a small NZ business, because their ecommerce website was offline due to an HTTP Flood DDoS attack.
These things don’t often get seen by small players, and they are scary and expensive to suffer through. This is the story of what happened, as it happened, and what impact it had. I won’t be identifying any of the people/companies involved unless they ask me to. I’ll be keeping the technical level reasonably high though, so don’t worry if you miss some of the details!
The Players
- SB - the Small Business who called me for help
- CMS - The Content Management System company who write & operate SB’s website
- ISP - The Internet Service Provider who host the CMS servers and provide connectivity
Initial Detection
Early one Friday afternoon, a network system administrator at “ISP” noticed that the load on his firewall was unusually high. Digging into the situation, he saw large volumes of HTTP requests coming through to “CMS’s” webserver, flooding it.
On investigation, he saw only simple requests (”GET /” without identifying which website name was required) coming from many different origins. The CMS webserver in question hosts multiple customer websites, and it wasn’t clear if any of these were a specific target for the attack, or whether it was just the server itself.
The engineer then tried to configure the Intrusion Protection System (IPS) on the firewall to detect and block these requests, which are basically not valid anyway. However, in order to try what he wanted, the IPS software had to be upgraded and then reconfigured, and he was not sure how best to configure the system to be fully effective. The attack continued to keep the CMS webserver effectively offline.
A Change In The Wind
Overnight, the attack changed profile; instead of a simple non-specific request, it had changed to specifically request one of the hosted websites, one belonging to the end-customer “SB”. From the perspective of the ISP and the CMS company, this was great news — they had a chance to isolate the attack and bring their webserver back for the other customers.
The change might have come about because the IPS was being successful, or it might have just been the original attacker zeroing in on his original target; we can’t tell.
Utilising a ’spare’ virtual server hosted offshore, the ISP quickly configured a reverse proxy and loaded up some extra software that tries to fight off denial of service atacks. They then changed DNS to send requests for the SB website to that proxy instead of directly to the CMS server; this was successful, and the DDoS attack traffic started arriving at this new location. Unfortunately, the new server was unable to keep up with the size of the attack, and SB’s website was still offline. However, the CMS webserver was no longer being overloaded, and other customer websites came back online.
Stalemate
The ISP had put a lot of resources into fighting off this attack so far, and were reaching the edge of their experience and expertise. They had managed to move the attack away from their core systems, recovering service for the majority of the affected websites. However, they could not “beat” the attack and restore SB’s website. They had been talking to their Upstream provider, who could provide mechanisms to help beat a simple Denial of Service attack, but not a distributed one — at least, not at a cost-effective rate. The ISP did some research online to find companies to help, and identified one possible trustworthy-looking provider who offered a specific DDoS-protection service. They suggested to the CMS company that SB should pay for this extra service.
Second Opinion
It was roughly at this stage that SB got in contact with me; he didn’t fully understand what was happening, and didn’t understand what his real options were. This sort of communication problem is all too common where you have an intensly technical issue affecting the business — trying to explain technical things to non-technical people is very difficult.
My first approach was to talk to the CMS company, and find out what they thought the issues were; after all, they are the one with a financial/contractual relationship with SB. From there, it was easy to track back to the ISP, and they were willing to spend some time talking to me, explaining in their own language what they had seen and what they had done. This enabled me to get back to SB quite quickly, and tell him that there was a real “external” problem, not caused by CMS or ISP, and that they had genuinely been trying to address the issue, and had been acting competantly and in good faith. Maintaining trust between customer and supplier is very important, especially when unusual costs are being experienced, and someone is going to end up paying for it.
One of the questions from SB was about just this, “who should pay?”. The contract trail wasn’t easily available, but I was able to turn up a copy of the original hosting contract that SB entered into before the CMS company took it over; there was a reasonable clause in there indemnifying the host from the effects of virus attacks, and it’s not a long step from there to DDoS. I had used the analogy that a DDoS was like a lightning strike — inherently unpredictable but very destructive — but the next step up from the physical world isn’t directly applicable; if lightning strikes your rented house chimney, your landlord would have to pay to fix it, and would invoke insurance to cover his losses. In the virtual world, the contract with the ‘landlord’ has been able to exclude these things.
Trying To Beat The Attack
The reverse proxy setup that the ISP had deployed was reasonable, but I had a few different ideas. I suggested some experiments and configuration changes that might help, but without hard data to analyse it was unclear whether these would help. Initially, the proxy was trying to simply pass all successful requests back to the CMS webserver; I suggested trying to handle the initial “GET /” request directly from the proxy, therefore reducing load on the CMS — also with the hope that the atackers were not actually listening to the responses, and would therefore never end up sending a redirected request to the CMS. This did appear to be potentially successful, but the attack was still coming in fast enough that the proxy was not able to handle the traffic, and therefore was never able to effectively proxy legitimate requests back to the CMS.
Additionally, the proxy server had very quickly exhausted the bandwidth/data limits set by its contract, which would have made any further attempts increasingly expensive. In order to reduce the cost of traffic to the proxy server, the ISP configured its firewall to reject all HTTP requests. Even so, there was over 2Mbps of traffic trying to get through.
The route to the external “DDoS Protection” service seemed to be the only reasonable short-term business decision, and this was accepted all round as a good choice. I tried to validate this choice by speaking to a large number of experts around the NZ marketplace — security auditors, security equipment vendors, network security technicians and other consultants.
Luck Rather Than Judgement?
Just as I was talking to one last security consultant, we checked the website and things seemed to be working once more! I got hold of the ISP again and this was confirmed; the attack volumes had dropped significantly, and the reverse proxy was now able to handle the load. Luckily the ISP had been constantly monitoring the situation and just before the contract to the external DDoS service had been activated, they noticed that the attack seemed to have mostly abated. Over the next day the attack traffic reduced further; there are still some remnants, and even some attacks still arriving directly on the CMS machine as they have obviously ignored DNS updates. However the levels are well within the capacity of the normal systems to aborb without ill effects. SB is back online.
Conclusions
Painting A Target On Your Back
One question that was difficult to answer, and quite important, was “who was the attack aimed at?”. The initial attack arrived at the CMS webserver with no identifying details, and could have been aimed at the ISP, the CMS company, or any one of the multiple customer sites hosted there. When the attack changed to name the SB website, that would seem to indicate that it was aimed at SB’s business; however SB operates other domain names on the same CMS service, and they were not attacked at all. It is therefore possible that the selection of SB’s domain name was basically only an attempt to overcome the IPS blocking simple requests.
Another possibility is that the attack was aimed at the CMS server, and not the CMS company. One of the observed habits of DDoS attacks is that they are sometimes aimed at each other’s control systems, as part of the cut-throat world they come from. If the CMS server had been infected with some malware that acted as part of the control network for a rival DDoS system, then the server itself would have been the target — the invocation of the SB domain name would effectively been a mistake while trying to work around the early protection provided by the IPS.
If the attack had been aimed at SB specifically, experience from the wider world suggests that there would probably have been some sort of extortion attempts. If a competitor had been trying to shut SB down, they would have targetted all of his domains.
Without any real indication of intelligent decisions behind the targetting, we’re left with the unsavoury possibility that this whole attack was simple mindless vandalism.
A Better Approach?
Throughout this experience, most of the technical people, myself included, focussed on the technical details of the attack, and attempted to beat it despite insufficient experience and resources. However we were losing sight of the business cost; SB has had to hand over extra money, because we were spending it, and possibly suffer a longer downtime.
One piece of advice that I received afterwards was that a good approach for a small company is to just shut down completely for a day or so, and allow the attacker to detect and claim “victory”. While we were unsuccessfully trying to absorb the attack, the website was effectively down from the perspective of real customers, but may have appeared to be “up enough” to encourge the attack to continue. Its an unknown, but perhaps SB would have suffered less total outage if we had taken the site down. Its very hard for technical people to admit defeat while there are still reasonable options to be tried, but at the end of the day a customer’s website is a business tool, not a technical one.
The Losses
The ISP have incurred costs in relation to their time & effort, and the costs of exceeding the traffic quota on their virtual server. On the other hand, they have learned a lot about DDoS attack. The biggest loser is the SB, who have had their main ecommerce offline for a number of days — long enough to affect their search-engine ranking, as well as to lose potential sales. Luckily for this particular SB, I don’t think that would represent an unrecoverable problem, given the nature of their business; but it could be a fatal interruption to cashflow for many companies.
Other Comments
The attack was described as being detected by the ISP; but it should have been detected by the CMS noticing that their webserver was under high load, or perhaps effectively offline. Having an effective system monitoring system can help you to detect problems early on, giving more time to deal with issues.
Tags: DDos, security
Shell
9) PFtop
10) Filter Logs
11) Restart webConfigurator
12) pfSense Developer Shell
13) Upgrade from console
14) Enable Secure Shell (sshd)
Enter an option: 