By Andy Dentone and Jeffrey Long
On April 3rd, 2019 at about 8:30 AM PT we started noticing the beginning of a **DDoS attack.
It seemed just a little odd at first – with a few websites running slow and some backups we were doing at the time seemed to stall.
Then we noticed our monitoring software’s alerts, and within a few minutes, the first support ticket came through.
By 9:00 AM PT, we had stopped working on building & backup up websites and started focusing only on fixing whatever was causing the major slow-down.
By 10:00 AM PT, we had received more support tickets than we usually see in several months.
While the attack was in progress, we did our best to answer the phone every time a concerned customer called in while we did everything we could to add a layer of protection for each and every hosting account.
How long did it last?
This attack made any website on one of our web hosting servers inaccessible from any public web traffic for about 22 hours. Since the end of this attack (just after 6:00 AM PT April 4th, 2019) we have been running without issue.
Did the “hackers get anything”?
No. It looks like tens of thousands of malware-infected computers were directed to attack one of our main hosting servers. Our other servers were unaffected by this issue. Other servers on the same network were also unaffected. The attack caused frustrating and costly downtime, but that is all.
What did we learn?
We have learned many things from this attack.
Keeping ahead of hackers
We have been actively and routinely updating our server hardware and software technology since we started hosting websites in 2010. This is one important way we keep ahead of hackers. In fact, we updated our best-practices in late 2018 and we’ve been anticipating the need for an upgrade to the servers, including the one that got attacked. The attack happened on a server where the new protections had not yet been widely implemented.
There are advancements in operating systems and server software technology that allows for better separation between each hosting account’s available resources. It isolates each customer into a separate Virtualized Environment which partitions, allocates, and limits server resources.
Set Acceptable Limits
All of our new servers have defined limits set on each account to prevent one website or a combination of websites using too many resources that it could take the whole server down.
Note: These limits should not affect most of our clients unless there is a large spike in traffic that we did not anticipate. If you are anticipating a large spike in traffic (I.E. new product or news release) let us know so we can adjust the limits for your account.
We also keep track of server resources and monitor them so that we don’t “oversell” hosting packages. This makes sure each account has plenty of resources to perform at its maximum.
We have learned that we need to continue with our upgrade the server to more powerful one with more controls and monitoring. We’ve started planning a new hosting server (rolling out June 2019) with new protections and security added.
Better Domain Routing Security (+ improved load times)
As part of our recommended setup, we now leverage CloudFlare’s DNS for nearly all websites. If an attack happens, CloudFlare steps in and saves the day with a local copy of your website. This brings an extra level of security and peace of mind. As a bonus, it also provides a free Content Delivery Network (CDN) for our clients which makes your website load faster for your visitors.
We are working to expand our offerings from CloudFlare in the near future.
Can it happen again?
Yes. It could happen again. There is no 100% guarantee on website hosting unfortunately. The Internet is a wild and crazy place. Fear not though – we have your back!!
We’re doing our best to prevent another complete outage. We are working on a plan to have more protections in place to prevent another attack like this by the end of Summer 2019.
**DDoS Attack Explained: