I’m a loyal user of AWS since years, I simply like the concept of doing everything in the cloud and I always believed they have an amazing set of services to offer. Although I’m not a heavy user at all (I typically spend less than 10USD per month), I use it quite frequently. I launch an EC2 instance every now and then to run an experiment, I put my photo’s on S3, I’m developing against Elastic transcoder….you know, the usual stuff geeks do.
On Wednesday, 24th September, I received an email from Amazon saying the following:
Greetings from Amazon Web Services.
Your security is important to us and we have detected suspicious activity on your Amazon Web Services account ending in 9123. We currently see current charges of 1.14 due to increased EC2 usage.
Please log into your AWS Management Console at https://console.aws.amazon.
com/, check if all usage is authorized, and delete all unauthorized resources. Please pay special attention to the running EC2 instances and IAM users, roles, and groups (please check all regions – to switch between regions use the drop-down in the top-right corner of the management console screen).
You must also change your AWS account password and rotate and delete your old AWS access credentials.
Also, please make sure that you never share your AWS Access Key ID as well as your AWS Secret Access Key with anyone and never publish them in an environment where other people have access to them. In addition, industry best practice recommends frequent access key rotation. Exposing your credentials would allow other people to access your account and you will be responsible for the billing charges for their usage.
If you are unable to delete your AWS access key and stop any unauthorized usage within a reasonable time, we may need to suspend your account to protect you from unauthorized charges. . If you have verified that all usage is authorized and you accept the billing for this usage, please respond to this email and confirm.
…..and so on
No big deal I thought, in the end the charges were still very low. So I logged into my AWS account and I saw indeed that 20 EC2 instances were launched. What the heck…who did this? So I immediately terminated these instances, changed my password and deleted all of my AWS keys. So I felt pretty safe again and could sleep on both ears that a hacker wouldn’t get access again to my account. Case closed…I thought at least!
Then last Saturday evening, the 27th September, I received a call from AWS security department. To be honest, I wasn’t really in the mood to pick up the phone so I decided not to take the call immediately. Who does on a Saturday evening while having dinner with the family? In the end, I changed everything I could change and everything was secure again, nothing to worry about. I decided I would take care of it after the weekend.
That same number called me a couple of times in less than 10 minutes so I thought it was probably not that innocent after all and I decided to pick up the phone eventually. I’m glad I did … or actually…I wasn’t if I think about it now. I had no reason to be happy at all I would soon find out. A lady from AWS security informed me that they’d detected some suspicious activity on my AWS account. Obviously, I knew this already because I received the email couple of days before. That’s also what I told her on the phone. Then she told me the account was displaying a billable amount of close to 40.000USD!!! Needless to say that my heart rate went up immediately. See the screenshot below in case you don’t believe it immediately.
I simply couldn’t believe it as I killed those 20 EC2 instances on Wednesday already, so what else could it be. Then, the lady on the phone told me to check in each AWS region and effectively….I saw 20 EC2 instances in each region. And there are 8 regions in total. So the hacker did launch 160 EC2 instances in total!! After 4 days, these servers accumulated to a billable amount of close to 40.000USD. To be honest, I was shocked…as far as I know I had not exposed any of my AWS credentials anywhere. She advised me to open a support case immediately. Which I did…I asked the case manager to call me back and they did in less than 5 minutes.
The helpdesk engineer calmed me down a bit by mentioning that it happened to more people. I spent about 1 hour on the call with the support engineer and we went through my AWS account. In fact, we were having a walkthrough how to make the account secure again and he basically told me all the things I had already done in the past couple of days. He also told me he would start a procedure to waive the costs from my account. This would take 7-14 days and he could not guarantee me the costs would be waived. This depends on upper management he told me. He also told me not to be too concerned in the coming days although he could not make any promised that the outstanding sum would be waived.
Now we’re a couple of days later and I have to admit that it keeps me busy despite the gentle words of the AWS engineer. I’m struggling with a lot of questions: what if I need to pay that amount, how did this happen, why didn’t I look into all the different regions the first time….The rational ‘me’ thinks they will not charge me as it’s clear that this is the work of a hacker. On the other side, it’s just not so obvious that large companies like Amazon will waive this huge amount easily away. In the end, the resources (in this case EC2 instances) have been used and somebody will need to pay for it.
Update 01 October: I googled a bit and it seems more people had the same issue. Apparently these hackers are doing this for Litecoin mining.