March 25, 2015 at 01:17How I managed to owe Amazon $ 12,000 in 1 day
I am the head of a small IT company from Zelenograd. We are engaged in the integration of 1C and telephony programs. The company employs a little less than 20 people, and it so happened that I myself am responsible for the entire IT infrastructure.
Basically, I love doing this and getting to know various new technologies. One of such technologies has become virtualization and, in particular, such an interesting service as Amazon AWS.
It is often necessary to quickly deploy several virtual machines with white IP addresses for laboratory purposes, use them for a couple of days, and nail them without regret. In Amazon terminology, this service is called EC2, and allows you to perform such manipulations in minutes. It is very convenient and costs a penny, because Payment takes place at the hourly rate.
So, back in 2011, I actively studied Asterisk and used the service, at the end of the month I was charged $ 20-$ 30 off my card. Then the need for the service disappeared, I turned off all the machines, made some backups and began to receive monthly bills of $ 3.66 and somehow all my hands did not reach me to find out what this amount was written off for.
We started a new project and there was a need to create a fault-tolerant configuration. Of course, I remembered Amazon AWS. I went to the admin panel, looked what’s new, went through the menu items, figured out what 3.66 was written off for, it was a 50 gigabyte virtual machine backup, deleted it, and went to bed with a sense of accomplishment.
Monday morning, a lot of work and a hundred letters from customers and colleagues, an inconspicuous letter from Amazon with the following content:
And I’m also not good at English, well, I think that’s cranks, I went in yesterday and pressed all the buttons there after half a year, and they are already panicking, okay, I’ll go in the evening to continue what I started ...
Before going to bed, I like to do something useful, especially from 12 to 2 nights. I opened the letter, I go into my account and see ... that I have 20 virtual machines running, and quite powerful, which means expensive ones.
My heart was beating, I singled them out together and in a couple of seconds I try to remove it, it does not work, then I just stop it. Well, I think they hacked the bastards, got a hundred different dollars, but I have all the cards tied to my account.
I go into billing, and there is such a picture:
It turns out that I have not launched 20 machines, but 140. On every continent , in every Amazon data center . This is what I got!
Characteristics of rented cars:
Over the next hour, I tried to stop all this, changed the login password, turned on two-factor authorization and created a case in the support service with the question: “What should I do with all this?”
In the morning I looked at the price of my car, well, about half the amount is there. Naturally, I didn’t have any desire to part with such a huge amount, by today's times. Case continued to hang in the status not reviewed, and I got another “funny” story about how to get to $ 12,000, and I’m fairly careful about security issues and the question: “How did I get this password?” Continued me to torment.
I tried to untie my cards from Amazonian billing, to which he refused and demanded that I go to the Amazon.com store website and manipulate my account there.
Amazon.com met me with a couple of questions about who my first manager is and what my childhood friend’s name is, and I was allowed to untie my credit card. Half the job done.
Then a call is heard on a cell phone, with a clearly non-Russian country code. Since I do not speak any languages other than 1C, C ++, C # and PHP, I did not pick up the phone. Fortunately, after 15 minutes I saw the first support response.
A small conversation ensued with the help of a colleague and a Google translator. Support understood the situation and began giving me assignments. Change passwords, delete all virtual machines in all data centers, delete all applications at computing power purchase auctions. And since I didn’t have anything mine in my account, I politely asked to remove everything from me centrally, which I was also politely refused.
Each virtual machine was protected from deletion, had to go into the settings and uncheck a special box. This entertainment took me another hour and a half. When I deleted the machines, I noticed that they all had the same creation and launch time, so they were created with some kind of script.
After removing all 140 machines, I decided to go over all the tabs and check if everything is ok. And what do I see? Right before my eyes, new cars appear, unite in a cluster and start up. And this is after changing all passwords and turning on two-factor authorization ...
I turn off the machines, read the support task carefully again, and I see a line about the rotation of API keys. Here is a pancake, I find them in the bowels of the settings of the authorization system, I look at the date - 2011, and without regret I delete nafig, I don’t even remember, I created them myself or got them in the load during registration.
After that, it takes me about 40 minutes to remove the new 60 cars.
I inform the support about the completion of the quest, to which I receive an answer that my question will be transferred to the financial group, after all, the amount is not small and do not expect a quick answer.
Here is a snippet from my account statement:
After some time, the bell rings again. Again, I do not take, to no avail mumble something into the phone, not understanding the questions. In the mail there is a letter, in Russian, from a certain dude who is responsible for the European segment, with the question of when you can phone and talk. I was delighted, finally you can talk with a Russian-speaking person and find out the prospects. Please call me back.
Dude seems to be a sales manager, he is interested in what I have for a project that required so much power and if I need a discount or some help. I explained the situation, told me that they had hacked me, gave me the case number. The man at that end was sympathetic to the problem, sympathized and said to turn on any questions. It's nice when the company has a human face.
From the useful, I have already enabled two-factor authorization wherever I could. I tormented my MacBook with various antiviruses that did not find anything.
About 5 days have passed, while there is no news, the amount of debt hangs behind me, April 4, settlement day ...
upd. There is a suspicion that the hacking occurred after an attack on the Bitrix website, in which the S3 storage was connected and the API key was specified, because a day before the hacking in the productive protection module it was written that 314K hacking attempts were reflected, and 1800 forms were filled out on my website feedback :(
upd2. The guys from Bitrix support analyzed the site’s logs. There was an attempt to attack, they tried to pick up passwords for the admin panel, use SQL injection and XSS vulnerabilities, but there was no confirmation of breaking.
More API key was not used anywhere ... Amazon is silent for now.
upd3. The issue is resolved; as a one-time share, I agreed to write off the debt. Respect to the guys from Amazon :)
I asked them for logs with IP addresses. I still want to understand if they hijacked a password or API keys. I have static IP addresses both at home and at work, and by the logs I can establish the first unauthorized entry.
Basically, I love doing this and getting to know various new technologies. One of such technologies has become virtualization and, in particular, such an interesting service as Amazon AWS.
It is often necessary to quickly deploy several virtual machines with white IP addresses for laboratory purposes, use them for a couple of days, and nail them without regret. In Amazon terminology, this service is called EC2, and allows you to perform such manipulations in minutes. It is very convenient and costs a penny, because Payment takes place at the hourly rate.
So, back in 2011, I actively studied Asterisk and used the service, at the end of the month I was charged $ 20-$ 30 off my card. Then the need for the service disappeared, I turned off all the machines, made some backups and began to receive monthly bills of $ 3.66 and somehow all my hands did not reach me to find out what this amount was written off for.
First swallow
We started a new project and there was a need to create a fault-tolerant configuration. Of course, I remembered Amazon AWS. I went to the admin panel, looked what’s new, went through the menu items, figured out what 3.66 was written off for, it was a 50 gigabyte virtual machine backup, deleted it, and went to bed with a sense of accomplishment.
Monday morning, a lot of work and a hundred letters from customers and colleagues, an inconspicuous letter from Amazon with the following content:
Account Closure: Please Read
Greetings from Amazon Web Services,
Please take the time to read this message - it contains important information about your Amazon Web Services account.
At Amazon Web Services, we routinely perform reviews of orders and customer accounts to protect our customers. After careful review of your account, we believe it may have been accessed and used by a third-party without your permission. Please review your AWS account for unauthorized activity per the directions below. It seems that someone obtained your personal account and / or financial information elsewhere, and used it to access your Amazon Web Services account.
If the activity was not authorized, your account will remain open for Five (5) days in order that you may secure your account. To regain access to your account, first, you will need to reset your password when you return to our site. Just click “Your Account” at the top of our Home page and select “Forgot your Password?” Enter your email address as prompted, and once completed, we'll send you an e-mail containing a personalized link. Click the link from the e-mail and follow the directions provided. Your new password will be effective immediately. If there is no unauthorized activity please reply as such and there are no additional actions needed.
In addition to resetting your password, we require that you log into your AWS Management Console at console.aws.amazon.com, check if all usage is authorized, and delete all unauthorized resources. Please pay special attention to the running EC2 instances, EC2 Spot Bids, IAM users, and access keys (please check all regions - to switch between regions use the drop-down in the top-right corner of the management console screen).
Please take the time to read this message - it contains important information about your Amazon Web Services account.
At Amazon Web Services, we routinely perform reviews of orders and customer accounts to protect our customers. After careful review of your account, we believe it may have been accessed and used by a third-party without your permission. Please review your AWS account for unauthorized activity per the directions below. It seems that someone obtained your personal account and / or financial information elsewhere, and used it to access your Amazon Web Services account.
If the activity was not authorized, your account will remain open for Five (5) days in order that you may secure your account. To regain access to your account, first, you will need to reset your password when you return to our site. Just click “Your Account” at the top of our Home page and select “Forgot your Password?” Enter your email address as prompted, and once completed, we'll send you an e-mail containing a personalized link. Click the link from the e-mail and follow the directions provided. Your new password will be effective immediately. If there is no unauthorized activity please reply as such and there are no additional actions needed.
In addition to resetting your password, we require that you log into your AWS Management Console at console.aws.amazon.com, check if all usage is authorized, and delete all unauthorized resources. Please pay special attention to the running EC2 instances, EC2 Spot Bids, IAM users, and access keys (please check all regions - to switch between regions use the drop-down in the top-right corner of the management console screen).
And I’m also not good at English, well, I think that’s cranks, I went in yesterday and pressed all the buttons there after half a year, and they are already panicking, okay, I’ll go in the evening to continue what I started ...
In the evening
Before going to bed, I like to do something useful, especially from 12 to 2 nights. I opened the letter, I go into my account and see ... that I have 20 virtual machines running, and quite powerful, which means expensive ones.
My heart was beating, I singled them out together and in a couple of seconds I try to remove it, it does not work, then I just stop it. Well, I think they hacked the bastards, got a hundred different dollars, but I have all the cards tied to my account.
I go into billing, and there is such a picture:
It turns out that I have not launched 20 machines, but 140. On every continent , in every Amazon data center . This is what I got!
Characteristics of rented cars:
Over the next hour, I tried to stop all this, changed the login password, turned on two-factor authorization and created a case in the support service with the question: “What should I do with all this?”
In the morning
In the morning I looked at the price of my car, well, about half the amount is there. Naturally, I didn’t have any desire to part with such a huge amount, by today's times. Case continued to hang in the status not reviewed, and I got another “funny” story about how to get to $ 12,000, and I’m fairly careful about security issues and the question: “How did I get this password?” Continued me to torment.
I tried to untie my cards from Amazonian billing, to which he refused and demanded that I go to the Amazon.com store website and manipulate my account there.
Amazon.com met me with a couple of questions about who my first manager is and what my childhood friend’s name is, and I was allowed to untie my credit card. Half the job done.
Then a call is heard on a cell phone, with a clearly non-Russian country code. Since I do not speak any languages other than 1C, C ++, C # and PHP, I did not pick up the phone. Fortunately, after 15 minutes I saw the first support response.
A small conversation ensued with the help of a colleague and a Google translator. Support understood the situation and began giving me assignments. Change passwords, delete all virtual machines in all data centers, delete all applications at computing power purchase auctions. And since I didn’t have anything mine in my account, I politely asked to remove everything from me centrally, which I was also politely refused.
Each virtual machine was protected from deletion, had to go into the settings and uncheck a special box. This entertainment took me another hour and a half. When I deleted the machines, I noticed that they all had the same creation and launch time, so they were created with some kind of script.
After removing all 140 machines, I decided to go over all the tabs and check if everything is ok. And what do I see? Right before my eyes, new cars appear, unite in a cluster and start up. And this is after changing all passwords and turning on two-factor authorization ...
I turn off the machines, read the support task carefully again, and I see a line about the rotation of API keys. Here is a pancake, I find them in the bowels of the settings of the authorization system, I look at the date - 2011, and without regret I delete nafig, I don’t even remember, I created them myself or got them in the load during registration.
After that, it takes me about 40 minutes to remove the new 60 cars.
I inform the support about the completion of the quest, to which I receive an answer that my question will be transferred to the financial group, after all, the amount is not small and do not expect a quick answer.
Here is a snippet from my account statement:
Call from Luxembourg
After some time, the bell rings again. Again, I do not take, to no avail mumble something into the phone, not understanding the questions. In the mail there is a letter, in Russian, from a certain dude who is responsible for the European segment, with the question of when you can phone and talk. I was delighted, finally you can talk with a Russian-speaking person and find out the prospects. Please call me back.
Dude seems to be a sales manager, he is interested in what I have for a project that required so much power and if I need a discount or some help. I explained the situation, told me that they had hacked me, gave me the case number. The man at that end was sympathetic to the problem, sympathized and said to turn on any questions. It's nice when the company has a human face.
To be continued...
From the useful, I have already enabled two-factor authorization wherever I could. I tormented my MacBook with various antiviruses that did not find anything.
About 5 days have passed, while there is no news, the amount of debt hangs behind me, April 4, settlement day ...
upd. There is a suspicion that the hacking occurred after an attack on the Bitrix website, in which the S3 storage was connected and the API key was specified, because a day before the hacking in the productive protection module it was written that 314K hacking attempts were reflected, and 1800 forms were filled out on my website feedback :(
upd2. The guys from Bitrix support analyzed the site’s logs. There was an attempt to attack, they tried to pick up passwords for the admin panel, use SQL injection and XSS vulnerabilities, but there was no confirmation of breaking.
More API key was not used anywhere ... Amazon is silent for now.
upd3. The issue is resolved; as a one-time share, I agreed to write off the debt. Respect to the guys from Amazon :)
I asked them for logs with IP addresses. I still want to understand if they hijacked a password or API keys. I have static IP addresses both at home and at work, and by the logs I can establish the first unauthorized entry.
