DEV Community

I was billed for 14k USD on Amazon Web Services 😱

Juan Manuel Ramallo on July 03, 2018

We may agree that we lie to ourselves once in a while. I remember thinking of: I'll never put my code on a public repository since it's a freel...
Collapse
 
yechielk profile image
Yechiel Kalmenson • Edited

Wow! Scary story!

To GitHub's credit, I know that if they detect that one of their own keys were uploaded to GitHub they'll notify you immediately and disable that key.

I know that because... err a friend... made that mistake once 😳

Collapse
 
ghost profile image
Ghost

Did AWS Support cancel the charge or refund you, or something? I've heard they're pretty understanding and often do stuff like that if you weren't responsible or it was accidental and there was no gain?

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

Yeah they were so reasonable.. they issued a refund request for me so I didn't had to pay anything I didn't use. They called it "unauthorized usage". And the communication with the support team was very gently and agile

Collapse
 
ghost profile image
Ghost

Sounds like they really understand what happened - how awesome. The only other CS that I could think of that is that lit 🔥 would rather be Spotify (@SpotifyCares) or Slack (@SlackHQ).

Thread Thread
 
tarzan212 profile image
tarzan212

Netflix is pretty chill on payments issue as well. At least for me, they allowed me to watch and pay later when my credit card got compromised! Not related to the subject, but I felt that they should have their name mentionned here :)

Thread Thread
 
okolbay profile image
andrew

for them 10(20,30?) bucks of delayed payment is less than nothing - recommendation from happy customer - priceless )

Thread Thread
 
iambalajirk profile image
balaji radhakrishnan

Nice story :)
AWS knows before the hacker that your keys are compromised. They are running a similar script/program to check whether your keys are publicly exposed.

Thread Thread
 
xtofl profile image
xtofl

So you know that for a fact? Or are you guessing?

Thread Thread
 
danielcrabtree profile image
Daniel Crabtree

I've found Slack (@SlackHQ) support to be awesome too.

Thread Thread
 
iambalajirk profile image
balaji radhakrishnan

I have seen instances like this before where some guys would have exposed their repo publicly with the keys and immediately started receiving emails from AWS that their keys are compromised. I am guessing the fact they would have done it the same way the hackers are doing it.

Collapse
 
lalitakashyap profile image
Lalita Kashyap

Hey! I am a student and I have fallen in a similar problem and got a bill of 1.5k dollars and need urgent help. Can you please tell what did you write in your support message. Please help me.

Collapse
 
ben profile image
Ben Halpern

Wow........... A true horror story........

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

Sure it was ☠️

Collapse
 
ejeker profile image
Eric Jeker

Ahah, I just got billed an insane amount because I forgot to remove one table after a DynamoDB tutorial I followed. Just contacted the support. I hope I won't have to pay this.

I thought I was on a On-Demand pricing but I had reserved WCU. Still don't really understand how they came up to that amount.

Collapse
 
djviolin profile image
István Lantos • Edited

For safety measure, create a Budget under Billing in your AWS web console with 2 factor authentication! This will set the maximum how much they can charge you monthly (or quarterly, yearly). More info about Budgets and it's types (be advised that some budget types only trigger notifications, not service halt): docs.aws.amazon.com/awsaccountbill...

This should be your first thing when you register on AWS.

Collapse
 
mkschueler profile image
Mike Schueler

budgets don't put a hardcap on your spend/usage.. it just sends out alerts based on thresholds you set.

you mention 'some budgets only trigger notification, not service halt' I'm not aware of any budget that halts service, please enlighten me

Collapse
 
johand profile image
Johan

That's why I also tend to ignore files in .gitignore_global so if I forget to do it from the .gitignore of the project I can avoid things like this.

Collapse
 
erebos-manannan profile image
Erebos Manannán

I just don't understand why it's such a big deal to actually look at what is being committed before doing it

Too many people do some sort of git add . && git commit -m "progress" && git push blind commits and it baffles me how they feel comfortable doing it.

Collapse
 
xngwng profile image
Xing Wang • Edited

few more ideas:

Never put your api keys in repos. period. (public or private).

Put them in environment variables but keep them completely separate from the source code repo. like development.env, production.env.

Always follow the practice of least privilege. Even most engineers at a company should only need
development.env variables if they are just writing code and not doing production support.

production.env should be guarded with only very few people that must have access to production system.

Collapse
 
kayis profile image
K

I had the impression KMS is the way to go here.

Put the encrypted keys in the repo and decrypt them before usage.

Collapse
 
mbuechmann profile image
Malte Büchmann • Edited

I do not think that this is a good idea. You would put encrypted keys and the decryption algorithm in the repo. It is still possible to get to the keys.

Thread Thread
 
Sloan, the sloth mascot
Comment deleted
 
mbuechmann profile image
Malte Büchmann • Edited

First of all: The named function hash_hmac does not encrypt. It creates a hash, which cannot be used to restore the original value. It is one way.

If you would use a proper encryption the logic is still faulty.

You have a secret, that cannot be added plainly to the repository. You add some decryption logic, encrypt the original secret and add it to the repository. The original secret is now safe. But now you have another secret (the private key needed for decryption) that cannot be added plainly to the repository.

You still have the same situation plus some extra decryption code, which has to be maintained. Also your build process has to handle the encryption.

Collapse
 
bgadrian profile image
Adrian B.G.

The worst nightmare of a cloud user coming true.

But my keys are in the .ssh home folder, protected, why would you put them in configs? What kind of awful deployment tools requires that?

Also, the first thing I do (and AWS recommends) is to setup billing alerts, at least you know that something bad is going on.

Collapse
 
databasesponge profile image
MetaDave 🇪🇺

Seconding the advice to set up billing alerts, and I would make sure it goes to a variety of email addresses in case one of them is compromised.

Collapse
 
numerxofficial profile image
Deak Kevin

also setup a Budget w/ 2fa

Collapse
 
empty2k12 profile image
Gero

Your account was not hacked. You sent out the password (yes, access keys are passwords) to the world. You should consider yourself lucky AWS refunded you the money, as technically you are responsible for those charges.

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

lucky me I didn't had to pay for my mistake 😅 lesson learned anyways 😇

Collapse
 
theodesp profile image
Theofanis Despoudis

Maybe this will help next time:
github.com/awslabs/git-secrets

Collapse
 
joelebeau profile image
Joe LeBeau

Just in case no one else mentioned it, if you mistakenly commit a secret to a git repo, you need to make sure to get it out of the git history as well. help.github.com/articles/removing-... If you are curious if a git repo has had secrets accidentally committed to it in the past you can use tools like TruffleHog to search for them github.com/dxa4481/truffleHog

Collapse
 
kjir profile image
Stéphane Bisinger

You need to consider the secret compromised, and you hate to revoke that key. Cleaning up the history is then a lesser concern...

Collapse
 
putopavel profile image
Pavel Razgovorov

I experienced a similar story while I was doing a group assignment in college. They charged me 800€ for absolutely nothing. Hopefully I contacted the support centre and hopefully they retired all those charges

Collapse
 
miguel456 profile image
Miguel Nogueira

Same. I was only owing 300€ but it was scary
they were pretty fast in resolving the issue, hats off to AWS customer support

Collapse
 
andy profile image
Andy Zhao (he/him)

Yikes! I've heard that there are bots that scour GitHub for new public repos, seeing if there are any public keys for them to steal. Guess it's true. 😱

Collapse
 
ciantic profile image
Jari Pennanen

If your account was hacked, even the spending limit (which AWS does not have) system would not work.

But in Azure you can set a spending limit:

"When your usage results in charges that exhaust the monthly amounts included with your subscription, the services that you deployed are disabled for the rest of that billing period."

And I think all other cloud services should have it, if you are running hobby servers and one of them is hacked. You are screwed with these non-limited services. At least with Azure you can set a spending limit.

Collapse
 
exbe profile image
exbe

What about budgets on AWS, you can set it there to limit your actual spending.

Collapse
 
mkschueler profile image
Mike Schueler

budgets is nothing more than an alert

Collapse
 
perttisoomann profile image
Pert Soomann • Edited

I remember story few years back when someone's blog or app went viral unexpectedly, so it auto-scaled up infrastructure to $5,000 worth of AWS bills. Downside of zero downtime :)

Collapse
 
defman profile image
Sergey Kislyakov

5000k or 5k?

Collapse
 
perttisoomann profile image
Pert Soomann

Ah yeah, it's 5,000 not 5,000K indeed. Good spot :)

Collapse
 
t0d0r profile image
t0d0r

There is a very nice tool, because "git rm filename" is not an option, the key remain in the git changelog/history, you can use BFG:

brew info bfg
bfg: stable 1.13.0
Remove large files or passwords from Git history like git-filter-branch

Collapse
 
miguel456 profile image
Miguel Nogueira

Once I was owing AWS 300$... For something I didn't use and had accidentally forgotten.
It was a forgotten Certificate Manager cert authority. Fortunately my account wasn't compromised!

And there I was, literally crying over 300$, when someone came to me and said, hey, call AWS support

And so I did, the story went smoothly, they were quick and really understanding and finally pardoned my debt... Lol!
I was really grateful that day, a guy from AWS support literally saved my ass. I'm glad you didn't have to pay the 14k and that all went well. I wouldn't know what do do in that case!

I wonder what the hackers were doing with EC2 to owe 14k right away... It must've been a gold pot for them hahahah

Collapse
 
eli profile image
Eli Bierman

What a great story. Glad it worked out OK. I made the same mistake once, and I guess AWS had the same script running because they found my key on GitHub and notified me to de-authorize it before it could be abused. I was very grateful that they were that much more professional than I was :)

Collapse
 
kjir profile image
Stéphane Bisinger

Did the same mistake, pushed just before dinner, while eating I realized I pushed my secret key to a public repo and when I was finished my account was already compromised. I had to manually remove all the garbage, but in those 15-20 minutes I already got a $ 150 bill.
Lesson learned: security as the first thing, even for toy projects. Always.

Collapse
 
djkad profile image
Marcello • Edited

I had a similar situation with AWS free tier plan. I've received a bill because i made some mistakes doing configurations with some instances. They helped me resetting the bill amount and they also recharged some credit too in order to extend the trial period. It has been a very pleasure talk with the service support by phone.

Collapse
 
david_j_eddy profile image
David J Eddy

:) Lesson learned, glad AWS did not actually charge you the 14k. Checkout this resource for some best practices when it comes to app development (language agnostic): 12factor.net/. Super helpful and has saved me many times.

Collapse
 
markwinap profile image
Marco David Martinez

I had a similar situation 4 years ago and i know how it feels when you have a bill that reaches the sky. In my case the hacked my account and they spawned EC2 instances to mine bitcon. Thankfully AWS automatically noticed the hack and they fixed my bill.

Collapse
 
africasiaeuro profile image
Heinz Rainer

I cancelled AWS after 1 month. To hell with their cloud. Even though there was no activity they still kept on charging me.
Biggest problem is your credit card. Open season. Well no longer. I use a virtual card now, and I freeze it after each transaction.

Collapse
 
asciidude profile image
asciidude • Edited

Even though there was no activity they still kept on charging me.

If there is no activity you should not be using it.

Collapse
 
hoelzro profile image
Rob Hoelz

Yikes, that sounds awful!

I haven't used this, but I stumbled upon a piece of software a while back that looks for high entropy strings - I wonder if it would have picked up your committed credentials?

It was because of a fear for this exact situation that I switched all of my private credentials for my applications to being encrypted at rest; I wrote this tool to load credentials from a KeePass database into my shell environment, which my scripts and applications can pick up. It's not perfectly secure, but it's a step in the right direction!

Collapse
 
munamohamed94 profile image
Muna Mohamed

Oh my goodness, that's crazy 😱! That's one of my biggest fears 😩. Can't imagine what waking up to that felt like! I'm glad that it got sorted out 👍

Collapse
 
miguel456 profile image
Miguel Nogueira • Edited

How it felt like? Like the world falling down on you 😱😱

Collapse
 
leob profile image
leob

Thanks for sharing ... great response from Amazon! Never would have expected a big corporation to be so customer friendly (the stories from people trying to get any sort of customer service from Google, Facebook and so on are pretty negative).

Collapse
 
nikhil_gupta profile image
Nikhil Gupta

I heard that AWS team also scans public git repos for presence of AWS ACCESS Keys and notifies the owner of repo if they find one.

Collapse
 
grayest profile image
Grayest

The security part is fine but when it comes for refunds , AWS is not so helpful in my case. My a/c got hacked and was charged for around 8K USD. Weirdly AWS says and refunded 90% of the amount and for rest 10% they say should be borne by customer which is absurd (considering all the resources were used by hacker to spun the ec2s)

Collapse
 
feynmanfan profile image
Chris B. Behrens

I thought I'd drop you a line and let you know that I've used this story in a number of my courses to illustrate the simple truth:

Secrets do not belong in version control.

I always encourage the learner to have mad respect for somebody who admitted to an embarrassing mistake that everyone is likely to make unless they're educated out of it. Thanks for the article.

Collapse
 
hellinajully profile image
heliana jully

Necesita un pirata informático con experiencia y, a continuación, CONTACTE a alliancehacker999@gmail.com, brindan el siguiente servicio en 2 horas. Proporcionamos el servicio más rápido de cualquier trabajo de hacker. Con diversas habilidades y técnicas de pirateo, proporciona el mejor trabajo en cualquier trabajo de pirateo, y se especializa en lo siguiente

Cambio de notas universitarias

Twitters hackear

Hackear cuentas de correo

Cambio de notas

Sitio web se estrelló hackear

servidor pirateado

Recuperar archivos / documentos perdidos

Eliminar antecedentes penales hackear

Hackeo de bancos de datos

Hackear o skype

Sitios web hackear

Hack de Facebook

Hackear whatsapp

Hackear la intercepción de mensajes de texto

Interceptación de correo electrónico.

Hackers de tarjetas de crédito, limpieza de tarjetas de crédito, aumento de tarjetas de crédito.
Para obtener más información, puede comunicarse con él a través de alliancehacker999@gmail.cm

Collapse
 
mikaoelitiana profile image
Mika Andrianarijaona

So did you finally had to pay anything, like some penalty?

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

Not even a dime, they considered it as "unauthorized usage" and refunded me all the extra💰

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

Though they filled me up with documentation on how to strengthen my account and correctly use and storage of secret keys.

Thread Thread
 
mikaoelitiana profile image
Mika Andrianarijaona

That's really great support

Collapse
 
dzonibg profile image
Nikola Stojisavljević

Thumbs up for AWS guys.

Most of their services are quite expensive compared to other providers, but I've been nothing but happy with them. What I like the most are straight-forward answeres to anything I had to ask. Your question goes directly to a person that can really help you, no 2-3 day delays till you get a reply from a department in charge.

Collapse
 
bl3z3 profile image
Ir Nahayo Ndindayino

Waaoouh, same thing has happened to me in the past. accidentally left my sendgrid keys in a project i pushed. However, sendgrid in some way was able to detect them and automatically block my account. How? i dont know.
But i couldn't access my own project until i changed my password, key and enabled 2FA.

Collapse
 
ann0nip profile image
Juan Martin Gimenez

Uh menos mal que zafaste, con el dolar a 40 para 14,267.86 te queres matar xD

Collapse
 
kalinchernev profile image
Kalin Chernev

I'm honestly truly surprised by the light conclusion after the whole story. Are you serious?

Juan, for the g** sake, please put this link in the end of the article: aws.amazon.com/premiumsupport/know... (at least)

If the story is true and you see so many people paying a visit to your article, make some efforts to go a bit deeper than "Never leave your api keys on public repositories".

I mean, really - use the leverage of your story and influence. Add value instead of closing it as lightly as something so obvious when there are other and better ways, which btw are mentioned in comments which are not effective ...

Collapse
 
gregorys100 profile image
Gregory Sequeira

Appreciate the effort in sharing this with the community. I am preparing for an AWS cert right now and will be using AWS soon. Your post was helpful!

Collapse
 
alessandrostamatto profile image
Alessandro Stamatto

Most times it's a malicious person that spins your AWS for Bitcoin Mining... Malicious people love to steal computer resources for gaining some Bitcoin buckets =p

Collapse
 
bbcat profile image
BBCAt

I just got the same problem as yours? I only used S3 from AWS for a small personal project. I used it for 1 month. I got an email from AWS and got charged $1700 for Elastic Compute Cloud. I already submitted the case for the AWS Support and not so sure when they are going to reply to me because I do not live in the USA . Do you know, in the worst case, what could happen if I do not pay the bill? Is it related to legal stuff?

Collapse
 
imvjsai profile image
Vijaya Sai Nallani

I have the same query. May I know the actions you have taken so far please? It will help us.

Collapse
 
imvjsai profile image
Vijaya Sai Nallani • Edited

My friend is in a similar situation now. His account got hacked twice. First-time AWS waive off the bill (4000$) and 2nd time (got hacked within 10 days) got a bill of 1000$, but they revert as it's the customer's responsibility. Can you please advise if he keeps his AWS account untouched, will it close automatically, and not need to pay the bill?

Collapse
 
maxmaxymenko profile image
Max Maxymenko

Thanks for sharing your story. Glad it all worked out fine for you ;)

Collapse
 
pau1rw profile image
Paul

I had a bill for almost £5000, absolutely terrifying!

Collapse
 
jrevillas profile image
Javier Revillas • Edited

Did you request a limit increase in the past for EC2 services?

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

TBH, at that time I didn't even know about EC2 services 🤷🏻‍♂️

Collapse
 
moonbury profile image
Keith • Edited

Thank for sharing your experience and good to see AWS is very concern.

Collapse
 
ankuagar profile image
ankuagar

Yes AWS support people are very understanding and nice. Personal experience.

Collapse
 
tcelestino profile image
Tiago Celestino

I have not remembered what the name, but there is website list many public repos on Github has access keys of services in the project.

Collapse
 
claudiordgz profile image
Claudio Rodriguez

This is horrifying, glad Customer Service was able to solve the breach for you.

Collapse
 
meanin profile image
Paweł Ruciński

Nice experience with AWS. On the other hand GCP ;)

Collapse
 
devopsdymyr profile image
devopsdymyr • Edited

Please help me out the same issue happen for me today but the charge is $19K,
some hacker hosted everything in every region.

Collapse
 
juanmanuelramallo profile image
Juan Manuel Ramallo

Send an email to AWS support, they will help you out.

Collapse
 
omawhite profile image
Omar White

This makes for a great story to stress the importance of secrets management!

Collapse
 
lalitakashyap profile image
Lalita Kashyap

Hey! I am a student and I have fallen in a similar problem and got a bill of 1.5k dollars and need urgent help. Can you please tell what did you write in your support message. Please help me.