It all started with a simple task: "Hey, let’s check if two databases have the same data over time!" Sounds harmless, right? Oh, how young and foolish we were.
Our team did some research (aka Googling) and found a potential solution. That’s when our senior teammate our so-called MongoDB expert stepped in. We called him that because he had slightly more experience than the rest of us.
Anyway, he found The Command a magical piece of code that would generate the hash for the collection
db.runCommand({ dbHash: 1, collections: [] }).md5
He was thrilled! Excited! Practically bouncing in his chair! His master plan? Run this command on both databases on each collection, compare the hashed, and if they didn’t match boom! Data inconsistency detected!
First, he tested it on a small database. It worked. Beautiful. Perfect. No issues at all. So, naturally, he did what any overconfident engineer would do:
"Let’s try this on the live database with 200 million records!"
This, my friends, was the moment it all went horribly, terribly wrong.
The Moment of Doom
At first, everything seemed fine. He ran the command. He waited. And waited. And waited.
Then he got impatient and thought, Huh, maybe I should stop this command? So he tried to kill it.
Plot twist: The command refused to die.
Seconds later, all our database connections spiked to 2,900%. Every single query stopped working. The app crashed. Users were probably screaming at their screens.
At this point, chaos erupted. Alarms were ringing (in our heads). People were yelling. Our team was frantically Googling like hackers in a bad Hollywood movie.
But no matter what we tried, nothing worked. But to reduce the connection in mongodb we were stoped some not crucial service and we try to keep the connection as low as possible
Our one innocent looking command had completely paralyzed the entire database. It was like throwing one small rock into a river and somehow causing a tsunami.
The Horrifying Truth
Then, finally, our MongoDB expert (who, at this point, had aged 10 years in stress) decided to read the official documentation. And there, hidden like a villain’s evil plot, was this small but deadly sentence:
The dbHash command locks the database and doesn’t allow any changes until it finishes.
Translation: "HAHA, YOU CAN’T DO ANYTHING UNTIL I’M DONE. GOOD LUCK, LOSERS!"
At this point, we had two choices:
- Sit there and hope for a miracle.
- Find a way to forcefully wake up our poor, frozen database.
Luckily, our DevOps teammate had an idea. "Let’s scale up the database. That’ll force a restart and kill the stuck command!"
We nodded furiously and watched as he logged into MongoDB Atlas, cranked up the database size like turning up the heat on a broken stove, and hit the magic button.
The Resurrection
For the next five minutes, we sat there in complete silence, watching the screen, praying to the database gods.
Then BAM! The database restarted! The stuck command was gone! The app was alive again!
We cheered. We cried. We swore to never, ever, EVER run random commands on a live database again.
Lesson Learned:
Always read the instructions before trying something new. And maybe just maybe don’t test experimental ideas on a live system unless you enjoy heart attacks.
Finally, if the article was helpful, please clap 👏and follow, thank you!
Top comments (42)
That's one hell of a story dude
Yep. Wait for upcoming similar incideint where we accidently deleted our production cluster
😳👀
omg, gotta hear this. giving you a follow.
Thank you :)
right
Nicely written! I am always scared of database migrations, so because of this paranoia, I double-triple check documentation to make sure everything is correct or not, and also have some backup plan ready in case something goes wrong. LMAO!
What a story, haha! If we think for a second, it actually makes total sense for such command to lock the database, but we never think these things when we have to 🤣
I'm curious tho - why you had to check if the two databases had the same data?
We doing some migration. after migration we would like to check data consistency between two db
Yes That's one hell of a story dude
Tip: Even experts are not trustable!
8 hours of coding and crying saves 5 minutes of reading the document.
Well written hahaha, I can feel the pain and the stress of the mongodb expert 😁
😂
👏 Yikes! Glad your team was able to recover. That would give me a heart attack!
Thank you for sharing
Ah! Being young and stupid. We all have to go through one of these "oh crap!"moments.
Well done on finding out what, to me at least, seems obvious :)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.