DEV Community

Cover image for How We Accidentally Declared War on Our Own Database

How We Accidentally Declared War on Our Own Database

Boopathi on February 15, 2025

It all started with a simple task: "Hey, let’s check if two databases have the same data over time!" Sounds harmless, right? Oh, how young and fool...
Collapse
 
safin_ahmed_9add049c2af82 profile image
Safin Ahmed

That's one hell of a story dude

Collapse
 
programmerraja profile image
Boopathi

Yep. Wait for upcoming similar incideint where we accidently deleted our production cluster

Collapse
 
dansasser profile image
Daniel T Sasser II

😳👀

Collapse
 
safin_ahmed_9add049c2af82 profile image
Safin Ahmed

omg, gotta hear this. giving you a follow.

Thread Thread
 
programmerraja profile image
Boopathi

Thank you :)

Collapse
 
bakir_claude01_4566c2125 profile image
bakir claude-01

right

Collapse
 
stanleysathler profile image
Stanley Sathler

What a story, haha! If we think for a second, it actually makes total sense for such command to lock the database, but we never think these things when we have to 🤣

I'm curious tho - why you had to check if the two databases had the same data?

Collapse
 
programmerraja profile image
Boopathi

We doing some migration. after migration we would like to check data consistency between two db

Collapse
 
thecodingbam profile image
TheCodingBam

Lol 😂.... Like what the hell for? 🙄 😏 😂

Collapse
 
thatnoobdev profile image
NoobDev

Nicely written! I am always scared of database migrations, so because of this paranoia, I double-triple check documentation to make sure everything is correct or not, and also have some backup plan ready in case something goes wrong. LMAO!

Collapse
 
brense profile image
Rense Bakker

Well written hahaha, I can feel the pain and the stress of the mongodb expert 😁

Collapse
 
programmerraja profile image
Boopathi

😂

Collapse
 
nht_trngnguynh profile image
Nhật Trường Nguyễn Hữu

8 hours of coding and crying saves 5 minutes of reading the document.

Collapse
 
ddhruviot profile image
Ddhruv

Tip: Even experts are not trustable!

Collapse
 
bakir_claude01_4566c2125 profile image
bakir claude-01

Yes That's one hell of a story dude

Collapse
 
shricodev profile image
Shrijal Acharya

The dbHash command locks the database and doesn’t allow any changes until it finishes.

Didn't know this stuff. Thanks for this.

Collapse
 
programmerraja profile image
Boopathi

Your welcome :)

Collapse
 
emtesta profile image
Emiliano

Ah! Being young and stupid. We all have to go through one of these "oh crap!"moments.

Well done on finding out what, to me at least, seems obvious :)

Collapse
 
shmuelko profile image
Shmuel

Thank you for sharing

Collapse
 
fstrube profile image
Franklin Strube

👏 Yikes! Glad your team was able to recover. That would give me a heart attack!

Collapse
 
mbienvenu profile image
mbienvenu

Good share.. expecting much more content from you

Collapse
 
programmerraja profile image
Boopathi

Thank you for your kind words. sure will share more my experience

Collapse
 
swati_mishra_d6a3910f4bab profile image
Swati Mishra

Awesome story. 🤭🙌

Collapse
 
srikanthkapila profile image
Srikanth Venkata Kapila

This is crazy. Never seen someone do this on a production database. Hope the senior fellow had a good laugh after some time

Collapse
 
programmerraja profile image
Boopathi

Yeah we have lot of fun and learning after that 😁

Collapse
 
kwnaidoo profile image
Kevin Naidoo

Interesting, very well written! Reminds me of dark days 😃. Had my fair share of DB servers go down, most were related to hardware failures or scaling issues and legacy DBs.

Collapse
 
madhurima_rawat profile image
Madhurima Rawat

That’s quite a story! I really loved the way you wrote it, though. Also, thanks—I learned a new command from this.

Collapse
 
michael_phillips_356cb1ff profile image
Michael Phillips

Accidentally locking up your entire production database, that's got to be every engineer's worst nightmare.

Collapse
 
i4nstigator profile image
ianstigator

"Change management" is the concept your company is looking for.

Collapse
 
noxyucm profile image
Trumps Ear

Too much useless (pathetic) code in this article.

Instead of a simple: "RTFM before executing something unknown in prod env".

I didn't like the presentation.

Collapse
 
programmerraja profile image
Boopathi

Thank you :)

Collapse
 
heyakhil profile image
Akhil Srivastava

Any suggestions for serverless database of MySQL

Collapse
 
programmerraja profile image
Boopathi

Sorry I don't not have expereince with MYSQL

Collapse
 
humayon_khan_0a0633f0665e profile image
Humayon Khan

Hey,

Its a great read, being a DEV head keeps spinning and reading this was something soothing to relax and joyous for a few mins.

Collapse
 
programmerraja profile image
Boopathi • Edited

Thank you for kind words.
below few articels that will also give you the same feeling

Collapse
 
shawn_monteiro_1a016fa395 profile image
Shawn Monteiro

Damn that was a roller coaster. But comparing the hash was a curious idea that had to be tested 🙃.

Collapse
 
programmerraja profile image
Boopathi • Edited

Yeah that's what he did in live DB :)

Collapse
 
adityarpai843 profile image
Aditya R Pai

I was having a stomach pain while reading. BTW Good narration

Collapse
 
thecodingbam profile image
TheCodingBam

Lmao 😁 😂

This made my day 🌹 😆

Collapse
 
saurabh_rawat_6af9d43bf91 profile image
saurabh rawat

🫢🫢🫢