DEV Community

Cover image for Tens of thousands of $$$ lost because of a simple bug! How to tackle failures as a dev
kethmars
kethmars

Posted on

Tens of thousands of $$$ lost because of a simple bug! How to tackle failures as a dev

Tens of thousands of euros. That's the estimated cost of a bug I caused because of a missing parameter.

When I got my first professional job as a developer, I promised myself to work hard and learn as much as I can. And so I've done, I believe. During my almost-4-year career in Pipedrive(the company I work at), I've faced many interesting challenges of which some have truly made me think about what it means to be a developer.

Oh bug

The team I joined is responsible for our public web - the gateway to all our new potentials customers. If it's down, people can't sign up, meaning no money for us. During my career in Pipedrive, I've managed to halt it a couple of times, mostly because of some unexpected problems. That said, there's was an incident in 2018 that was fully my fault as it could've been easily avoided by some extra manual testing.

The part of the code I was developing did not have any unit tests at the time, so we were relying on manual testing and monitoring tools. I do not remember the technicalities but it was a really simple improvement connected to tracking snippets on a low-importance page. As there were no errors in our monitoring tools after deployment, I happily went on with my life. The next day I found out my code had caused one of our most popular landing pages to be down for the whole night.

After a short investigation, I was able to find the bug and deploy a fix. Great! We even found some misconfigurations in our monitoring tools. Life went on for others, but not for me. I knew that if I had done some manual testing in a live environment after my deployment, we could've avoided the escalation of the bug.

Am I a developer?

I've never been afraid of admitting my mistakes, but that one time I felt shame. It was my mistake that easily could've been avoided. Some of the questions I asked myself:

  • How could I miss such an easy thing?
  • Why didn’t I test more?
  • Why do I always fail to notice things like that?
  • Am I even a solid developer?
  • If I make mistakes, do I even belong here?

At home, I started digging into the topic of how developers deal with failures. Quite soon, I discovered that imposter syndrome, a feeling that we've fooled others into thinking we're more competent and talented than we actually are, is a real thing among developers[1]. Even David Walsh, creator of a blog most web devs have probably visited, has written about it.

To be honest, I'm not surprised that big names have mentioned the topic, because as developers

  • We must update our knowledge all the time
  • There are new shiny technologies emerging every day
  • Your code is constantly tested for speed, quality, etc
  • There's almost always someone better around you
  • What we build can become complex systems that must keep functioning at all times
  • ...

What I'm trying to say is that the world of development is full of rapid changes and constantly performing at your best is hard. In the end, it's natural that bugs will appear. But it's important to learn from them.

Let's get back on track

In my case, the key takeaways from the incident were less technical but more psychological. As time has passed, I've dug more into the topic of self-care as a developer. Here are some of the points I've come to believe during my short career:

  • It's completely ok to make mistakes. If they're costly, it's bad. But they're still one of the best ways to learn.
  • If you do produce a bug or incident, give 110% of yourself to understand what went wrong and what could be improved (in my incident's case, we created a task for unit tests, improved our monitoring and I got a lesson to never feel too comfortable).
  • When you make a mistake and lose your confidence, go tackle some simpler problems. They'll help you get back on the track.
  • Test, but remember - you can never test too much. The question is - do you have the need and resources to chase perfection? How critical is your system?
  • As a developer, you must often dive into the unknown. Coming up with solutions in an unknown environment is something worth admiring.
  • You can never know everything. You should find the confidence that in case of need, you'll be able to learn a subject.
  • There will always be people better than you. And that's a good thing - you can learn from them.

You will produce bugs

Potentially tens of thousands of euros lost because of my bug. Yes, it could've been avoided easily and I made a mistake. But it also showed us where we can improve our monitoring systems, unit tests, etc. My colleague asked me if I learned anything from the experience. After replying positively, he said that in that case, it's worth the money.

To conclude, I'd like you to remember that producing bugs is a normal sight in a developer's life. You must be a professional and try to give your best but in the end, if mistakes do happen, learn from them. And while at it, try to remember that it's actually quite awesome what you're doing. We're not machines, we just talk to them.


Btw, if you do feel insecure about yourself as a developer, then I definitely recommend you to read Amy Cuddy's Presence. Or just watch her Ted Talk. She tells her own story but also gives practicals tips on how to overcome the feeling.

References:

[1] I Don’t Deserve to be Here: Presence and the Impostor Syndrome, Amy Cuddy


Interested in topics related to growth mindset, developer lifestyle and webdev tutorials? Subscribe to my Youtube channel called developerHabits

Other channels
πŸ“· Youtube: developerHabits
πŸ‘‰ Twitter: https://twitter.com/developerHabits
πŸ‘‰ IG: https://www.instagram.com/developerhabits

Top comments (2)

Collapse
 
sargalias profile image
Spyros Argalias

Good article, thank you. Some very key points. I particularly liked the points that you will sometimes make bugs, try not to feel down about it and always go back and understand how you could have prevented it.

Collapse
 
kethmars profile image
kethmars

Thank you for the feedback, I'm glad you liked it. It was easy to write out those points but at least in my case, hard to do in practice