Imagine your web application goes down in the middle of the night. It’s 2 AM, but your business is global. You have users in every time zone. They’...
For further actions, you may consider blocking this person and/or reporting abuse
This should be required reading for anyone involved in software. So glad to see someone else experienced this. Literally everything thing you've described happened to me in my lat job.
When it happened to me I actually called myself a firefighter rather than a hero though. I was actually promoted partly because of my firefighting ability which seemed odd to me. I totally agree that if you gain notoriety fighting fires then suddenly there's not much motivation to try preventing them and when there's no fire to fight you feel useless. In the end I wasn't able to affect the kind of changes I wanted as a manger as I didn't have the stomach to fight the CTO (who was also a very intelligent and stubborn firefighter) and I'll admit I enjoyed the feeling of being depended on as much as I hate the stress of the problems themselves.
Sorry for the essay just wanted to say how much this resonated with me and confess my sins : )
I used to call it firefighter too!
One time at a stand-up meeting when I, once again, had to tell everyone I was solving a crisis (putting out fire). I asked the VP to buy me a fire fighter hat so that I don't have to come to these meetings during a crisis. I will wear the hat and everyone knows no to bother me :D
I love that we almost have the exact same story!
One issue I've found is convincing CTOs and VPs of engineering who love heroes that they should aim for a different engineering culture. It's something that's hard to argue away with logic and even harder to prove.
I think it takes your own time to prove. Once you are in this kind of environment, you are required to put in extra time outside of your work duties to bring it into the team. (ex-Amazon, startup) work.
Since if you are "wasting time" fiddling with infrastructure, why are you not doing actual sprint work.
This is a great article and it resonates with me as well.
One additional aspect I would add: if you are looked on as the hero, management starts to expect everything to be solved as though it were a crisis. Even new features are now presented as crises. This lends itself to clever, but unmaintainable code. It's clever because you are smart. It's unmaintainable because you are in hurry-up-and-duct-tape-it mode instead of spending time to design.
Followed to its logical conclusion, the code you develop is a fragile ball of mud and the job is unbearably stressful. Worse, a new dev from a more-healthy environment hires in, and shines a light on the fact that your code is crap. You can't believe it because your hero experience leads you to the opposite logical conclusion about your skill.
It's a setup for failure of the software as well as personal failure for the hero.
That's an interesting perspective. I've usually seen heroes, including me, suffer professionally mostly because we spend so much time fighting fires that we don't have time to actually build new features. We end up letting that skillset rust. What you say makes a ton of sense though.
This is so true. We are working hard with dev.to to massage out any hero need that currently exists and set ourselves up to avoid this situation as much as possible in the long run.
Awesome, this kind of change is definitely an institutional mindset. Cheers to happier devs in the long run.
Its always a sad when you propose to setup proper monitoring, log systems only to be shutdown by management to spend time on more revenue generating projects.
Another tiny note from experience:
Being the hero won't make you as valuable as you might think. A management who doesn't invest into avoiding crisis is obviously unexperienced or incompetent and will not hestitate to discipline or even Fire you if your criticism about the sources for the crisis gets more Intense.
Thanks for writing this excellent piece. As a product manager, I relate a lot to the persona for "the developer reliant on the hero to save the day." Being reliant on heroes is a common position for a product manager to be in.
Beekey, what can the work prioritizer (product manager, business analyst, scrum master, etc.) do to enable multiple crisis-solvers on the team? Say Bob is our hero and Julia is our mid-level dev. Should we just have Bob and Julia tag-team the crises, or should we attempt to give some of Bob's normal non-emergency work to Julia, too?
Definitely have them pair in the crisis. That's the best way to learn.
Also make sure to do post mortems of every crisis. Talk to Bob about letting Julia provide suggestions for improvement instead of speaking right away. That'll help Julia learn as well. Create action points, and possibly stories, for crisis prevention. Treat doing at least one of those tasks as important as handling the crisis itself. This will reduce the need for crisis management.
Let me know if there are more specific situations you'd like to discuss.
" If there are known stability issues with an application being live, then they should be addressed"
Indeed. This is the real issue. Your software should be fireproof, you should not rely on having a fire-fighter at hand.
Excellent read, I'll pass this one around the office for sure.
Don't they just call this "DevOps" now?
You can end up with the same problem if you have one person doing DevOps. Everything with prevention in software development also applies to sysadmin work.
Sorry. I was just making a statement about how this practice has been institutionalized. I don't consider System Administration to be the same as DevOps. Why else would it have a different name? Back in my day we called it "integration engineering". It was a process without heroes.
I inherited an organization that not only had heroes, but celebrated heroic efforts. One of the hardest things to change in the culture has been reliance on specific heroes, and the general sense of helplessness felt by everyone else without those heroes in the room. Fortunately, the heroes are tired of being heroic, and once given the space to implement the kind of systemic change that is required, have done so with enthusiasm. One of them sent me this article, as he was taking some time off to visit friends. We are still working on the powerlessness/helplessness in the rest of the org, which has dropped significantly in the teams that have started to own their quality and operational integrity. We have a ways to go, but I'll take any victory. It's not the kind of 'bottom line' progress that gets celebrated in the board room - it's the kind of progress that leads to sustainable bottom line success.
Thanks for putting so well into words something that has bothered me about some software team cultures.
And if you don't remove the hero, they become the villain...
Spot on, everyone has to be a hero from time to time, but you have to learn from a crisis and implement long term solutions.