Don't Let a Bad Deploy Teach the Wrong Lesson

Every incident teaches lessons. People want to know why it happened and how to prevent it from happening in the future. This often results in a “lessons learned” session to try to capture these learnings. However, not all lessons are good ones. Incidents can teach the wrong lessons and reinforce bad practices if care isn’t taken with these lessons. Ensuring that the right lessons are learned and that they reinforce, rather than contradict, good engineering values is one of the more important parts of technical leadership.

A few years ago I was contracting on a large enterprise system. I oversaw several teams, including the platform team that owned deployment pipelines. That team was drowning in operational toil, and a large share of the toil was deployments. The system had been sold as microservices, but it behaved like a distributed monolith. Several tightly coupled subsystems had to get deployed together, with the same versions, in the right order, and getting that order right took enough undocumented knowledge that a single release usually consumed most of a day and most of the platform team. They were the only people in the building who could ship the product, and shipping the product was eating them alive.

The platform team was actively trying to move towards self-service deployments. In whatever time they could snatch between operational toil and firefighting, they had been working to document and automate deploy procedures. It didn’t fix the architecture of the application, but it would allow dev teams to deploy code when they were ready. This would both remove the platform team as a bottleneck and lower their operational load. The goal was for the development teams to ship their own code safely, with the platform team providing the paved road underneath them: the guardrails, the documentation, the observability, and above all a recovery path for when something went wrong. Self-service was supposed to be a product the platform team offered, not a chore they handed off.

Eventually the platform team felt the road was ready, and a developer from one of the application teams volunteered to trigger a production deploy. This was the moment we had been working toward. For the first time, someone outside the small circle that held all the hidden knowledge was going to ship the system to production.

Production went down.

The good news, and it matters, is what happened next. The alerting fired the way it was supposed to. The rollback path worked. The team recovered quickly, and the impact stayed small. The change was rolled back before it could do much damage, and prod was back to green very quickly.

The customer’s reaction was immediate and completely understandable. This, they said, is exactly why only the platform team should be allowed to deploy. You let an application developer touch production and production fell over. The lesson wrote itself. This is a deeply human reaction: when something breaks, the reflex is to pull control inward, to shrink the set of hands that can touch the dangerous thing. It feels like safety. It is also a deeply flawed lesson in this context.

I argued the opposite, and not just because I wanted to defend the developer. The developer didn’t deserve anything but praise. He voluntarily stuck his neck out to help move the system forward, and the system failed him. That wasn’t a him problem. I didn’t know him, he didn’t work for one of my teams, but I still argued he needed to be thanked, not have his hand slapped.

I also argued it because the lesson the customer had drawn was wrong on its own terms. Bad deploys are going to happen. Centralizing them does not prevent them. The platform team had been the only ones deploying for years, and the system had gone down plenty on their watch. The question an incident like this actually poses is never whether you can prevent a bad deploy. It is who can recover when one happens. A team that ships its own code and knows how to roll it back will have far less downtime than a team that has to escalate every fix into a queue behind one overloaded group. The ones who wrote the code are far more likely to know exactly what went wrong and be able to fix it quickly than a platform team with zero context about the features they are deploying. Pulling deploys back to the platform team would not have prevented the next outage. It would have made it longer.

And the developer, by stepping into a model nobody had stress-tested yet, had handed us a list of exactly the things we still needed to fix. He surfaced an undocumented dependency that the platform team carried in their heads and had never written down. He surfaced a gap in the release documentation. He surfaced a procedural gap of our own making: for a team’s first self-service production deploy, the platform team should have been pairing with them, and we had let him go alone. Every one of those was a real finding, and not one of them was his fault. They were the unfinished edges of the platform, and the only way to find them was to put someone who did not already carry the hidden knowledge in front of the thing and watch where it failed them.

He also proved something we needed to know and showed it in the most compelling way possible. The alerting worked. The rollback worked. The system could take a bad deploy and come back from it fast. A self-service path that everyone is afraid to use is not actually finished, and we had just watched ours absorb a failure and recover in front of us. That is evidence the platform was working. It is not evidence that it should be taken away.

This is the reframe I keep coming back to. The failure was not that someone outside the platform team deployed. The failure was that our platform still required hidden knowledge to deploy safely. If the only people who can ship without breaking things are the ones who have memorized an undocumented order of operations, you do not have a platform. You have a priesthood and a bus factor, and you are one resignation away from a crisis. The whole point of the work we were doing was to get that knowledge out of a few people’s heads and into the road itself.

Incidents have a gravity to them, and the easy lesson is almost always the one that pulls toward less change. “Let fewer people do the risky thing” feels like control, and it has the great advantage of requiring nothing new from the platform team. The harder lesson is that the risky thing is still too hard to do safely, and that fixing it is on you. The first version is a retreat. The second is the next item on the roadmap. This is really just what a blameless postmortem is for: you treat the human action as a symptom, not a cause, and you go looking for the conditions that made it the reasonable thing to do. The developer pushed a button he had been invited to push. The interesting question is why the system behind that button still had a trap in it.

There is a comfortable bit of folk wisdom that says you trade speed for safety, that the road to reliability runs through slowing down and letting fewer hands near production. The research has been pointing the other way for years. High-performing teams do not get stability by making deploys rare and ceremonial. They get it through small changes, fast feedback, real guardrails, automation, and teams that own their work all the way into production. Speed and stability are not opposite ends of one dial; when the system is built well, they come from the same practices.

Centralizing deployment gives up the speed without buying the safety. It is the worst of the trade, dressed up as prudence. The safest deploy is not the rare giant release. It is the small, well-observed, reversible change.

So we did not take deploys away. We wrote down the dependency we had found. We fixed the release docs. We made pairing a standard part of every team’s first self-service production deploy. The next team to ship did not hit that wall, because the wall was gone. The bad deploy did not prove that developers should not deploy. It proved that our self-service path was not finished. By failing, it had given us a lesson that showed us where it still needed work.

Every incident is going to teach your organization something. Make sure you learn what it actually has to teach, and not the version that is easiest to hear.