Google’s site reliability engineers once watched their own automation erase the disks on every machine in their content delivery network. A decommissioning job for a single rack had failed and been restarted to debug it. On the second run, an empty list of targets was read as “everything.” In the words of Google’s SRE book, “Within minutes, the highly efficient Diskerase wiped the disks on all machines in our CDN.” Users barely noticed, thanks to spare capacity elsewhere. The engineers spent “the better part of two days” reinstalling machines by hand, then weeks adding sanity checks and rate limits to the automation.
That story usually gets told as a lesson about software. I read it as a lesson about people. The automation did its job at machine speed; the recovery happened at human speed, done by people who had to remember how to do by hand what the tooling had been doing for them.
A British psychologist named Lisanne Bainbridge put a name to this in 1983. Her paper, Ironies of Automation, runs five pages in the journal Automatica and is still the best thing a manager can read before approving an automation project. Below are the questions I hear most from managers whose teams are automating work, answered with what the research and my own operations years suggest.
What exactly is ironic about automation?
Bainbridge’s core point is that automating a task does not remove the human from it. It changes the human’s job, usually for the worse.
She names two ironies up front. The first is that the people who design automated systems make mistakes too, and those mistakes become operating problems. The second is the one managers should tape to their monitors: “the designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate.” The result, she writes, is that the operator “can be left with an arbitrary collection of tasks, and little thought may have been given to providing support for them.”
Look at what is in that leftover pile. It is the edge cases, the weird failures, the situations nobody anticipated. The automation takes the routine work, which was also the practice. The human keeps the rare, high-stakes work, with less practice than ever.
Her closing line is the real irony: “it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training.” The better your automation works, the less ready your team is for the day it doesn’t.
Do skills actually fade that fast, or is this theory?
Aviation has tested it more carefully than any other field, because it has the most to lose.
A NASA Ames researcher, Stephen Casner, and three colleagues put 16 airline pilots in a Boeing 747-400 simulator and published the results in Human Factors in December 2014 (full paper). The surprise was which skills held up. Hand flying and instrument scanning stayed reasonably strong, even among pilots who said they rarely practiced them. What decayed were the thinking skills: keeping track of where the aircraft was, recalling procedure steps, diagnosing a failure. When the flight management computer was taken away, only one of the 16 pilots worked through the full approach setup without an error. When faced with an unreliable airspeed indication, nearly all of them drifted toward a stall. The Flight Safety Foundation’s write-up of the study is titled, fairly, “Use It or Lose It.”
That pattern matches what I saw in network operations. The hands remember. The mental model goes first.
The US government has said the same thing on the record. In January 2016 the Department of Transportation’s Inspector General reported that the FAA estimates automation is used 90 percent of the time in flight, and that “pilots who typically fly with automation can make errors when confronted with an unexpected event or transitioning to manual flying.”
The case everyone cites is Air France Flight 447. On 1 June 2009, ice crystals blocked the Airbus A330’s airspeed sensors over the Atlantic and the autopilot disconnected. The aircraft was flyable. The crew, suddenly hand flying at cruise altitude in the dark with conflicting readings, pulled the nose up into a stall and never recovered. All 228 people aboard died. France’s accident investigators pointed to gaps in crew training for manual handling at high altitude and for unreliable airspeed situations. The automation did exactly what it was designed to do when its inputs went bad: it handed the problem back to the humans.
What does this look like on a team that doesn’t fly planes?
Google’s SRE authors draw the line from cockpits to server rooms directly. As automation covers more of the daily work, they write, “human operators are progressively more relieved of useful direct contact with the system.” Then the automation fails, and “the fluidity of their reactions has been lost due to lack of practice, and their mental models of what the system should be doing no longer reflect the reality of what it is doing.”
In my years running IT operations at a large telecom, the version I watched play out most often was rerouting. Early on, when a fiber cut took out a route, a technician worked the reroute by hand: they knew which circuits rode which path, which customers would scream first and what the backup capacity really was. Over time we automated most of it, and that was the right call. Restoration got faster and more consistent. But a few years later we had a failure the automation did not handle, a double fault where the protection path was already degraded. The people on shift were good. They had also never done a manual reroute on that part of the network, because the system had always done it for them. The person who knew it cold had moved to another group. We got it restored, slower than we should have, with a phone call to someone who no longer worked on that team.
Nobody made a bad decision. We automated something that deserved automating. We just didn’t account for what the automation was quietly taking away.
You will find the same pattern well outside of IT:
- Finance teams with automated reconciliation who can no longer explain why the numbers tie when an auditor asks.
- Customer support with a scripted triage bot, where agents only ever see the escalations and have lost the feel for what a normal ticket looks like.
- Scheduling and dispatch tools that optimize routes until the day the tool is down and nobody remembers how the board used to be run.
- Any team using AI drafting tools for first drafts, where the skill being bypassed is the thinking that used to happen while writing.
Should we stop automating, then?
No. Bainbridge wasn’t arguing against automation, and neither am I. Repetitive manual work burns people out and introduces errors; I’ve written before about toil and why removing it is one of the best things a manager can do for a team’s capacity. The argument is narrower: when you automate, you take on a new obligation to keep the humans able to step in.
Here is what that looks like in practice.
Decide what the human is for before you automate. Bainbridge’s complaint was that the operator ends up with whatever was left over. Flip it. Before the project ships, write down which situations the automation will hand back to a person, and what that person needs to know to handle them. If you can’t describe the handback, you aren’t ready.
Schedule manual reps on purpose. Bainbridge suggested letting operators run the process by hand “for a short period in each shift,” and said that if that idea is laughable, you need a simulator. Most teams don’t need a simulator. They need a quarterly session where the automation is switched off in a safe window and someone who hasn’t done the job by hand in a while does it, with an expert watching. Rotate who goes. Treat it like a fire drill, not a punishment.
Protect the mental model, not just the muscle memory. The Casner study says the cognitive skills fade first. So the drill that matters most isn’t “run the script manually.” It’s “walk me through what the system is doing right now and what it would do if this input went bad.” Make that a standing question in incident reviews and in onboarding.
Watch for the expert who is quietly leaving. Often the only person who can work without the automation is the one who built it or ran things before it existed. That is the setup for being one resignation away from chaos. Pair them with someone before they move on, not after.
Write the manual procedure down, and test it. A runbook nobody has run is a guess. The checklist lessons from aviation and surgery apply here; I covered why teams ignore procedures that don’t match the real work in an earlier piece. Test the fallback procedure the same way you would test a backup restore: by actually doing it.
Don’t hide the automation’s state. Bainbridge warned that automatic control can “camouflage” a failing system by compensating for it until the problem is beyond control. Ask your team whether the dashboards show what the automation is doing and fighting against, or only that everything is green.
How do I know if my team is already exposed?
Ask three questions at your next team meeting and listen to how long the pauses are.
- If this tool went down at 2 a.m. tonight, who on the team could do the work by hand, and when did they last do it?
- Which situations does our automation give back to a human, and has anyone here handled one in the last six months?
- If the automation were quietly doing the wrong thing, how would we notice?
If the answers come back as one name, “never” and “we’d probably get a complaint,” you have the same exposure Google’s engineers had before Diskerase, and the same one a lot of us carried in operations for years without knowing it. The fix is not exotic. It is practice, written procedures that get tested and a deliberate decision about what the humans are there to do.
Bainbridge wrote her paper before most office workers had a desktop computer. Every wave of tooling since then, from process control to cloud orchestration to the AI assistants now landing on every team, has proved her right. Automation removes the easy work and leaves your people with the hard work, so keeping them sharp for it is part of the job.