Where this comes from
Aviation turns up in AI writing mostly as decoration. Someone reaches for a checklist metaphor, nods at the black box, and moves on. That wastes a good body of engineering. The useful part of aviation safety is not the imagery. It is the specificity, and it has to cash out in something an engineer or operations lead can do on Monday.
The credential here is narrow, so state it exactly. From December 2009 to February 2019 my job was leading IT for a regulated flight-training organization in São Paulo. That included the procurement and maintenance lifecycle for the flight simulators. Not a pilot: the person who bought them, kept them serviceable and sat in the internal audits, the SoX control work and the civil-aviation compliance conversations.
A checklist is a forcing function, not documentation
Aviation runs 2 categories of checklist. Normal ones structure routine work. Non-normal ones handle failures, and they split again. Memory items are performed immediately, from memory, with nothing open. Read-and-do items have one person read while another confirms each step.
That split maps directly. Your agent’s memory-item set is the actions a duty manager takes with no reference when the system misbehaves: pull the kill switch, route the queue with context preserved, notify the named owner. Run past 3 items and it will not be executed correctly under pressure, the only condition under which it will ever be executed. Everything else belongs in a written runbook, read-and-do, by 2 people.
Challenge and response is the second piece, and it lands where teams resist. A prompt change is a behavior change to a customer-facing system. It deserves the same 2-person discipline as a code deploy: one proposes, one confirms, nobody pushes alone. Most organizations treat prompts as content because they look like content. Content does not talk to customers unsupervised.
Then there are killer items, where an omission is not recoverable. Every agent release has 4 worth printing on the template: retrieval index pinned to the release, escalation path tested end to end, redaction confirmed on, rollback verified. Verified means somebody performed it, not that somebody believes it.
Simulators, and the cost of practicing the wrong thing
The lesson from owning simulators rather than flying them is fidelity. Fidelity is a specification with a price attached. A device that behaves differently from the aircraft does not merely fail to teach. It teaches the wrong reflex. Negative training is a recognized hazard, not a theory.
Evaluation sets carry the identical risk. A set built from invented tickets no customer would write gives the whole team negative training: a green board on a distribution that does not exist. Build the set from real conversations, labeled by people who have worked the queue, including the ones that went badly.
Simulators transfer 2 further properties. They are re-evaluated on a schedule against the aircraft they represent, because both drift. Re-qualify your evaluation set against production every release cycle. A stale set is worse than none, and none at least keeps you cautious. Simulator time is also scheduled and budgeted as an asset with a cost of downtime. Evaluation time that is not on a calendar, with an owner’s name against it, is not happening.
Aviation did not get safe by hiring better pilots. It got safe by assuming good ones would fail, and writing down exactly what happens next.
The difference between an alert and a warning is a design constraint
Flight-deck alerting is a hierarchy with a rule attached to each level. A warning is red, carries an aural and demands immediate action. A caution is amber: immediate awareness, later action. An advisory is information. Crews are trained to that grammar, and non-critical alerts are inhibited during takeoff and landing so nothing trivial competes at the worst moment.
Most AI monitoring has 1 level, called an alert. Everything lands in one channel, the channel gets muted within a month, and the team learns the panel lies. Rebuild it on the aviation rule, which is stricter than it sounds: a warning names its immediate action. If you cannot write the action, it is not a warning. A cross-tenant leakage detector firing is a warning: pull the switch. A wrong-answer rate drifting up week over week is a caution: review it this week, wake nobody. Containment dipping for a day is an advisory, and belongs on a dashboard somebody reads on purpose. Then inhibit the rest during a major incident. A team fighting a live failure does not need the weekly quality digest competing for it.
Crew resource management is an escalation policy
Crew resource management came out of accident investigations where the aircraft was flyable and the crew qualified. The failure sat in how the flight deck handled information and authority. United Airlines Flight 173 is the canonical case. A crew worked a landing-gear indication so long that the aircraft ran out of fuel. The fuel state was known in the cockpit, and nobody had enough standing to make the point land. The fix was not a better first officer. It was procedure that turned challenging the captain into defined behavior rather than personal courage.
That yields 4 practices, all buildable this quarter.
Flatten the authority gradient. The specialist who watched the agent give bad advice is your first officer. If raising it costs social capital, you will hear about it later from a customer instead. Give it a named channel, a named owner and a response commitment.
Adopt the two-challenge rule. A failure pattern raised twice by frontline people without resolution escalates automatically into a review with the AI owner in the room. Automatic, so nobody has to be brave twice.
Run a sterile cockpit. Below 10,000 feet crews confine themselves to the operation. During a rollout window or a major incident, freeze unrelated prompt and workflow changes. Written down, and enforced.
Plan for task saturation. Flight 173’s crew was not idle, it was busy. A floor whose easy tickets are automated away is a floor where every remaining ticket is hard. Specialists then monitor an automated system, which is a harder job than doing the work, with fewer repetitions to stay sharp. Staffing plans that assume the opposite are how a good rollout becomes an attrition problem.
Degraded modes are a feature you ship
The minimum equipment list is the aviation practice I would hand to every AI team first. An aircraft can be dispatched with specified equipment inoperative under written restrictions, because the alternative is grounding anything imperfect, and that is not an operation.
Publish the equivalent for your agent: what it cannot do this month, with the compensating restriction for each. That page ends more arguments than any capability claim. It converts “the bot is unreliable” into “the bot does not touch entitlements and routes them to a specialist within one turn.” Known limitations with restrictions are an operation. Unknown ones are an incident waiting for a customer to find.
Then design the modes. Airbus flight control degrades through defined laws — normal, alternate, direct — each specified, annunciated to the crew and trained. Give your agent at least 2. Retrieval unavailable: answer only from a small pinned, verified set, and hand off sooner. Model provider slow or degraded: deterministic routing, knowledge-base links, faster handoff. Tell the customer. Annunciate the mode, so the floor knows which one it is running in.
The last step separates aviation from most software operations. Degraded modes get trained; crews fly the failure on a schedule. Run yours in production on purpose. Turn retrieval off inside a controlled window and watch what the agent, the queue and the specialists do. An untested fallback is not a fallback. It is a belief.
Careful is not a control
AI teams talk about guardrails as a tax on velocity. Aviation reached the opposite conclusion decades ago. Procedure is the reason the aircraft is permitted to leave the gate at all. The safety record was built by assuming competent people would fail on an ordinary Tuesday, then writing down what happens next.
That has nothing to do with being careful. Careful is a personality trait and it does not survive a bad week. What survives is a written procedure a tired person can execute at speed, and the willingness to have practiced it before it was needed. On a support floor, that is the only definition of reliability that means anything.
Written in a personal capacity. Nothing here represents the position of any employer, past or present.