Dark Decisions

The Orchestrator Framework

What has to be true before you let an AI system run unattended

Most AI products do not fail because the model was wrong. They fail somewhere in the gap between a demo everyone approves and a system nobody is watching, and that gap has a specific shape.

You have probably seen the pattern. The demo is excellent. Everyone in the room agrees it is ready. Then it goes into production, where nobody is watching every response, and something goes out that is confident, plausible, and quietly wrong.

The instinct is to blame the model. That is almost always the wrong place to look. The model did what it was asked. What was missing was the architecture around it: the gates that decide what gets through, the record of who accepted it, and the rule that says which decisions a machine may never make on its own.

The Orchestrator Framework is that architecture. It is not a tool and it is not a methodology you buy. It is a set of structural commitments you make before the system is trusted to operate without you, and the reason they have to come first is that most of them cannot be retrofitted once data has accumulated.

This page is the outline: the layers, the gates and the flow between them, on one page. The book, Dark Decisions, sets out each of them in full, with the real failures behind them and how to build every piece.

The five layers

Each layer answers a different question, and they only work in order. A pipeline with no constitution is a set of preferences. A constitution with no gates is a document nobody reads.

The Orchestrator Architecture, shown as stacked layers. At the top, The Orchestrator governs from above and decides, improves and expands. Below it, the Constitutional Layer holds six principles: strategy, honesty, transformation, human authority, transparency and independence. Beneath that, the Eight Gates cover input, retrieval, verification, confidence, permission, logging, rules and escalation. Below those sit the Pipeline, and at the base the Deployment Pattern, either a single product or a platform with children.
Figure 1. The architecture, from the human at the top to the deployed product at the base. Each layer inherits everything above it.

The Orchestrator

That is you, and the point of the name is that you sit above the system rather than inside it. An orchestrator does three things a machine cannot: decides what the system is not permitted to decide, improves the system based on what the feedback loop reveals, and expands it by designing new pipelines.

If you find yourself inside the system instead, checking outputs one by one, that is not diligence. It is a signal that a gate is missing, and the harder you work in that position, the less trustworthy the system actually becomes.

The Constitutional Layer

What the platform believes, written down before it is needed. Six principles govern everything beneath them, and they exist so that the hard calls are made once, calmly, rather than repeatedly under pressure.

The one that does the most work is simple: human authority is not negotiable. Some decisions stay with a person no matter how good the system gets, and naming them in advance is what stops that boundary moving quietly.

The Eight Gates

The gates are where belief becomes mechanism. Four act at a point in the flow, four govern everything and are always on. Together they are the difference between a system you hope is behaving and one you can demonstrate is behaving.

Each gate asks one question, and each has a recognisable shape when it is absent. All eight, with what failing one looks like.

How one input becomes a verified output

The layers describe the structure. This is the same structure in motion, following a single request from the moment it arrives to the moment someone can stand behind the answer.

System flow, showing how an input becomes a verified output. An input passes Gate 1, input quality, then processing passes Gate 2, retrieval. Generation passes Gate 3, verification. The output passes Gate 4, confidence, and is either marked verified or flagged and routed to a human to decide.
Figure 2. Nothing reaches the reader unlabelled. An output is either verified, or it is flagged and a named human decides.

Notice what the flow does not contain: a path where an unchecked answer reaches someone without a label on it. That is the whole design. You are not trying to make the system never wrong, which is not available. You are making sure that when it is wrong, the wrongness is visible, attributable, and caught by something other than luck.

Which gate are you missing?

If you have a product in production right now, the useful exercise is to audit it against the eight. Most teams find two or three missing, and rarely the ones they expected.

If you want a second opinion, write to darkdecisions@cerebrium.ca. Tell me what the product does and which gate you are least sure about, and I will tell you what usually breaks first. I read every message.