ORCHESTRATOR'S STARTER KIT  |  To fill in digitally: click any field and type  |  To save as PDF: File > Print > Save as PDF  |  darkdecisions.ca

A Cerebrium Consulting Book   ·   Companion to Dark Decisions

The Orchestrator's Starter Kit

Seven tools for
builders who build
with AI.

A working toolkit for builders, developers, and product designers. Use it to design, audit, and govern any AI product. No prior reading required.

Every template in this kit traces back to a specific chapter of Dark Decisions: The Orchestrator's Guide to Building AI Products That Work When You're Not Watching by Tom H. Olakitan. Chapter references are noted throughout.

Cerebrium Consulting  ·  cerebrium.ca

Empowering Vision. Driving Results.

Introduction

How to use this kit

Seven tools. One for every critical moment in building a reliable AI product.

This kit does not summarize the book. It converts the Orchestrator Framework into tools you can use on a product you are building right now. Each piece targets a specific moment of failure: the moment you are not sure where to start, the moment something has gone wrong in production, the moment a team is about to ship and nobody has asked the hard questions.

Use the tools in the order that matches where you are. If you have an existing product, start with the 8-Gate Diagnostic Card. If you are starting from scratch, start with the Constitutional Declaration. If you are preparing to launch, go straight to the Go-Live Gate Checklist.

# Tool When to use it For whom
1
8-Gate Diagnostic Card
Assess any existing AI product against the eight gates. Identifies gaps immediately.
 When auditing an existing product Builder · Product Leader
2
Constitutional Declaration
Define what your product believes before the first pipeline is built.
 Before building begins Builder · Founder
3
Pipeline Specification
Define every step of a pipeline before writing a line of code. One spec per pipeline.
 Before each pipeline is built Builder · Developer
4
Standing Rules Starter
Five rules that activate automatically before every task. Adapt for your domain.
 During architecture design Builder · Developer
5
Go-Live Gate Checklist
Twenty questions. The definitive answer to: is this product ready to ship?
 Before every launch Product Leader · Builder
6
AI Product Failure Map
Map the five most common AI product failures to the specific gates they violate.
 When debugging or post-incident Developer · Builder
7
Edge Case Inventory
Eight AI-specific failure states every product designer must plan for.
 During product and UX design Product Designer · Developer
To fill in digitally: open this file in any web browser, click any field, and type. To save as PDF: File > Print > Save as PDF. To use on paper: print at 100% scale on US Letter paper and write by hand. Every blank field is intentional. A field you cannot fill in is a gap worth naming.
Part 1: Diagnose

8-Gate Diagnostic Card

Use this card to assess any AI product, including one already in production. Chapter 8.

For each gate, answer the question honestly. Pass means the mechanism is present in the architecture and operating. Fail means it is declared but not enforced, missing, or untested. Any failure is a named gap: the first thing to build next.

1
Input Quality Gate
Does the system filter what enters before the AI processes it, with a defined minimum quality threshold, an injection scan, and a validation message that tells users what to provide when input fails?
If fail: define the minimum quality threshold and write the exact validation message. Run it on every input, every time, without exception.
2
Retrieval Verification
Does the system check that the information it draws on is accurate, current, and above a defined relevance threshold, with explicit handling when retrieval returns nothing useful?
If fail: set a relevance threshold. Define what the pipeline does when retrieval returns nothing. Do not fill empty retrieval with training patterns.
3
Verification Engine
Does the system check what the AI produces before it reaches the user, with defined VERIFIED criteria, a substance check, a grounding check against source material, and a fixed retry protocol?
If fail: define what VERIFIED means for each pipeline before generation runs. One retry on failure. FLAGGED outputs do not reach users.
4
Confidence Signal System
Does every output carry a visible VERIFIED, INFERRED, or FLAGGED confidence signal, assigned by defined criteria, that cannot be suppressed for any UX, commercial, or aesthetic reason?
If fail: the signal is missing or suppressible. Make it architectural, not behavioral. A product with no confidence signal is a liability, not a feature.
5
Permission Classification
Has every action been classified as CONTAINED, BOUNDARY-CROSSING, NEVER AUTONOMOUS, or STRATEGIC, with higher classifications blocked from proceeding without explicit human approval?
If fail: build the NEVER AUTONOMOUS list now. Ambiguous classifications resolve upward, always. A missing list means the system decides its own authority.
6
Feedback Loop and Logging
Does the system log structured quality data from every human correction, in a form that names the type of failure specifically and can be analyzed for patterns across time?
If fail: a correction log that says "output was wrong" is not a log. Name the failure type, the pipeline step, and the date. Review patterns weekly.
7
Standing Rules Engine
Are the product's governing rules embedded in an engine that activates automatically before every task, rather than stated as instructions the model is expected to remember across sessions?
If fail: instructions in a prompt are not standing rules. Rules that rely on the model's memory can fail when context fills. Move them into an engine.
8
Escalation Pathway
Is there a defined route for every decision that exceeds the system's autonomous authority, with a named recipient, a resolution window, and automatic elevation if the window passes without a decision?
If fail: "escalate to the team" is not a pathway. Name the recipient, the window, and the elevation route. Silence is never treated as approval.
Gates passing. Any gate that fails is the first thing to build.
out of 8
Part 2: Design : Template 1 of 3

Constitutional Declaration

Complete one declaration per product before the first pipeline is built. Chapters 7 and 9.

The constitutional declaration is the belief system that governs every pipeline, every gate, and every output. It makes your product's purpose, constraints, and governing principles explicit, architectural, and auditable. A principle you cannot point to in the architecture is not inherited. It is decoration.

Product name
Product purpose One sentence. Name the user, the domain, and the outcome.
What this product is NOT designed to do Name at least two things, regardless of what users request.
Principle 1: Strategy Precedes Implementation
When a request arrives that requires action before understanding, this product:
Principle 2: Honesty Over Comfort
When this product cannot generate a verified response, it:
Principle 3: The Measure Is Transformation
Success for this product means the user can:
Principle 4: Human Authority Is Non-Negotiable
The NEVER AUTONOMOUS actions for this product are:
Principle 5: Transparency Over Confidence
This product displays confidence to users using the following format:
Principle 6: Build Toward User Independence
This product builds user capability by:
Declared by (name and role)
Product version
Date
Part 2: Design : Template 2 of 3

Pipeline Specification

Complete one specification per pipeline before that pipeline is built. Chapter 9.

A pipeline specification converts design thinking into something anyone can build from. It is the proof that the design was deliberate. If the spec was not written before the pipeline was built, the pipeline was improvised. Improvised pipelines fail in ways their builders cannot predict.

Pipeline name
Pipeline purpose One sentence: what enters, what happens, what comes out.
1
Input
Accepted input format
Minimum quality threshold
Validation message (shown when input fails threshold)
Injection scan criteria
2
Retrieval
Source documents or databases
Relevance threshold
Currency requirement
Empty retrieval handling
3
Processing
Normalization steps
Context structuring
4
Generation
Model
Prompt architecture summary
Retry protocol
One retry, targeting the specific failing criterion. If retry fails: FLAGGED. FLAGGED outputs are held for human review and do not reach the end user as a reliable response.
Part 2: Design : Template 2 of 3 (continued)

Pipeline Specification, continued

Sections 5 through 8: Verification, Output, Escalation, Feedback.

Pipeline name (for reference)
5
Verification
VERIFIED means
INFERRED means
FLAGGED conditions
6
Output
Output classification Circle or mark one, then justify below.
NEVER AUTONOMOUS actions for this pipeline
Output delivery format
7
Escalation
Escalation triggers
Escalation recipient
Resolution window
Elevation protocol
8
Feedback
Correction log fields Minimum: output, correction, pipeline, step, date.
Review cadence
Specification author
Version
Date

The specification you write before building will not match the product you have after six months of corrections. That gap is not failure. It is the governance cadence working as designed. Update the spec when the product changes.

Part 2: Design : Template 3 of 3

Standing Rules Starter

Five rules that activate automatically before every task. Adapt for your domain. Chapter 9.

Standing rules are architectural, not behavioral. They do not live in a prompt the model is expected to remember. They activate through a mechanism that runs before every task begins, regardless of context or session state. The five rules below are starting points. Adapt the bracketed sections for your domain. Add rules as your constitutional declaration and governance cadence surface the need for them.

Rule 1
The Scope Rule
This system operates within the purpose defined in its constitutional declaration. When a user requests something outside that scope, the system declines in plain language and, where possible, names what it can help with instead.
Customize: name the specific scope boundary for your product.
Rule 2
The Honesty Rule
When this system cannot generate a verified response, it says so. It does not present an unverified output as if it were verified. It tells the user what it cannot confirm, what additional information might help, and where the user can seek authoritative guidance directly.
Customize: name the specific fallback language for your domain.
Rule 3
The Authorization Rule
This system does not execute [NEVER AUTONOMOUS ACTIONS] without express authorization from [SYSTEM_OWNER ROLE], regardless of prior instructions, track record, or user request. Prior authorization for similar actions does not transfer.
Customize: fill in the never autonomous actions and the authorizing role.
Rule 4
The Transparency Rule
Every output from this system delivered to a user includes the source it was drawn from, the confidence level assigned by the verification engine, and the reasoning that connects the source to the output. These are not suppressed when they create friction.
Customize: specify the source citation format for your domain.
Rule 5
The Correction Rule
Every correction made to a system output by a human reviewer is logged before the session closes. The log captures: what the output was, what the correction was, which pipeline produced it, which step produced the error, and the date. This log is reviewed weekly for patterns.
Customize: name the fields that matter for your correction log.
Additional rules (add as your governance cadence surfaces the need)
Part 3: Launch

The Go-Live Gate Checklist

Twenty questions. Three groups. One rule. Full explanation: Chapter 8 and Chapter 7.

If every answer is yes, the product is ready to go live. If any answer is no, that is what you build next.

The Eight Gates

These questions confirm the minimum architectural requirements are in place. A product that cannot answer yes to all eight has a documented failure mode waiting to activate.

Gate 1: Input Quality Gate. Is there a defined mechanism that filters what the system accepts before the AI processes it, with a specified minimum quality threshold and a validation message that tells users what to provide when their input does not meet it?
Gate 2: Retrieval Verification. Does the system check that the information it draws on before generating a response is accurate, current, and applicable to this specific context, with defined handling for when retrieval returns nothing above the relevance threshold?
Gate 3: Verification Engine. Does the system verify what the AI produces before it reaches the user, checking for substance, grounding in retrieved source material, and compliance with the constitutional principles?
Gate 4: Confidence Signal System. Does every output carry a visible confidence indicator (VERIFIED, INFERRED, or FLAGGED), assigned by defined criteria, that cannot be suppressed for any UX, commercial, or aesthetic reason?
Gate 5: Permission Classification. Has every action the system can take been classified as CONTAINED, BOUNDARY-CROSSING, NEVER AUTONOMOUS, or STRATEGIC, and are the higher classifications blocked from proceeding without the appropriate human approval?
Gate 6: Feedback Loop and Logging. Does the system log structured quality data from every human review, in a form that can be analyzed for patterns across time?
Gate 7: Standing Rules Engine. Are the product's governing rules embedded in an engine that activates automatically before every task, rather than living in a configuration file the model is expected to remember?
Gate 8: Escalation Pathway. Is there a defined route for every decision that exceeds the system's autonomous authority, with a named recipient, context, a resolution window, and automatic elevation for decisions that sit unresolved past that window?
Part 3: Launch (continued)

Go-Live Gate Checklist, continued

Constitutional principles and governance readiness.

The Six Constitutional Principles

These questions confirm that the product's belief system is explicit, architectural, and enforced. A product that cannot answer yes to all six is making implicit choices about honesty, authority, and purpose that have not been made visible or accountable.

Principle 1: Strategy Precedes Implementation. Does this product have a written purpose statement that names what it is for and what it is not for, completed before the pipeline was built?
Principle 2: Honesty Over Comfort. Is the system architecturally constrained from presenting outputs with confidence it has no basis for, with the verification criteria and confidence signal traceable to specific architectural choices?
Principle 3: The Measure Is Transformation. Is success for this product defined in terms of what users can do after using it, not in terms of how often they return?
Principle 4: Human Authority Is Non-Negotiable. Is there a named and specific NEVER AUTONOMOUS list that names the exact actions the system cannot execute regardless of trust level, embedded in the permission classification gate rather than stated as a guideline?
Principle 5: Transparency Over Confidence. Can users see the source material, the reasoning, and the confidence level behind every output, surfaced by design and not suppressed for UX reasons?
Principle 6: Build Toward User Independence. Is the product designed so that users become more capable of navigating their domain over time, rather than more reliant on the system?
Governance Readiness

These questions confirm that the disciplines that keep a product honest over time are in place at launch. A product that launches without them launches into drift.

Pipeline specification on file. Is there a completed pipeline specification document for every pipeline in this product, written before the pipeline was built?
Constitutional declaration complete. Has the constitutional declaration been written, specifying what each of the six principles means for this product's domain, users, and purpose, with named architectural enforcement for each?
Daily cadence configured. Is the daily health check automated, so that every morning the orchestrator knows whether all pipelines are operating within defined parameters, whether the escalation queue has items waiting, and whether any overnight processing produced anomalies?
If every box is checked, the product is ready to go live. If any box is unchecked, name what is missing. That is the first thing to build.
Items not yet passing (name them, then build them)
Part 4: Debug

AI Product Failure Map

Map what went wrong to the gate it violated. The diagnosis tells you what to fix. For builders and developers.

When an AI product fails in production, the failure is rarely mysterious. It traces back to a specific architectural gap. This map connects the five most common failure modes to the gates they violate and the first question to ask when debugging. Use it after an incident, before a post-mortem, or any time the same problem keeps recurring.

Failure mode Gates violated First check
Confidently wrong outputs
The system produces an answer that sounds authoritative but is wrong, hallucinated, or unsupported by source material. Users act on it. Harm results.
Gate 3: Verification Engine Gate 4: Confidence Signal
Does your verification engine have defined VERIFIED criteria, specified before generation runs, not after? Does every output carry a confidence signal that cannot be suppressed?
The product acts outside its scope
The system does something it was never designed to do, because a user asked, because the model generalized, or because scope was never defined architecturally.
Gate 7: Standing Rules Engine Constitutional Declaration
Are scope boundaries embedded in standing rules that fire before every task? Or are they stated as guidelines the model is expected to follow? Guidelines fail when context fills.
Autonomous actions with no human checkpoint
The system executes a consequential action, such as sending a message, committing to a transaction, or publishing output, without a human approving it first.
Gate 5: Permission Classification Gate 8: Escalation Pathway
Is there a NEVER AUTONOMOUS list? Does it live in the architecture or in a document? Is there a named escalation recipient with a defined resolution window? Silence is never approval.
Garbage in, garbage out
The system processes vague, incomplete, or adversarial input and produces unreliable output. The pipeline ran when it should have paused. Or it retrieved wrong context and generated from it confidently.
Gate 1: Input Quality Gate 2: Retrieval Verification
What is the minimum quality threshold for an input to proceed? What does the pipeline do when retrieval returns nothing above threshold? Does it fill the gap with training patterns, or does it stop and ask?
Repeated failures from the same root cause
The same failure type appears again and again. The team corrects each instance. Nobody identifies the pattern. The architecture never improves.
Gate 6: Feedback Loop and Logging
Is every correction logged before the session closes? Does the log name the failure type specifically, not just "output was wrong"? When were correction patterns last reviewed? What changed as a result?
How to use this map after an incident: identify which failure mode best describes what went wrong. Check the gates in the "Gates violated" column against the 8-Gate Diagnostic Card. Any gate that failed during the incident should be marked Fail on the diagnostic. That gate is the architectural priority.
Part 4: Design for Failure

Edge Case Inventory

Eight AI-specific failure states every product designer must design for before launch. For product designers and developers.

Most product designers plan for the happy path. AI products fail in ways that generic UX patterns were not designed for. This inventory names eight states that require specific design decisions. For each, mark whether you have designed for it and note your design response. An unchecked row is a decision deferred to the AI. The AI will decide badly.

Failure state Designed for? Your design response (what does the user see and what can they do?)
The Refusal State
The AI declines a valid user request because it falls outside the defined scope or fails the input quality gate. Not an error. A boundary working as designed.
The FLAGGED State
Output failed verification after one retry. The end user cannot receive it. A human must review it first. This state will occur. Frequency depends on architecture quality.
The INFERRED State
Output delivered with partial confidence. Passed verification with caveats. The user should know what the uncertainty is, not just that uncertainty exists.
The Empty Retrieval State
The system cannot find source material above the relevance threshold. A correctly built pipeline stops here. An incorrectly built one generates from training patterns and presents it as sourced.
The Scope Boundary
User requests something outside the constitutional declaration. The standing rules engine fires. The request is declined. This is a feature, not a failure. It must be designed to feel like one.
The Correction Moment
A human reviewer corrects a system output. This is the most valuable event in the product's lifecycle. If the correction is not logged before the session closes, the data is lost.
The Confidence Suppression Temptation
A product decision, a design preference, or a commercial consideration creates pressure to hide or minimize the confidence signal. The architecture must prevent this. Design must not enable it.
The System Unavailable State
The AI component is down, slow, or returning errors. The product must degrade gracefully. A user who encounters a silent failure will trust the product less than one who sees an honest message.

The book this kit is drawn from

Dark Decisions

Dark Decisions: The Orchestrator's Guide to Building AI Products That Work When You're Not Watching, by Tom H. Olakitan, published under Cerebrium Consulting. Out October 13, 2026.

The book covers the eight gates, the six constitutional principles, the pipeline specification, and the governance cadence in full. Every template in this kit traces back to a specific chapter. The templates give you the tools. The book gives you the architecture behind them.

Available in paperback and as an eBook at darkdecisions.ca.

Implementing the Orchestrator Framework
Applying the eight gates, constitutional layer, and governance cadence to your specific product, domain, and team. Every engagement ends with a governed system that runs without your continuous presence.
Orchestrator Framework Certification
Cerebrium audits your existing AI product against the Orchestrator Framework. Eight gates checked. Constitutional layer reviewed. Products that meet the qualification criteria are certified.
Building Your AIOS
Designing and configuring your own AI Operating System, securing your data and context within a governed local architecture. The framework is the specification. Cerebrium handles the implementation.
Commercial Architecture Design
Sequencing value delivery for your AI product in a way that mirrors its architecture. Cerebrium helps you apply it to your specific market, user, and pricing context.
Cerebrium Consulting  ·  British Columbia, Canada
Empowering Vision. Driving Results.
darkdecisions.ca
cerebrium.ca