# From Notebook Demo to Production AI: A Buyer's Commissioning, Acceptance Testing, and Handover Checklist

Canonical URL: https://alectorlab.com/insights/prototype-to-production-ai-commissioning-acceptance-checklist
Raw Markdown: https://alectorlab.com/insights/prototype-to-production-ai-commissioning-acceptance-checklist/raw
Author: Muhammad Faysal (Founder & Technical Lead) — https://alectorlab.com/team/muhammad-faysal
Category: AI Product Engineering & Governance
Published: 2026-11-03
Updated: 2026-11-03
Editorial Review: 2026-11-03 (engineering-guidance)

> Source-backed engineering commissioning guidance by Alector Lab. The five-gate checklist, error taxonomy, and release-gate code are illustrative frameworks rather than legal certifications or reports of specific customer outcomes. Primary sources checked November 3, 2026.

## Executive Summary
How technical buyers evaluate, test, and commission production AI products—covering failure taxonomies, security boundaries, release gates, and handover.

## Executive answer: treat a working prototype as a discovery input, not a partial production build

**A notebook demo or hackathon prototype proves that a model can produce a plausible output on curated examples; a production AI product proves that a complete workflow can handle messy inputs, enforce access permissions, route exceptions to a human reviewer, and stay within an operating budget.** When commissioning [AI product development](/services/ai-product-development), buyers should structure delivery around explicit acceptance gates rather than paying for open-ended prompt tuning.

This commissioning guide outlines the five acceptance gates Alector Lab uses to transition AI prototypes into dependable software. The checklists, release-gate code, and governance tables below are engineering recommendations informed by the [NIST AI Risk Management Framework (AI RMF 1.0)](https://www.nist.gov/itl/ai-risk-management-framework), the [OWASP Top 10 for Large Language Model Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/), and [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). They are not a legal compliance certification or a claim of guaranteed commercial outcomes.

If your team already has a working prototype, do not throw away the domain lessons it uncovered—but do audit which components are reusable product code and which are fragile scaffolding before committing to a launch date.

## Why promising AI prototypes stall during production integration

> **[NOTE] The Prototype-to-Production Gap:** Prototypes optimize for speed of demonstration on clean examples. Production systems optimize for bounded failure modes, least-privilege data access, source traceability, and predictable operating cost across thousands of real user sessions.

Across enterprise AI initiatives, projects rarely stall because a frontier model lacks general language or vision capability. They stall when an unhardened prototype meets four production realities:

1. **Unbounded Failure Modes and Silent Hallucinations:** A prototype demo shows the model answering five well-formed questions. In production, users upload degraded scans, ask ambiguous questions about conflicting records, or paste untrusted external text. Without a severity-ranked error taxonomy, a single fabricated policy number or misrouted action erodes operator trust.
2. **Flat Permissions and Prompt Injection Exposure:** Prototypes often connect to databases or internal APIs using a single shared service key. The [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) highlights prompt injection, sensitive information disclosure, and excessive agency as primary application risks—all of which require architectural controls outside the model prompt.
3. **Missing Review Ergonomics:** When an AI assistant is ninety-something percent right but offers no source links or confidence boundaries, human reviewers must manually re-verify every single item from scratch, erasing the time savings the tool was meant to deliver.
4. **Unmonitored Token, Retry, and Latency Spikes:** Multi-step agent loops or unconstrained document passes that work smoothly for one test user can trigger cascading timeouts and runaway API bills under concurrent production load.

Addressing these gaps requires treating the model as one bounded component inside a broader software system—complete with typed contracts, deterministic validators, human review queues, and release gates. For a deeper look at multimodal system boundaries, see [Production Architecture for Multimodal AI Systems](/insights/production-architecture-for-multimodal-ai).

## The five commissioning gates from prototype review to production handover

Use the five-gate commissioning table below to structure milestones with an internal engineering team or an external development partner. Each gate pairs an engineering question with an explicit buyer acceptance artifact:

| Commissioning Gate | Primary Engineering Question | Required Verification Test | Buyer Handover Artifact |
| --- | --- | --- | --- |
| **Gate 01: Task & Failure Taxonomy** | Which specific decision is assisted, who owns acceptance, and which mistakes are unacceptable? | Walkthrough of normal, ambiguous, and out-of-scope inputs with the process owner | Signed workflow specification and severity-ranked error taxonomy |
| **Gate 02: Data & Access Boundary** | Which data sources may be read, how are tenant permissions enforced, and where is data retained? | Unauthorized-access test, untrusted-input injection test, and retention audit | Data-flow diagram, permission matrix, and provider terms checklist |
| **Gate 03: Representative Evaluation** | Does the candidate system beat the baseline on held-out cases within latency and cost limits? | Automated evaluation run across normal, edge, and adversarial test sets | Versioned golden dataset and repeatable release scorecard |
| **Gate 04: Human Review & Rollback** | How do operators inspect source evidence, approve writes, and halt or reverse bad outputs? | End-to-end test of exception queues, idempotent retries, and kill-switch fallback | Review UI, tool idempotency contracts, and tested rollback procedure |
| **Gate 05: Observability & Handover** | Can the customer's team monitor quality, cost, and incidents after the agency steps back? | Simulated dependency failure, alert routing check, and clean-room deployment test | Source code, infrastructure templates, trace dashboards, and runbook |

**Text equivalent of the diagram:** Progression from prototype to production passes sequentially through Gate 01 (Task & Failure Taxonomy, producing a signed scope spec), Gate 02 (Data & Access Boundary, producing a data and auth matrix), Gate 03 (Representative Evaluation, producing a release scorecard), Gate 04 (Human Review & Rollback, producing a verified control test), and Gate 05 (Observability & Handover, producing the production handover package).

This structure follows the lifecycle spirit of the [NIST AI Risk Management Framework (AI RMF 1.0)](https://www.nist.gov/itl/ai-risk-management-framework)—connecting governance and mapping directly to measurement and operational management.

## Designing an acceptance test suite and severity-ranked error taxonomy

A single average accuracy number is never a sufficient release criterion for a business AI system. An average hides whether mistakes are harmless formatting quirks or severe operational failures. Before Gate 03 sign-off, partition evaluation outcomes into three explicit severity tiers:

* **Severity 1 (Blocker — Zero Tolerance in Release Set):** Unauthorized tool invocation, cross-tenant or unauthorized record disclosure, fabricated source citations, or unflagged schema/arithmetic corruption passed as verified data.
* **Severity 2 (Guarded Exception — Must Route to Review):** Incomplete source context, low-confidence extraction on degraded inputs, or conflicting policy documents where the system must abstain and escalate to a human queue rather than guessing.
* **Severity 3 (Quality & Latency Variance — Tracked Against Threshold):** Suboptimal summary phrasing, unnecessary clarification turns, or tail latency exceeding target interactive budgets.

The illustrative TypeScript release-gate module below demonstrates how a CI/CD evaluation runner can enforce zero tolerance on Severity 1 blockers while checking task completion, escalation quality, and cost per accepted completion:

```typescript
export interface EvalCaseResult {
  caseId: string;
  category: "normal" | "ambiguous" | "adversarial";
  acceptedByRubric: boolean;
  properlyEscalated: boolean;
  severity1Violations: Array<
    "unauthorized_tool" | "cross_tenant_leak" | "fabricated_citation" | "unflagged_schema_error"
  >;
  latencyMs: number;
  costUsd: number;
}

export interface ReleaseGatePolicy {
  minNormalAcceptanceRatio: number;
  minAmbiguousEscalationRatio: number;
  maxP95LatencyMs: number;
  maxCostPerAcceptedTaskUsd: number;
}

export function evaluateReleaseCandidate(
  results: EvalCaseResult[],
  policy: ReleaseGatePolicy
) {
  const blockers = results.filter((r) => r.severity1Violations.length > 0);
  const normalCases = results.filter((r) => r.category === "normal");
  const ambiguousCases = results.filter((r) => r.category === "ambiguous");

  const normalAcceptRatio =
    normalCases.filter((r) => r.acceptedByRubric).length / Math.max(normalCases.length, 1);
  const ambiguousEscalationRatio =
    ambiguousCases.filter((r) => r.properlyEscalated).length / Math.max(ambiguousCases.length, 1);

  const sortedLatencies = [...results].map((r) => r.latencyMs).sort((a, b) => a - b);
  const p95Index = Math.min(sortedLatencies.length - 1, Math.floor(sortedLatencies.length * 0.95));
  const p95LatencyMs = sortedLatencies[p95Index] ?? 0;

  const totalCostUsd = results.reduce((sum, r) => sum + r.costUsd, 0);
  const acceptedCount = results.filter((r) => r.acceptedByRubric || r.properlyEscalated).length;
  const costPerAcceptedUsd = totalCostUsd / Math.max(acceptedCount, 1);

  const reasons: string[] = [];
  if (blockers.length > 0) {
    reasons.push(`Blocked by ${blockers.length} Severity-1 safety/integrity violation(s).`);
  }
  if (normalAcceptRatio < policy.minNormalAcceptanceRatio) {
    reasons.push(`Normal task acceptance (${normalAcceptRatio.toFixed(2)}) below policy.`);
  }
  if (ambiguousEscalationRatio < policy.minAmbiguousEscalationRatio) {
    reasons.push(`Ambiguous-case escalation (${ambiguousEscalationRatio.toFixed(2)}) below policy.`);
  }
  if (p95LatencyMs > policy.maxP95LatencyMs) {
    reasons.push(`p95 latency (${p95LatencyMs}ms) exceeds budget.`);
  }
  if (costPerAcceptedUsd > policy.maxCostPerAcceptedTaskUsd) {
    reasons.push(`Cost per accepted task ($${costPerAcceptedUsd.toFixed(4)}) exceeds budget.`);
  }

  return {
    approvedForRollout: reasons.length === 0,
    p95LatencyMs,
    costPerAcceptedUsd: Number(costPerAcceptedUsd.toFixed(4)),
    blockingReasons: reasons,
  };
}
```

## What a complete agency handover and operating runbook must include

When commissioning an external partner to build or harden an AI product, ensure the commercial agreement defines handover artifacts from day one so your team is never locked into opaque glue code:

* **Full Application, Prompt, and Infrastructure Repository:** All application code, versioned prompt templates, schema validators, and deployment manifests should live in a repository your organization controls, with clear separation between provider-specific adapters and core domain logic.
* **Standardized Production Telemetry:** Instrument model calls, retrieval lookups, tool executions, token consumption, and reviewer overrides using vendor-neutral standards such as [OpenTelemetry's semantic conventions for generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/). That visibility allows your engineering team to diagnose latency regressions or swap model endpoints without rewriting observability pipelines.
* **Maintained Evaluation Dataset and Regression Script:** The acceptance test set used at Gate 03 should be handed over as a runnable command so your team can re-verify quality whenever a model provider deprecates a checkpoint or an internal data source changes. See our [AI Evaluation & Observability service](/services/ai-evaluation-observability) and [Evaluating Hallucinations in Enterprise AI Systems](/insights/evaluating-hallucinations-in-enterprise-ai-systems) for how we structure ongoing regression suites.
* **Explicit Operating Budget and Incident Runbook:** Document who owns API credentials, spend caps, rate-limit alerts, review-queue SLAs, and the manual fallback path if an external model provider experiences an outage.

For an example of how multi-surface product engineering, voice workflows, and structured review come together in a live platform, explore our [Dandy AI real estate platform case study](/work/dandy-ai-real-estate-platform), review our [AI Product Development service](/services/ai-product-development), or [contact Alector Lab](/contact) to audit an existing prototype.

## Frequently asked questions

### Can an agency reuse our existing AI prototype instead of rebuilding from scratch?

Yes—start with a structured prototype audit. Domain prompts, sample datasets, and UI workflows are often reusable, whereas shared API keys, unvalidated JSON parsing, and synchronous blocking calls usually need refactoring before production traffic is safe.

### What acceptance tests should we require before paying for production rollout?

Require a held-out evaluation set covering normal tasks, incomplete or ambiguous inputs that should escalate to human review, and adversarial or unauthorized requests that the system must refuse. Pair quality checks with tail-latency measurements and cost per accepted task.

### How do we prevent vendor lock-in when commissioning an AI product?

Require that your organization owns the repository, evaluation dataset, and deployment configurations, and ask the engineering team to isolate model calls behind typed interfaces and standard OpenTelemetry traces so models or hosting providers can be swapped with bounded effort.

### How should we budget for ongoing model and review costs after launch?

Track total operating cost divided by accepted completions—including model tokens, retrieval infrastructure, retries, and human review time. Set hard monthly provider spend alerts and test whether smaller or routed models can handle routine classification steps.

### When should a workflow stay deterministic software instead of using an LLM?

Keep validation rules, permissions, arithmetic, and fixed routing in ordinary deterministic software. Use language or vision models only for steps that genuinely require unstructured interpretation, and verify their outputs with deterministic code before committing changes.

## Verified Claims & Scope
- **Claim:** The NIST AI Risk Management Framework (AI RMF 1.0) structures trustworthy AI risk management around governance, mapping, measurement, and management functions.
  - **Scope:** Voluntary risk-management framework reference; applying it to a specific software release requires organization-specific acceptance criteria.
  - **Source (NIST AI RMF 1.0 portal, checked November 3, 2026):** https://www.nist.gov/itl/ai-risk-management-framework
- **Claim:** The OWASP Top 10 for Large Language Model Applications identifies prompt injection, sensitive information disclosure, and excessive agency among primary application security risks.
  - **Scope:** Application threat taxonomy; mitigations must be implemented and tested against each product's specific tools and data boundaries.
  - **Source (OWASP LLM Top 10 project page, checked November 3, 2026):** https://owasp.org/www-project-top-10-for-large-language-model-applications/
- **Claim:** OpenTelemetry defines semantic conventions for generative AI telemetry covering model requests, token usage, latency, and tool spans.
  - **Scope:** Observability instrumentation specification; does not replace human calibration or domain evaluation.
  - **Source (OpenTelemetry GenAI Semantic Conventions, checked November 3, 2026):** https://opentelemetry.io/docs/specs/semconv/gen-ai/

## Primary Citations
- [Artificial Intelligence Risk Management Framework (AI RMF 1.0)](https://www.nist.gov/itl/ai-risk-management-framework) — *National Institute of Standards and Technology (NIST) · checked November 3, 2026*
- [OWASP Top 10 for Large Language Model Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) — *OWASP Foundation · checked November 3, 2026*
- [Semantic Conventions for Generative AI Systems](https://opentelemetry.io/docs/specs/semconv/gen-ai/) — *OpenTelemetry Documentation · checked November 3, 2026*