From AI prototype to production: what changes

A demo proves something is possible. Production proves it is dependable. The work in between is evaluation, data, review, cost and ownership.

By Elias Limouni · 12 November 2026 · 5 min read

A prototype shows a model can do the task on examples you chose. Production means doing it on inputs you did not choose, every day, at a known cost, with someone accountable.
Build an evaluation set before launch and measure quality on every change.
Decide where people review outputs, and design the product around that.
Plan cost, data protection, monitoring and ownership from the start, and hand over a system your own team can run.

Why the demo was easy

AI prototypes are quick to build and impressive to watch. A capable model, a well-written prompt and a handful of documents can produce something in days that looks close to finished. It is not, and the gap is mostly not about the model.

A demo is run by the person who built it, on examples they chose, with nobody depending on the result. Production is the opposite: inputs nobody anticipated, users who will trust the output, and a business process that stops when the system does. Getting from one to the other is engineering and operational work, and in my experience it is where most of the effort goes.

Evaluation: know whether it works

The most important step is an evaluation set: a collection of real inputs with the outputs a competent person would expect, covering ordinary cases, awkward ones and the ones that must never go wrong. Without it, every change to a prompt, a model or a data source is a guess, and quality drifts without anyone noticing.

Start with a modest set drawn from real work, labelled by the people who do that work today, not by the engineers.
Decide what good means for each task: an exact answer, an acceptable range, or a rubric a reviewer can apply.
Run the set on every change, and before switching model or provider.
Add the failures you find in production. The set should grow with the system.

Evaluation also changes the conversation with the business. Instead of “it seems better”, you can say which cases improved, which got worse and whether the trade is acceptable. That is the basis for every later decision about models, cost and how much review is still needed.

Data: the part the demo skipped

Prototypes usually run on a clean extract. Production runs on the real thing: documents in odd formats, blank fields, records that contradict each other, and data that changes after the system has read it. The work is in ingestion, cleaning, keeping sources current, and making sure the system only retrieves what each user is allowed to see.

Data protection belongs here too. Know what personal data flows into prompts and logs, where it is processed, which suppliers see it and on what terms, and how long it is kept. Under UK GDPR these are questions to answer before launch, not after a customer asks.

Be wary of fixing data problems inside the prompt. If the system needs to know that two fields mean the same thing, fix it in the pipeline, where it can be tested, rather than in instructions the model may or may not follow.

Human review: design it in

Few business processes should take an AI output and act on it with no person involved, at least at first. The design question is where people review, not whether.

Grade outputs by risk Some can go straight through, some need a check, and some need a decision by a qualified person.
Make review fast Show the source beside the output, flag uncertainty, and make correcting easier than starting again.
Capture corrections They are your best source of new evaluation cases.
Agree when review can be reduced Decide with the process owner what has to be true, and measured, before people step back.

Cost, reliability and operations

A prototype's running cost is a rounding error. At production volume, model calls, retrieval and storage become a real line in the budget, and cost per task is worth tracking alongside quality. Choose the smallest model that passes your evaluation set, cache what you can, and set limits so a fault cannot run up an unbounded bill.

Reliability needs the same thought as any other system. What happens when the model provider is slow or unavailable, when an output is malformed, or when a new model version behaves differently? Log enough to explain any individual output after the fact, monitor quality, cost and latency over time, and keep a way to roll back.

Ownership and handover

The last gap is the one most often missed. Someone in the business must own the system's results, not just its code. They decide what good looks like, approve changes that affect quality, and answer for it when something goes wrong.

If an outside team built it, the handover is part of the work, not a courtesy at the end: documentation of how it works and why, the evaluation set and how to run it, runbooks for common failures, and time spent alongside the people who will maintain it. The aim is a system your own team can run, change and explain without the people who built it.

A production-readiness checklist

Before an AI system carries real work, every line below should have an answer someone has written down.

Before an AI system goes live

An evaluation set built from real cases, run automatically on every change.
A definition of acceptable quality for each task, agreed with the process owner.
Production data pipelines, with access control that matches the source systems.
A record of personal data in prompts and logs, the suppliers involved, and retention.
Human review designed by risk level, with corrections captured.
Cost per task measured, with limits and alerts.
Fallbacks for provider outages and malformed outputs.
Monitoring of quality, cost and latency, with a way to roll back.
A named business owner and a named technical owner.
Documentation, runbooks and a completed handover to the team that will run it.