A working demo is evidence of possibility

A prototype can succeed on a handful of carefully chosen prompts and still struggle with real users. Production introduces messy inputs, provider outages, changing data, unpredictable cost and requests nobody anticipated.

The right question is not whether the demo works. It is whether the team can detect, contain and recover when it does not.

1. Define the service boundary

Write down the system's purpose, users, dependencies and the actions it may perform. Identify the source of truth for each piece of data and the owner of each downstream integration.

If an AI feature can change records or trigger transactions, its permissions and approval rules belong in the application architecture, not solely in the prompt.

2. Evaluate the behavior that matters

Build a test set from representative tasks, difficult cases and known failures. Assess factual quality where it can be checked, but also measure task completion, unsafe actions, refusal behavior and escalation quality.

Record the model, prompt, retrieval configuration and tool versions associated with each evaluation. Otherwise a passing score becomes difficult to reproduce.

3. Plan for unreliable dependencies

Model providers time out. Retrieval systems return stale documents. Tools may succeed after the caller gives up. Decide what users see in each case and which operations can be retried safely.

  • Use timeouts and bounded retries.
  • Make side-effecting operations idempotent where possible.
  • Provide fallback behavior and a way to stop unsafe workflows.
  • Test partial failure rather than only complete outages.

4. Observe the full workflow

Latency and token cost matter, but they do not tell you whether the answer was useful or whether an action was authorized. Instrument retrieval, model calls, policy decisions, tool execution and user-visible outcomes.

Build alerts around meaningful failure signals. A dashboard that nobody owns will not help during an incident.

5. Secure data and control spend

Review which data can enter prompts, which providers receive it and how long it is retained. Apply access checks before retrieval and again before tools act. Set budgets and rate limits for expensive or recursive workflows.

Cost controls are also reliability controls. A runaway agent can exhaust a quota and disrupt unrelated users.

6. Give the system an operator

Name the team that responds to incidents, approves changes and decides when to roll back. Document how to disable the AI feature without taking down the surrounding product.

Run a release rehearsal: introduce a bad retrieval result, a slow model response and a failed tool call. Ask whether the on-call engineer can identify the problem and restore a safe state. If not, that is your next engineering task.