A demo is built for the happy path. One clean question, one good answer, one impressed stakeholder. Production is the other set of cases: incomplete data, ambiguous requests, stale documents, and users who need an answer now.
The distance between those two states is not a better prompt. It is product design and engineering: a narrow task, reliable context, a way to measure quality, and a path for the model to hand work back to a person.
Pick a task with a checkable output
The first AI feature should produce something a person can verify quickly: a suggested category, a draft reply, a summary with citations, a flagged anomaly. Open-ended chat over every company document is difficult to evaluate and harder to trust.
A narrow output also makes the interface simpler. The user sees the suggestion, the source, and the action. They do not have to compose a perfect prompt or interpret a paragraph of prose.
Treat retrieval as the product
Most disappointing AI answers are context problems. The model did not see the right document, saw an outdated version, or mixed two similar records. Retrieval quality matters more than prompt wording.
- Chunk documents by their structure, not by a fixed character count.
- Store source, section, version, and access scope with every chunk.
- Test questions that should return nothing. Empty results are a feature.
- Measure recall on real questions from the people who will use the system.
- Keep an updated copy of the source material, and delete what is no longer valid.
Build an evaluation set before tuning prompts
Collect 50 to 200 real examples from the workflow. For each one, record the input and what a good output looks like, including acceptable variations. Run the set on every change to retrieval, prompts, or models.
Track two things: how often the output is acceptable, and how often it fails in a way that would cause harm. A model that is right 85 percent of the time but confidently wrong on refunds is worse than a model that is right 70 percent and escalates the rest.
Design for escalation
A confidence score from a model is not a safety mechanism. Use checks the system can verify: required fields, agreement between sources, schema validation, and business rules. Route high-impact actions to a person.
The handoff should preserve context. When the model escalates, the reviewer needs the input, the sources it used, the proposed output, and the reason for escalation. A bare error message pushes the work back to the start.
Measure cost per successful task
Latency and token cost are part of the product experience. Measure end-to-end time for a real task, not just the model call. Cache repeated retrieval, route simple work to smaller models, and set limits on context size.
A feature that saves an employee two minutes but takes forty seconds to respond and costs more than the saved time is not a good trade. The economics should be visible before rollout, not discovered on the first invoice.
Log inputs, outputs, and decisions
Production AI needs a record of what happened. Store the request, the retrieved sources, the model output, the checks that ran, and the human decision that followed. Redact sensitive fields, and set a retention period that matches your privacy policy.
This log is how you find regressions, answer customer questions, and improve the evaluation set. Without it, every complaint becomes a guessing game.
Settle data boundaries early
Know which data can leave your infrastructure and which cannot. Review provider terms for training, retention, and subprocessors. If the work involves health, financial, or personal data, involve the people responsible for compliance before the prototype, not after.
Roll out in stages
Start with internal users and a read-only feature. Move to suggestions, then to approved actions, then to automation for the cases that have earned it. Each stage should have a metric and a rollback path.
An AI system that holds up in production is usually less ambitious than the demo and more useful. It does one job, shows its sources, knows when to stop, and leaves a trail that a team can inspect.
Elyonox Technologies
An independent software company in Addis Ababa.