An AI feature usually breaks between demo and production for one of four reasons. The real data is messier than the demo data. Cost or latency that was fine for one request falls apart at real volume. Edge cases the demo never hit show up constantly at scale. Or users don't trust an output they can't verify, so they stop using it. These are engineering and design gaps around the model rather than problems with the model itself, which is why they can be fixed before launch instead of after.
Gap 1: the data is messier than the demo data
A demo usually runs on a handful of hand-picked, clean examples. Production runs on scanned PDFs at an angle, WhatsApp messages with typos and voice notes, and spreadsheets with three different date formats in the same column. A model that scored well on clean inputs can degrade sharply on the real distribution, and nobody notices until it's already live.
The fix is unglamorous. Pull a real, messy sample of production-like data before launch, and test against that instead of a curated set.
Gap 2: cost and latency that don't survive real volume
One call to a large model in a demo costs a few cents and takes a couple of seconds, so nobody feels it. At thousands of requests a day, both the bill and the latency compound, and a feature that felt instant in the demo starts timing out or getting throttled.
The usual solution is routing. A fast, cheap model handles the high-volume, low-complexity cases, and a stronger, slower model is reserved for the fraction of requests that actually need it.
Gap 3: edge cases the demo never hit
A demo shows the happy path, once. Production hits the unhappy path constantly: the document that's actually two documents stapled together, the customer who asks something completely outside the workflow's scope, the record that doesn't exist in the system it's supposed to look up. Each edge case is rare on its own and common in aggregate.
The practical fix is a clear, narrow definition of what the system handles automatically, with everything else escalated to a human on purpose. Where you draw that boundary is a design decision, and it shouldn't be treated as a bug to eliminate.
Gap 4: nobody trusts an answer they can't check
Even a technically correct AI output gets ignored if the person using it can't tell whether to trust it. Production systems that succeed show their sources, flag their own uncertainty, and log every action. That lets a human verify quickly instead of redoing the work from scratch, which would defeat the purpose of automating it in the first place.
Product Engineering
We build the architecture, reliability and release process that keep a growing product from falling over.
See how we build it →