The Model Is the Easy Part: What Actually Breaks When AI Goes to Production

AI demos have become remarkably easy to build. A capable model, a prompt, an API call, and a developer can have something impressive running before the end of the day.Production is different. Once real users arrive, the conversation shifts away from which model performs best on a benchmark and toward much less glamorous problems: latency, inconsistent outputs, context management, moderation, observability, infrastructure costs, and the long tail of failures that demos rarely expose. Dmytro Voroshylov has spent 12 years building consumer software, engineering platforms, and production AI systems. His experience spans B2B engineering products such as Blauberg Selector, a platform for ventilation-system selection, as well as a conversational AI startup serving hundreds of thousands of monthly users. Earlier in his career, he was involved in building and scaling three trivia games distributed through the App Store and Google Play, whose combined audience exceeded one million users. Across these projects, his work has ranged from consumer-facing applications to large-scale AI systems operating in production. Today, as CTO of Inception AI, he works on production AI infrastructure and products including MyAnima, a conversational AI platform serving thousands of users daily. At that scale, questions around memory, personalization, model orchestration, latency, moderation, and inference economics stop being theoretical engineering problems. I spoke with Voroshylov about what actually happens when an AI prototype meets real users – and why the foundation model may be one of the least durable parts of the system. You have said that the model is often the easiest part of a production AI product. What do you mean? The model is obviously important, but today it is also one of the easiest components to replace. You can test another provider, route part of the traffic to a different model, or swap a model behind an interface relatively quickly. The difficult part is everything that accumulates around it. A serious AI product may have an intent classifier deciding what the user is trying to do, routing logic deciding which model or workflow should handle the request, a memory layer assembling relevant history, moderation before or after generation, external tools, fallbacks, analytics, evaluation, and infrastructure responsible for making all of this work reliably. Once you have that, the model is one component in a much larger system. I learned a version of this lesson before the current generative AI wave. When we built engineering software such as Blauberg Selector, the visible interface was only part of the problem. The hard work was encoding domain logic correctly, integrating data, handling edge cases, and making the system reliable enough for professional use. AI products are different, but the production lesson is similar: what users see is only the tip of the system. Teams can spend weeks comparing models while ignoring architectural decisions that have a much larger impact on users. A model can look excellent in a benchmark and still be the wrong production choice if it is too slow, too expensive, unstable under your workload, difficult to moderate, or impossible to operate economically at scale. The question I prefer is not, “What is the smartest model?” It is, “What combination of components gives us the best product behavior?” What is the biggest difference between an impressive AI demo and a product that can actually survive real users? A demo demonstrates the happy path. Production is mostly about everything outside the happy path. In a prototype, a response that occasionally takes several extra seconds is not a crisis. If one request behaves strangely, you retry it. If the prompt is inefficient, the bill is still small because only a few people are testing the system. Once usage grows, those small imperfections become a different class of problem.\. A latency issue becomes a user-experience problem. A rare failure becomes thousands of failures. An unnecessarily long context becomes an infrastructure cost. An edge case becomes something users will discover almost immediately. The biggest mistake I see is taking the architecture that produced the demo and simply adding production requirements around it. Sometimes that works, but often the architecture itself needs to change. You need timeouts, retries, fallbacks, observability, quality checks, prompt and model versioning, and a way to roll back changes. You also need to understand which parts of the experience must be deterministic and which parts can remain probabilistic. Generative AI made prototypes so impressive that the industry temporarily forgot a lesson software engineering learned years ago: prototypes and production systems have different objectives. A demo proves that something can work. Production engineering proves that it keeps working when users behave in ways you did not predict. Does that mean using one powerful LLM to run the entire product is usually the wrong architecture? One powerful model for every task is architecturally simple, so it is a reasonable place to start. But it is often inefficient and sometimes less predictable. Not every problem is a generative reasoning problem. In conversational systems, for example, you may first need to determine whether the user is continuing a conversation, asking for a particular type of content, invoking a tool, or triggering a specialized workflow. A relatively small classifier can often make that decision faster, cheaper, and more consistently than a large general-purpose model. The same principle applies throughout the stack. You might use separate components for intent classification, moderation, memory extraction, summarization, conversation generation, or routing. In some places, traditional code is better than any model at all. This is where model routing becomes a real engineering problem. The objective is not to minimize model size or maximize benchmark quality. It is to send each task to the least expensive and fastest component that can solve it reliably enough. Sometimes that is a large model. Sometimes it is a smaller model. Sometimes it is a rules engine. The intelligence is increasingly in the architecture that knows the difference. MyAnima has to maintain conversations over time. Why isn’t a larger context window enough to give an AI product memory? Because context and memory are not the same thing. A larger context window tells you how much information a model can receive. It does not tell you what information the model should receive. If someone has interacted with an AI system for weeks or months, sending the entire history into every request is usually a bad strategy. It increases cost and latency, but the bigger problem is relevance. Most historical information is not useful for the current interaction, and too much irrelevant context can make the system less focused rather than more intelligent. A useful memory system has to make decisions. What is worth remembering? Is the information temporary or persistent? Is it a stable preference or a one-time event? Has it become outdated? Which memories are relevant to this request? Should the system store a fact, a summary, an embedding, a structured attribute, or nothing at all? In production systems, I think about memory as a pipeline: extraction, representation, ranking, retrieval, conflict resolution, and context construction. The quality of that pipeline directly affects personalization. This becomes very concrete in a conversational AI product such as MyAnima. Continuity is part of the product itself. If the system forgets an important preference, recalls something irrelevant, or treats every session as a blank slate, the interaction immediately feels less coherent. Two companies can use exactly the same foundation model and produce very different experiences because one feeds it a much better representation of the user and the current situation. That is why I expect memory and context construction to become an important source of product differentiation. Foundation models are increasingly accessible to everyone. High-quality, product-specific context is not. In conversational AI, users experience latency very differently from engineers. How do you optimize it when one response may pass through several systems? Users experience latency emotionally. In a conversational product, response time changes the perception of intelligence. A very sophisticated answer that consistently arrives too late can feel worse than a slightly simpler answer that feels immediate. The difficult part is that end-to-end latency is cumulative. A single interaction may involve classification, memory retrieval, moderation, model inference, external APIs, and post-processing. Every component can look acceptable in isolation while the total experience becomes slow. If five layers each add a few hundred milliseconds, suddenly you have a problem that no single component appears responsible for. So optimization has to happen at the system level. Sometimes the solution is a faster model. Sometimes it is parallelizing independent operations, prefix caching, avoiding repeated preprocessing, shortening the context, or deciding that one step should not call an LLM at all. When you self-host models, infrastructure details start to matter even more: batching, queueing, cache hit rates, GPU utilization, and how requests of different lengths interact. Those aren’t purely infrastructure concerns. They determine how the product feels – and what each user interaction costs. Model quality cannot be optimized independently from latency, reliability, and cost. There is no universally best model because there is no universally best point in that trade-off. Generative AI is probabilistic by nature. How do you know whether a change actually made the product better? This is one of the most underestimated parts of AI engineering. Traditional software testing is comfortable because we often know the expected result: for input A, the system should produce output B. Generative AI is less deterministic, and multiple very different outputs can all be acceptable. Worse, an answer can be factually correct and still be wrong for the product. It can ignore context, repeat itself, use the wrong tone, fail to remember something important, or technically answer the question while missing what the user was actually trying to achieve. So I don’t believe in a single evaluation metric. You need layers. Deterministic tests still matter for structured behavior. Curated regression conversations help catch recurring failures. Human review is valuable for ambiguous quality questions. Model-based evaluation can scale some forms of review, but it has to be calibrated carefully. Production behavior matters enormously too: user satisfaction, retries, abandonment, conversation depth, refusal rates, latency, and other product metrics can tell you whether an apparently “better” model actually improved the experience. In a conversational product such as MyAnima, that also means evaluating continuity: whether memory was used appropriately, whether the tone remained consistent, and whether the system avoided repetitive or contextually awkward responses. One practice I consider essential is versioning prompts and model configurations like software. If a prompt changes, you should know which version went to which users, what metrics moved, and how to roll it back. Benchmarks are useful for screening models, but they are not product evaluation. A model can gain several benchmark points and have almost no measurable effect on users. A small improvement in memory retrieval or response time can have a much larger practical impact. At what point does AI architecture become an economics problem rather than purely an engineering problem? Very quickly, because generative AI gives intelligence a visible variable cost. Every additional token, longer context, larger model, redundant inference step, and extra retry changes the economics of the product. Suppose a new model produces noticeably better answers but costs several times more. Is it a better production model? You cannot answer that from a benchmark alone. You have to ask whether users notice the improvement, whether it affects retention or conversion, what it does to latency, and whether the resulting economics make sense. The same principle applies to the choice between using model APIs and running models on your own infrastructure. I don’t think there is a universally better option. Different workloads have different requirements. APIs can be the better choice when you want to move quickly, avoid infrastructure and maintenance overhead, handle unpredictable traffic, or gain access to frontier models without operating them yourself. In those cases, paying a higher unit cost can be completely rational because you are also paying for simplicity, reliability, scaling, and reduced engineering overhead. Self-hosting becomes more attractive when workloads are large and predictable enough that infrastructure utilization can be optimized. At sufficient scale, running selected models on your own GPUs can significantly reduce inference costs and give you more control over latency, batching, deployment, and model configuration. But that control comes with responsibility. Self-hosting also makes scaling considerably more complex: instead of relying on a provider to absorb traffic spikes, you have to plan GPU capacity, manage autoscaling or reserved capacity, deal with uneven utilization, and make sure the system can handle peak demand without maintaining excessive idle infrastructure. Capacity planning, monitoring, failover, deployment, upgrades, GPU utilization, and ongoing maintenance all become your responsibility. That is why I don’t compare an API price with a GPU rental price and stop there. I look at the total system: traffic patterns, utilization, engineering time, redundancy, peak capacity, operational risk, and the latency targets the product actually needs. Sometimes the right architecture is entirely API-based. Sometimes self-hosting makes more sense. And quite often, the best solution is hybrid — running predictable, high-volume workloads on your own infrastructure while using external APIs for models or capabilities that would be inefficient to operate internally. A cheap GPU that sits idle is not cheap infrastructure. At the same time, an API that looks expensive per token may still be the more economical choice if it eliminates a large operational burden. For me, the useful metric is not cost per token or cost per GPU. It is the total cost of reliably producing one useful AI interaction at the quality and latency level the product requires. If a team is building its first serious AI product today, what architectural principle would you insist on from day one? Design the system so the model can change. The AI ecosystem is moving too quickly to make one model or one provider the structural center of your product. The model that looks perfect today may be too expensive six months from now. A smaller model may reach comparable quality. A provider may change pricing. Your latency requirements may change. A new open model may suddenly become good enough to self-host. If replacing the model requires redesigning the application, the architecture is too tightly coupled. I prefer to treat models as powerful but replaceable components behind clearly defined interfaces. The same applies to moderation, memory, and other AI services. You want the freedom to test alternatives without rebuilding the product. Teams should also resist the urge to make the AI layer responsible for things that can remain deterministic. The more of your product behavior you can keep explicit, testable, and observable, the easier the probabilistic components are to manage. Over time, I expect competitive advantage to move away from access to a particular foundation model and toward the system around it: proprietary data, memory, routing, evaluation, infrastructure, domain knowledge, and the feedback loop with users. Foundation models are increasingly becoming infrastructure. The product is the system you build around them.