Parallel Orchestration Patterns for ML Workloads in Production

The Multi-Model Problem Many ML systems start simple. That's the right call to start with as it helps with quick prototyping and building the integrations. The problems start when requirements sharpen and a single general-purpose model stops being the right answer. We want computer vision for frames, NLP for transcripts, audio analysis for the audio track. We want models purpose-built for their domain, not one model doing all three passably. A decision engine should not use the same model for behavioral signals and graph traversal. What started as a single inference step becomes five, then a pipeline pulling signals from three modalities simultaneously, then a decision engine combining outputs from behavioral models, graph models, and rule-based engines. Now we are not orchestrating one model. We are orchestrating a fleet. The instinctive approach is sequential execution. We call model A, wait for it to finish, call model B, and so on. That's fine when we have two or three models. But pipeline latency becomes the sum of all model latencies, and a slow model holds up every step that follows it. For most fan-outs, the models don't depend on each other at all. Model B doesn't need model A's output to do its job. There's no reason they can't run at the same time. Parallel orchestration solves the latency problem. But it introduces failure modes that sequential pipelines don't have: consistency issues, partial failures, straggler handling, and observability gaps. This post covers the patterns that work and the design decisions that matter most. Fan-Out / Fan-In: The Core Pattern When we have multiple models that need to run on the same asset, the first thing we try is running them one at a time. This works until we look at the numbers. Adding models makes the pipeline measurably slower, linearly. The fix is to dispatch all models at the same time and collect results once they're all done. That's fan-out/fan-in. Here's where most implementations go wrong. They treat fan-in as a stream: the moment a model's result lands, they act on it, forward it downstream, log it as done. Futures resolve out of order, so downstream ends up seeing results from six models while the other two are still running. Execution was parallel, but exposure wasn't. The right approach is a barrier: hold every result in a staging area as it arrives, and release the full set downstream only once all N are present. Total wait time doesn't change, we're still bound by the slowest model either way, but downstream sees nothing until it sees everything. This matters because downstream steps, whether a human reviewer, a decision engine, or another model, should never see a partial result set. Partial results create a category of bugs that are hard to detect because the system doesn't fail loudly. The reviewer sees some signals but not others and makes a decision based on incomplete evidence. The decision engine produces a plausible-looking answer that's wrong. Nothing registers as broken. The rule is all or nothing. Either all models have returned and the task advances, or the task stays waiting. No partial states surfaced downstream. Handling Partial Failures "All or nothing" breaks when a model fails in production, which it will. Our fan-out probably covers models from multiple teams running on infrastructure none of them fully controls. Not all failures mean the same thing. A model returning a 503 because it's temporarily overloaded is different from a model that's been timing out for ten minutes. The first case should trigger automatic retry with backoff. The task stays waiting, and the orchestrator retries only the failed model without re-running the ones that already succeeded. Retrying the full fan-out wastes compute and, in systems with side effects, produces duplicate writes. We track completion state per model per task, and retry only what actually failed. The second case, persistent failure, requires a decision we should make before we're in an incident: fail the whole task, or proceed with a null result for that model? The answer depends on whether the missing signal is load-bearing. For some pipelines, seven signals with a noted gap is workable. The downstream step can flag reduced confidence and proceed. For others, that model's output is required for downstream correctness and the task shouldn't advance without it. We make this explicit in configuration. Each model in the fan-out should declare its criticality, required versus optional, so failure behavior is legible and testable, not buried in retry logic that someone will misread during an incident. The Snapshot Problem Here's a bug that's easy to miss in development. We notice outputs from two models in a fan-out are inconsistent in a way that shouldn't be possible. They ran at the same time, against what should be the same input. But one ran against version N of the source asset, and the other against version N+1, because a new version was ingested while the fan-out was mid-flight. Sequential pipelines don't have this problem. Each step runs after the last, and the input at each step is whatever it is at that moment. Parallel pipelines do, because model results arrive at different times, and the input data can change in the window between when the first model starts and when the last one finishes. The fix is to snapshot inputs at fan-out time. When the orchestrator dispatches models, it locks the input state for that task. All models run against the same snapshot, regardless of how long individual models take or what changes in upstream data in the interim. Snapshotting has a cost. We're storing input state for every in-flight task, and at scale this adds up. But the alternative is a consistency bug that only manifests when a specific sequence of timing events occurs in production, produces plausible-looking wrong answers, and is nearly impossible to reproduce in staging. Snapshot invalidation also needs explicit design, and most teams skip this part. If a new model version is deployed while a task is mid-flight, should the in-flight task use the old model or the new one? If the source asset is corrected, should the task restart? These are policy decisions. We make them explicitly before we discover them as surprises. Sometimes snapshotting isn't an option. Our pipelines might be configured separately, or the models are pre-existing systems owned by other teams that can't be batched into a shared snapshot boundary because of their own dependencies. When we can't snapshot, the fallback is data lineage: every result we write records the model version and which input version produced it. This isn't optional bookkeeping. Without it, two inconsistent outputs are just a mystery. With it, "these outputs are inconsistent" becomes "model X ran against input version N, model Y ran against version N+1," which is a fact we can debug instead of a bug we can't reproduce. Lineage doesn't prevent the inconsistency, but it makes the inconsistency legible after the fact, which is what actually matters during an incident. Latency Asymmetry and Stragglers Run a fan-out in production long enough and we'll see the same pattern: seven models finish in 200 milliseconds, then we wait. The eighth takes 45 seconds. The seven fast results are sitting complete in the state store, going nowhere. The task is blocked on one model. In a strict all-or-nothing fan-in, this is unavoidable. Pipeline latency is determined by the slowest required model. But we can keep it from dominating our p95. We start with per-model timeouts. A model that exceeds its timeout is treated as a transient failure and retried. This bounds worst-case latency without giving up on eventual completion. More importantly, we don't block a thread waiting for fan-in to complete. The orchestrator tracks state asynchronously. Fast models write their results to the shared state store as they finish, and the task advances only when all required results are present. If a model hasn't responded within its SLA window, the orchestrator triggers a retry. Nothing is actively waiting. We set SLA windows based on how each model actually behaves, not a uniform timeout applied to everything. A model that routinely takes two seconds isn't a straggler. It's just slow. A model that usually takes 200 milliseconds and is now taking three minutes is degrading, and we want to catch that before it shows up in task completion rates. One thing that consistently catches teams building human-reviewed pipelines: if model results are surfaced to a reviewer as they arrive, the reviewer may start working before the fan-in is complete. Then a late result changes the picture mid-session. Fan-in means fan-in. We surface results only when all required models have completed. Caching Model Outputs Models are expensive. Running eight of them against every item in a large backfill is a significant compute cost, and in many cases the outputs are deterministic. The same input through the same model version produces the same result. Caching seems obvious. The traps are in the details. The cache key has to encode both the input and the model version. Either changes and the cached output is stale. This means maintaining explicit versioning for both, tying cache keys to both, and invalidating correctly when either changes. More bookkeeping than most teams expect upfront, and more than most teams get right the first time. Model version updates require invalidating all cached outputs for that model. Input changes require invalidating for that asset. Silent stale results are worse than no cache at all, because the system behaves as if it's current when it isn't. The consistency issue from the snapshot problem applies here too. If a task fans out and some model outputs come from cache while others are freshly computed, they all need to have been computed against the same input version. A cache hit from a different input version is a consistency bug. It doesn't enforce itself. We have to check this boundary explicitly. One more thing to calibrate before assuming caching helps: for real-time pipelines where inputs change frequently, a cache hit from six hours ago may be worse than just re-running the model. We measure hit rate and staleness distribution first. Operational Visibility A task is stuck. We go to the dashboard and it shows "in progress." Eight models were dispatched. We can't tell which ones finished, which ones are still running, and which one is the reason nothing has advanced. We start digging through logs. That's what happens when we design observability for sequential pipelines and then add parallelism. In a sequential pipeline, a task is always at step N. That's enough to know where to look. In a parallel pipeline, a task might be waiting on models 3 and 7 while models 1, 2, 4, 5, 6, and 8 completed ten minutes ago. Step-level status doesn't capture that. We design observability for the parallel case from the start. We track completion state per model per task, not just overall task state. We surface which models are pending, which have completed, and which have failed for any in-flight task. We track per-model latency distributions separately. A model that's degrading shows up as a straggler well before it shows up as a failure. We alert on fan-in wait time, not just task duration. A task blocked on one model for ten minutes is a different problem from a task that's been running across all models for ten minutes, and our alerting should distinguish between the two. The target: answer "why is this task stuck?" in under two minutes from a dashboard, without touching logs. What This Enables The complexity cost of parallel orchestration is real. It's harder to build and harder to operate than sequential pipelines. But at the scale where we're running multiple ML models per task across high-volume workloads, sequential execution isn't actually the simpler option. It just has a different set of problems: latency that grows linearly with model count, slow models that hold up fast ones, and no path to the consistency guarantees production quality requires. Done right, a parallel orchestration platform starts feeling like infrastructure. Adding a new model to the fan-out is a config change. Pipeline latency stays bounded by the slowest required model regardless of how many we add. A degraded model retries independently without blocking the rest. Downstream steps always receive a consistent, complete result set. When one model's weights are updated, only that model re-runs for tasks where the others already completed. Retrofitting this onto a sequential pipeline is possible but expensive. We're essentially rebuilding the execution layer while keeping the business logic in place. The decisions that make parallelism work well (snapshot inputs, declare model criticality, track completion per model) need to be made before we have a fleet of tasks in flight against a design that doesn't support them. We should build it parallel from the start. The cost is front-loaded. It doesn't compound.