Switchyard: NVIDIA’s Open Source Routing Library

Most production AI agents still send every LLM call to the same expensive frontier model. Classification steps, simple tool calls, progress checks, and hard reasoning all hit the same endpoint. The result is unnecessary cost and latency. NVIDIA NeMo Switchyard solves this. It is an open-source routing layer (proxy + library) that sits between your agent and the models. It decides, request by request or turn by turn, which model should handle the work. In this tutorial, we'll build a working two-model router and gradually move from random routing to content-aware routing. So, let's get started. What Exactly Does Switchyard Do? A normal LLM application might look like this: Application | v GPT / Claude / Local LLM Switchyard adds a routing layer: Application | v Switchyard / \ v v Cheap Powerful Model Model The application does not need to know which upstream model ultimately serves the request. Switchyard selects the actual target and forwards the request. Let's see this practically. Step 1: Installing Switchyard For the CLI/server path, the project documentation provides a uv installation route: uv tool install "nemo-switchyard[cli,server]" Verify the installation: switchyard --version Output: switchyard 0.2.0 nemo-switchyard v0.2.0 Alternatively, the native Rust server can be installed directly with Cargo: cargo install --locked switchyard-server For this tutorial, we'll route models through OpenRouter, so export your API key: export OPENROUTER_API_KEY="your-key-here" Do not store the API key directly in the configuration file. Step 2: Understanding a Switchyard Configuration Let's start with the simplest possible setup: two models and random routing. Create a YAML file named routes.random.yaml and add: defaults: base_url: https://openrouter.ai/api/v1 api_key: ${OPENROUTER_API_KEY} routes: ab-test: type: random_routing strong: model: openai/gpt-4o weak: model: openai/gpt-4o-mini strong_probability: 0.3 rng_seed: 42 fallback_target_on_evict: weak The key setting is: strong_probability: 0.3 Switchyard interprets this as roughly: 30% -> strong model 70% -> weak model Random routing is not intelligent routing, but it is useful for A/B tests and for validating the proxy before introducing a classifier. fallback_target_on_evict is required for this route type and refers to a tier ID such as strong or weak. Step 3: Starting the Routing Server Start Switchyard with: switchyard serve \ -c routes.random.yaml \ --host 127.0.0.1 \ --port 4000 There is no --dry-run option in the tested serve CLI. Starting the server is effectively the validation step: an invalid routing bundle fails during startup. You can verify that the proxy is alive with: curl -s http://127.0.0.1:4000/health Output: {"status":"ok"} Step 4: Sending a Request Through the Router Now send an OpenAI-compatible request: curl http://localhost:4000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"ab-test","messages":[{"role":"user","content":"Explain gradient descent in simple terms."}]}' Notice this field: "model": "ab-test" Your client is not asking for a specific model (gpt-4o or gpt-4o-mini). Switchyard chooses the actual model. For this example, the request landed on the weak tier: "model": "openai/gpt-4o-mini", "usage": { "prompt_tokens": 14, "completion_tokens": 247, "cost": 0.0001503 } Response: Gradient descent is a method used in optimization to find the minimum of a function. Imagine you're on a hilly landscape, and your goal is to get to the lowest point in the valley. Here's how it works, step by step: 1) Start at a Random Point: You begin at a random location on the hill. 2) Find the Slope: You look around and determine the steepness of the hill (the gradient) at your current location. This tells you which direction is downhill. 3) Take a Step Downhill: You take a step in the direction that goes down the steepest slope. The length of your step is called the "learning rate" — if you take small steps, you're cautious, while larger steps will get you there faster but might lead you off course. 4) Repeat: You keep repeating this process, recalculating the slope and stepping down until you can't go any lower — that's the bottom of the valley or the minimum of the function. In simple terms, gradient descent is about marching down the hill step by step until you reach the lowest point. It's widely used in machine learning to adjust models so they make better predictions. Step 5: Upgrading to Intelligent Routing Random routing is great for experiments, but suppose we want this behavior: Simple request — cheap model Hard request — strong model Switchyard provides a classifier route for exactly this purpose. The classifier estimates whether the weaker model can solve the task, then applies a configured threshold. Create routes.smart.yaml and write this configuration: defaults: base_url: https://openrouter.ai/api/v1 api_key: ${OPENROUTER_API_KEY} routes: smart: type: deterministic classifier: model: openai/gpt-4o-mini strong: model: openai/gpt-4o weak: model: openai/gpt-4o-mini profile: general session_affinity: true fallback_target_on_evict: weak And start it: switchyard serve \ -c routes.smart.yaml \ --host 127.0.0.1 \ --port 4000 Now there are three roles: classifier | | predicts weak-model capability v +-------------------+ | Should weak solve?| +-------------------+ / \ / \ yes no | | v v weak strong The classifier produces a structured estimate containing a value called p_solve: an estimate of the probability that the weak model can successfully complete the request. Step 6: Testing the Smart Route Try an easy question: curl http://localhost:4000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "smart", "messages": [ { "role": "user", "content": "What is 15% of 200?" } ] }' Output: To find 15% of 200, you can multiply 200 by 0.15: 200 × 0.15 = 30 So, 15% of 200 is 30. Then try a harder one: curl http://localhost:4000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "smart", "max_tokens": 1500, "messages": [ { "role": "user", "content": "Find the race condition in a distributed job queue where workers acquire leases using non-transactional Redis operations, then propose a failure-safe redesign." } ] }' Output: In a distributed job queue system using Redis to manage and lease jobs to workers, race conditions can occur if multiple workers attempt to acquire a lease for the same job simultaneously using non-transactional operations. This can lead to multiple workers incorrectly believing they have successfully acquired the lease, resulting in duplicate processing of the same job. ### Typical Race Condition Scenario ... By incorporating these redesign elements into the distributed job queue architecture, race conditions can be significantly reduced and job leases can be handled more reliably and safely. We didn't hard-code the model selection here. Instead, the classifier determines the appropriate tier for each prompt and routes the request accordingly. If you look at the logs, you can see which model was ultimately selected for each request. Prompt Served Model Tier Latency "What is 15% of 200?" openai/gpt-4o-mini weak 1,428 ms Redis race-condition redesign openai/gpt-4o strong 4,475 ms Step 7: Routing Coding Agents Based on Their Progress Prompt difficulty is not the only useful routing signal. Consider a coding agent working for 30 turns. It may spend early turns exploring files, debugging failures, and reasoning about architecture. Later turns may simply apply an established plan or make repetitive edits. Using the strongest model for every turn wastes inference budget. Switchyard's stage_router is designed for this kind of multi-turn workload. It uses conversation and tool-result signals to decide whether a turn should go to a capable or efficient tier. You can create a configuration like this: routes: stage: type: stage_router strong: model: openai/gpt-4o weak: model: openai/gpt-4o-mini picker: efficient_first confidence_threshold: 0.5 signal_recent_window: 3 fallback_target_on_evict: weak The idea is: Agent turn | v Recent progress / failure signals | v Is extra capability useful now? / \ / \ weak strong Here, the router looks for signals associated with things such as errors, repeated unproductive behavior, exploration, and recent productive changes. The goal is to reserve the stronger model for turns where extra capability appears useful. Step 8: Escalating Only After the Weak Model Struggles Another strategy is to avoid predicting difficulty up front. Let the cheap model try first, then escalate when evidence of sustained trouble appears. The flow becomes: Request | v Weak model | v Judge result / \ okay struggling | | v v stay strong model Switchyard calls this escalation routing. You can create a configuration like this: routes: agent: type: escalation_router strong: model: openai/gpt-4o weak: model: openai/gpt-4o-mini judge: model: openai/gpt-4o-mini confirmations: 2 recent_turn_window: 28 window_message_chars: 500 fallback_target_on_evict: weak This is conceptually different from up-front deterministic classification. Deterministic/capability routing asks: How difficult does this request appear? Escalation routing asks: Is the weak model actually getting into trouble? This makes escalation useful for long-running agent sessions where task difficulty can change over time. Step 9: Measuring Whether Routing Is Actually Helping A router is only useful if it improves the quality-cost trade-off. Switchyard exposes Prometheus metrics and statistics around requests, errors, latency, tokens, and routing behavior. The project also supports structured request telemetry and optional routing logs. You can get server metrics with: curl -s http://localhost:4000/metrics | head and aggregate JSON statistics: curl -s http://localhost:4000/v1/stats | python3 -m json.tool For experiments, compare at least three runs: Configuration Purpose Always strong Quality ceiling and cost baseline Always weak Cheap baseline Switchyard router Test whether routing captures most strong-model quality at lower cost The more useful question is not whether the router was 85% accurate, but how much of the strong model's quality did routing preserve, and how much cost and latency did it reduce? For example: Strong-only: $20 92% task success Weak-only: $5 71% task success Router: $9 89% task success This tells you whether routing is economically useful. Final Thoughts As LLM systems become more agentic, the question is shifting from: Which model should I use? to: Which model should I use for this request, at this point in the workflow, under this cost budget? Switchyard is NVIDIA's attempt to turn that decision into reusable infrastructure. For a first experiment, don't jump directly into stage routing or complex agent escalation. Start with two models. Measure them independently. Use weighted random routing to verify your setup. Then introduce capability-based routing and measure whether it preserves most of the strong model's quality while moving a meaningful percentage of requests to the cheaper tier. This experiment gives you something far more useful than another LLM benchmark: a quality-versus-cost curve for your actual workload. And that is ultimately what intelligent model routing is trying to optimize. Kanwal Mehreen is a machine learning engineer and a technical writer with a profound passion for data science and the intersection of AI with medicine. She co-authored the ebook "Maximizing Productivity with ChatGPT". As a Google Generation Scholar 2022 for APAC, she champions diversity and academic excellence. She's also recognized as a Teradata Diversity in Tech Scholar, Mitacs Globalink Research Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having founded FEMCodes to empower women in STEM fields.