# NVIDIA Nemotron 3.5 Lightning and Switchyard Guide

> Deploy Nemotron 3.5 Lightning and route agent steps across specialized models with NVIDIA NeMo Switchyard, while measuring cost, latency, and quality.

- **Published**: 2026-08-13
- **Category**: AI Infrastructure
- **URL**: https://agentpedia.codes/blog/nvidia-nemotron-3-5-switchyard-agent-routing-guide

---

NVIDIA's latest agent release is two related products, not one giant model upgrade. **Nemotron 3.5 Lightning** is a 30B mixture-of-experts execution model with 3B active parameters. **NeMo Switchyard** is the routing layer that sends each agent step to a model suited to its task, cost, latency, and policy.

> **Note callout**

**Practical verdict:** use Lightning as a high-volume workhorse for tool calls, coding-agent steps, classification, and specialized execution. Use Switchyard when a single model is either too expensive for routine turns or too weak for difficult ones. Treat NVIDIA's speed, cost, and accuracy figures as vendor or partner results until they reproduce on your own traces.

## The practical verdict

The architecture is complementary to a frontier planner: a larger model can plan or escalate while Lightning handles routine execution. That is a routing policy, not a guarantee that every small-model call is safe or correct.

## What NVIDIA launched

Nemotron 3.5 Lightning has **30B total parameters and 3B active parameters**. The model combines Mamba-2, mixture-of-experts, and attention components, supports up to a 1M-token context in the official model card, and ships with NVFP4 and BF16 artifacts. NVIDIA also publishes speculative-decoding options including DSpark, DFlash, and multi-token prediction.

The model is released under the **OpenMDW-1.1 model license**. That is separate from Switchyard's software license: the [NVIDIA-NeMo/Switchyard repository](https://github.com/NVIDIA-NeMo/Switchyard) is Apache 2.0. Do not collapse "open model," "open weights," and "open-source routing software" into one license claim.

> NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard deliver faster, smarter, more efficient agentic AI.
>
> -- [@NVIDIAAIDev, August 11, 2026](https://x.com/NVIDIAAIDev/status/2087172614896988545)

NVIDIA positions Lightning for long-running, always-on agents and specialized high-volume work. The model is available through Hugging Face, ModelScope, build.nvidia.com, OpenRouter, NVIDIA NIM, and partner inference platforms. Availability, artifact variants, and provider terms can change, so pin the exact checkpoint and runtime in a deployment record.

Switchyard is a **Rust proxy and library for LLM traffic**. It can keep an OpenAI- or Anthropic-facing client stable while routing to different providers and translating supported request formats. The library path lets an application embed routing decisions without handing model calls to Switchyard itself.

The current Switchyard README labels the project **pre-alpha software** and says it is **not for production use**. Treat the proxy and library as a staging or research component until NVIDIA publishes a supported production lifecycle. Pin a commit, test failure behavior, and keep a simpler fixed-target fallback.

## Choose a deployment path

| Goal | Starting path | Caveat |
| --- | --- | --- |
| DGX Spark or low-concurrency Blackwell serving | vLLM with the DSpark draft model | NVIDIA's recipe is hardware- and version-specific |
| H100/H200 batch throughput | vLLM with the documented Mamba/MoE backend and no speculative decoder | Tune batch, KV cache, and context together |
| Local RTX or desktop evaluation | Ollama, LM Studio, or llama.cpp GGUF | VRAM determines context and CPU offload behavior |
| Provider-neutral routing | Switchyard server/proxy | Add routing latency and another failure boundary |
| Native Rust integration | `switchyard-libsy` | Your application owns the actual model call, credentials, and retries |
| Fine-tuning or RL | NeMo Automodel, NeMo RL, and NeMo Gym | Training infrastructure is separate from inference deployment |

For a DGX Spark-oriented vLLM path, NVIDIA's model card documents a recipe in the `vllm/vllm-openai:v0.27.1` container:

```bash
export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark

vllm serve --model "$MODEL_CKPT" \
  --moe-backend marlin \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.91 \
  --speculative_config.num_speculative_tokens 3 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --reasoning-parser nemotron_v3 \
  --speculative_config.model "$DSPARK_CKPT" \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice
```

Treat that as a pinned reference recipe, not a universal command. vLLM, CUDA, FlashInfer, backend, driver, and checkpoint revisions all affect whether it starts and whether the reported context or throughput is achievable.

For a simpler local smoke test, NVIDIA's model card documents an Ollama path:

```bash
ollama run nemotron-3.5-lightning
```

The card says Ollama dynamically chooses context based on available VRAM. Record the selected context and whether CPU offload occurs; "runs locally" is not the same as "runs entirely in VRAM."

## How Switchyard routes agent work

A router decides where a request goes; it does not make the chosen model more capable. Switchyard keeps routing logic separate from provider clients by giving targets semantic names and mapping those names to concrete endpoints.

The main routing families are:

- **LLM classifier:** classify a request and maintain session affinity.
- **Stage router:** infer where the agent is in a workflow; exploration and recovery can use a stronger model while steady edits use a cheaper one.
- **Escalation router:** start cheaply and move to a stronger model after repeated errors, loops, or difficulty signals.
- **Tunable routers:** learn workload-specific signals and blend predicted quality with cost or latency constraints.

A simple policy table is easier to audit than an opaque "best model" claim:

| Agent step | Default target | Escalation condition |
| --- | --- | --- |
| Tool schema selection | Small local model | Invalid arguments or repeated retries |
| Routine tool-result formatting | Lightning | Conflicting results or long-context synthesis |
| Repository exploration | Lightning or another efficient coder | Repeated test failures or architecture ambiguity |
| High-impact write | Strong model plus human approval | Never remove approval because routing confidence is high |
| Sensitive data operation | Approved private endpoint | Any unexpected data route or tool schema change |

The Switchyard proxy can also record the selected model, decision rationale, token use, latency, and call outcome. Those records are essential: without them, a cheaper route can look successful while silently increasing retries or human intervention.

A router should not receive unrestricted authority merely because it chooses a model. Keep credentials in the application or gateway, allowlist targets, and make unknown model names fail closed.

## Read the benchmarks correctly

NVIDIA reports up to **4x faster output** than similar-sized models and reports PinchBench results where Lightning reaches 86% accuracy while completing 10,000 tasks 30% faster than a comparison model. NVIDIA and partners also report routing results such as lower cost with a small fraction of calls reaching a frontier model.

These are useful hypotheses, not universal guarantees. The results vary with:

- model pool and provider prices;
- prompt and output lengths;
- concurrency and context length;
- router type and escalation policy;
- parser and tool-call behavior;
- cache hits and network distance;
- retries, failed tools, and human approvals.

| Claim type | What to reproduce |
| --- | --- |
| Token throughput | Prefill, decode, batch, device, quantization, and context |
| Agent speed | Wall-clock task completion, tool latency, retries, and queue time |
| Cost reduction | Actual provider price, tokens, routing decisions, and fallback calls |
| Quality preservation | Completed task, valid arguments, test result, and human intervention--not only model score |

The official model card publishes benchmark values for the NVFP4 checkpoint under NVIDIA's evaluation harnesses. Preserve the distinction between an evaluation score, a vendor comparison, and your end-to-end agent success rate.

## Set routing and security boundaries

Model routing changes the data path as well as the cost path. A request can move from a local endpoint to a hosted frontier model after an escalation rule fires.

Use these controls:

- Keep an explicit inventory of every target endpoint, region, provider, and data policy.
- Classify prompts and tool results before routing; do not send secrets to a target that cannot handle them.
- Use per-target credentials and least-privilege network access.
- Log route decisions without logging raw sensitive prompts by default.
- Make fallback and escalation limits explicit; stop after bounded failures.
- Require approval for publishing, financial actions, account changes, destructive tools, and external messaging regardless of model confidence.
- Test prompt injection in tool results. A router should not treat text from a webpage or document as a routing policy.
- Pin Switchyard, runtime, model, parser, and target configuration versions.

NVIDIA's model card also advises developers to add their own safety guardrails and validate the model for the intended use case. Open weights and local execution improve control; they do not remove application responsibility.

## Build a routing test plan

Start with a fixed replay set rather than live traffic:

1. Capture representative prompts and tool traces with sensitive values redacted.
2. Label the expected model class, acceptable latency, maximum cost, and escalation rule.
3. Run a single-frontier baseline, a single-Lightning baseline, and the Switchyard policy.
4. Measure completed tasks, valid tool arguments, retries, wall-clock time, token cost, route changes, and approvals.
5. Add adversarial cases: ambiguous intent, prompt injection, tool failure, provider timeout, malformed output, and a target that is intentionally unavailable.
6. Repeat after model, parser, router, or pricing changes.

A minimal acceptance table might look like this:

| Test | Pass condition |
| --- | --- |
| Routine read-only task | Lightning completes within the cost/latency budget |
| Difficult task | Escalates once, then completes without an unbounded loop |
| Provider outage | Fails over only to an approved target |
| Sensitive input | Never routes to a disallowed provider |
| High-impact write | Human approval remains mandatory |
| Route explanation | Decision, target, latency, and outcome are recorded |

## FAQ

## FAQ

### What is Nemotron 3.5 Lightning?

It is NVIDIA's 30-billion-parameter mixture-of-experts model with 3 billion active parameters, released for high-volume agent execution and available in NVFP4 and BF16 paths under the OpenMDW-1.1 model license.

### What is NeMo Switchyard?

Switchyard is an Apache-2.0 open-source proxy and library for routing requests across model providers and endpoints. It can translate supported OpenAI and Anthropic request formats and apply configurable routing algorithms.

### Can I run Nemotron 3.5 Lightning locally?

NVIDIA documents local paths including DGX Spark, GeForce RTX 5090, Jetson, Ollama, LM Studio, and llama.cpp. The NVFP4 model is still a large GPU workload; context, KV cache, and concurrency determine the practical hardware requirement.

### Does routing always reduce cost without reducing quality?

No. NVIDIA and partners report cost and latency improvements under specific workloads. A router can trade quality for cost, misclassify a request, or add its own latency. Measure end-to-end task completion on your traffic.


## Sources and links

- [NVIDIA: Nemotron 3.5 Lightning and NeMo Switchyard](https://blogs.nvidia.com/blog/nemotron-lightning-switchyard-rtx-dgx/)
- [NVIDIA Technical Blog: Nemotron 3.5 Lightning](https://developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-accurate-specialized-task-execution-for-long-running-agents/)
- [NVIDIA Technical Blog: NeMo Switchyard routing](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/)
- [Nemotron 3.5 Lightning model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4)
- [NVIDIA-NeMo/Switchyard repository](https://github.com/NVIDIA-NeMo/Switchyard)
- [NVIDIA NeMo Switchyard getting started](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/getting_started.md)
- [AgentPedia: Liquid LFM2.5 local-agent guide](/blog/liquid-lfm2-5-2-6b-on-device-agent-guide)
- [AgentPedia: Shepherd reversible execution traces guide](/blog/shepherd-reversible-agent-execution-traces-guide)
- [AgentPedia: OmniRoute gateway routing guide](/blog/omniroute-ai-gateway-routing-setup-guide)


---

- [All articles](https://agentpedia.codes/blog)