NVIDIA Nemotron 3.5 Lightning and Switchyard Guide

NVIDIA’s latest agent release is two related products, not one giant model upgrade. Nemotron 3.5 Lightning is a 30B mixture-of-experts execution model with 3B active parameters. NeMo Switchyard is the routing layer that sends each agent step to a model suited to its task, cost, latency, and policy.

The practical verdict

The architecture is complementary to a frontier planner: a larger model can plan or escalate while Lightning handles routine execution. That is a routing policy, not a guarantee that every small-model call is safe or correct.

What NVIDIA launched

Nemotron 3.5 Lightning has 30B total parameters and 3B active parameters. The model combines Mamba-2, mixture-of-experts, and attention components, supports up to a 1M-token context in the official model card, and ships with NVFP4 and BF16 artifacts. NVIDIA also publishes speculative-decoding options including DSpark, DFlash, and multi-token prediction.

The model is released under the OpenMDW-1.1 model license. That is separate from Switchyard’s software license: the NVIDIA-NeMo/Switchyard repository is Apache 2.0. Do not collapse “open model,” “open weights,” and “open-source routing software” into one license claim.

NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard deliver faster, smarter, more efficient agentic AI.
— @NVIDIAAIDev August 11, 2026

NVIDIA positions Lightning for long-running, always-on agents and specialized high-volume work. The model is available through Hugging Face, ModelScope, build.nvidia.com, OpenRouter, NVIDIA NIM, and partner inference platforms. Availability, artifact variants, and provider terms can change, so pin the exact checkpoint and runtime in a deployment record.

Switchyard is a Rust proxy and library for LLM traffic. It can keep an OpenAI- or Anthropic-facing client stable while routing to different providers and translating supported request formats. The library path lets an application embed routing decisions without handing model calls to Switchyard itself.

The current Switchyard README labels the project pre-alpha software and says it is not for production use. Treat the proxy and library as a staging or research component until NVIDIA publishes a supported production lifecycle. Pin a commit, test failure behavior, and keep a simpler fixed-target fallback.

Choose a deployment path

Goal	Starting path	Caveat
DGX Spark or low-concurrency Blackwell serving	vLLM with the DSpark draft model	NVIDIA’s recipe is hardware- and version-specific
H100/H200 batch throughput	vLLM with the documented Mamba/MoE backend and no speculative decoder	Tune batch, KV cache, and context together
Local RTX or desktop evaluation	Ollama, LM Studio, or llama.cpp GGUF	VRAM determines context and CPU offload behavior
Provider-neutral routing	Switchyard server/proxy	Add routing latency and another failure boundary
Native Rust integration	`switchyard-libsy`	Your application owns the actual model call, credentials, and retries
Fine-tuning or RL	NeMo Automodel, NeMo RL, and NeMo Gym	Training infrastructure is separate from inference deployment

For a DGX Spark-oriented vLLM path, NVIDIA’s model card documents a recipe in the vllm/vllm-openai:v0.27.1 container:

export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark

vllm serve --model "$MODEL_CKPT" \
  --moe-backend marlin \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.91 \
  --speculative_config.num_speculative_tokens 3 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --reasoning-parser nemotron_v3 \
  --speculative_config.model "$DSPARK_CKPT" \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

Treat that as a pinned reference recipe, not a universal command. vLLM, CUDA, FlashInfer, backend, driver, and checkpoint revisions all affect whether it starts and whether the reported context or throughput is achievable.

For a simpler local smoke test, NVIDIA’s model card documents an Ollama path:

ollama run nemotron-3.5-lightning

The card says Ollama dynamically chooses context based on available VRAM. Record the selected context and whether CPU offload occurs; “runs locally” is not the same as “runs entirely in VRAM.”

How Switchyard routes agent work

A router decides where a request goes; it does not make the chosen model more capable. Switchyard keeps routing logic separate from provider clients by giving targets semantic names and mapping those names to concrete endpoints.

The main routing families are:

LLM classifier: classify a request and maintain session affinity.
Stage router: infer where the agent is in a workflow; exploration and recovery can use a stronger model while steady edits use a cheaper one.
Escalation router: start cheaply and move to a stronger model after repeated errors, loops, or difficulty signals.
Tunable routers: learn workload-specific signals and blend predicted quality with cost or latency constraints.

A simple policy table is easier to audit than an opaque “best model” claim:

Agent step	Default target	Escalation condition
Tool schema selection	Small local model	Invalid arguments or repeated retries
Routine tool-result formatting	Lightning	Conflicting results or long-context synthesis
Repository exploration	Lightning or another efficient coder	Repeated test failures or architecture ambiguity
High-impact write	Strong model plus human approval	Never remove approval because routing confidence is high
Sensitive data operation	Approved private endpoint	Any unexpected data route or tool schema change

The Switchyard proxy can also record the selected model, decision rationale, token use, latency, and call outcome. Those records are essential: without them, a cheaper route can look successful while silently increasing retries or human intervention.

A router should not receive unrestricted authority merely because it chooses a model. Keep credentials in the application or gateway, allowlist targets, and make unknown model names fail closed.

Read the benchmarks correctly

NVIDIA reports up to 4× faster output than similar-sized models and reports PinchBench results where Lightning reaches 86% accuracy while completing 10,000 tasks 30% faster than a comparison model. NVIDIA and partners also report routing results such as lower cost with a small fraction of calls reaching a frontier model.

These are useful hypotheses, not universal guarantees. The results vary with:

model pool and provider prices;
prompt and output lengths;
concurrency and context length;
router type and escalation policy;
parser and tool-call behavior;
cache hits and network distance;
retries, failed tools, and human approvals.

Claim type	What to reproduce
Token throughput	Prefill, decode, batch, device, quantization, and context
Agent speed	Wall-clock task completion, tool latency, retries, and queue time
Cost reduction	Actual provider price, tokens, routing decisions, and fallback calls
Quality preservation	Completed task, valid arguments, test result, and human intervention—not only model score

The official model card publishes benchmark values for the NVFP4 checkpoint under NVIDIA’s evaluation harnesses. Preserve the distinction between an evaluation score, a vendor comparison, and your end-to-end agent success rate.

Set routing and security boundaries

Model routing changes the data path as well as the cost path. A request can move from a local endpoint to a hosted frontier model after an escalation rule fires.

Use these controls:

Keep an explicit inventory of every target endpoint, region, provider, and data policy.
Classify prompts and tool results before routing; do not send secrets to a target that cannot handle them.
Use per-target credentials and least-privilege network access.
Log route decisions without logging raw sensitive prompts by default.
Make fallback and escalation limits explicit; stop after bounded failures.
Require approval for publishing, financial actions, account changes, destructive tools, and external messaging regardless of model confidence.
Test prompt injection in tool results. A router should not treat text from a webpage or document as a routing policy.
Pin Switchyard, runtime, model, parser, and target configuration versions.

NVIDIA’s model card also advises developers to add their own safety guardrails and validate the model for the intended use case. Open weights and local execution improve control; they do not remove application responsibility.

Build a routing test plan

Start with a fixed replay set rather than live traffic:

Capture representative prompts and tool traces with sensitive values redacted.
Label the expected model class, acceptable latency, maximum cost, and escalation rule.
Run a single-frontier baseline, a single-Lightning baseline, and the Switchyard policy.
Measure completed tasks, valid tool arguments, retries, wall-clock time, token cost, route changes, and approvals.
Add adversarial cases: ambiguous intent, prompt injection, tool failure, provider timeout, malformed output, and a target that is intentionally unavailable.
Repeat after model, parser, router, or pricing changes.

A minimal acceptance table might look like this:

Test	Pass condition
Routine read-only task	Lightning completes within the cost/latency budget
Difficult task	Escalates once, then completes without an unbounded loop
Provider outage	Fails over only to an approved target
Sensitive input	Never routes to a disallowed provider
High-impact write	Human approval remains mandatory
Route explanation	Decision, target, latency, and outcome are recorded

FAQ

What is Nemotron 3.5 Lightning?

It is NVIDIA's 30-billion-parameter mixture-of-experts model with 3 billion active parameters, released for high-volume agent execution and available in NVFP4 and BF16 paths under the OpenMDW-1.1 model license.

What is NeMo Switchyard?

Switchyard is an Apache-2.0 open-source proxy and library for routing requests across model providers and endpoints. It can translate supported OpenAI and Anthropic request formats and apply configurable routing algorithms.

Can I run Nemotron 3.5 Lightning locally?

NVIDIA documents local paths including DGX Spark, GeForce RTX 5090, Jetson, Ollama, LM Studio, and llama.cpp. The NVFP4 model is still a large GPU workload; context, KV cache, and concurrency determine the practical hardware requirement.

Does routing always reduce cost without reducing quality?

No. NVIDIA and partners report cost and latency improvements under specific workloads. A router can trade quality for cost, misclassify a request, or add its own latency. Measure end-to-end task completion on your traffic.