# Inkling-Small Guide: Open Weights, Hardware and Agent Benchmarks

> Evaluate Inkling-Small for coding and tool use with exact checkpoint memory, deployment choices, benchmark caveats, and a safe smoke-test plan.

- **Published**: 2026-07-31
- **Category**: AI Infrastructure
- **URL**: https://agentpedia.codes/blog/inkling-small-open-weights-agent-model-guide

---

> **Important callout**

**Bottom line:** Inkling-Small is worth evaluating when you need a multimodal, open-weights model for coding or tool use and can supply at least 180 GB of aggregate VRAM. The Hugging Face repositories label the released materials Apache-2.0, while the provider model card separately subjects model use to the mutable Thinking Machines Model Acceptable Use Policy. Review both before deployment. Do not size it like a 12B model: it has 276B total parameters, and its headline agent scores are vendor-reported results produced with provider-specific harnesses.

[Thinking Machines released Inkling-Small](https://thinkingmachines.ai/news/inkling-small/) on July 30, 2026. The model combines text, image and audio input, text output, variable reasoning effort, and a context window of up to one million tokens. This guide turns the launch and model card into an evaluation plan. It does not independently reproduce the provider's benchmarks or validate every serving recipe.

## The practical verdict

Inkling-Small fits a fairly specific deployment envelope. It is more accessible than the provider's larger Inkling model, but it is not a workstation model. Its sparse Mixture-of-Experts design activates about 12B parameters per token while retaining 276B total parameters. Sparse activation lowers per-token compute; it does not remove the need to store the expert weights.

Use this decision table before downloading a checkpoint:

| Your constraint | Recommended first path | Why | Stop condition |
| --- | --- | --- | --- |
| One B300, SM100 or newer | NVFP4 W4A4 | Provider-documented 180 GB minimum path | Runtime lacks the required W4A4 kernels |
| Two H200 GPUs | NVFP4 W4A16 | Provider-documented alternative to W4A4 | Aggregate free VRAM falls below model, KV cache and runtime needs |
| Four B300 or eight H200 GPUs | BF16 | Closest released precision to the training numerics | Cost or latency misses the workload target |
| Smaller GPU estate | Hosted evaluation first | Avoids an unsupported local fit claim | Provider privacy, residency or cost terms do not fit |
| Need redistribution or modification | Pin the weights, Apache metadata and Model AUP revision | Treat the license metadata and separate use policy as one review surface | Legal or policy review cannot approve both, or required training provenance is unavailable |
| Need independently comparable coding scores | Re-run in your own harness | Published scores change harness and trajectory settings | You cannot hold task set, tools, budget and scoring constant |

For nearby deployment comparisons, the [Poolside Laguna S 2.1 guide](/blog/poolside-laguna-s-2-1-developer-guide) shows how another sparse coding model's active-parameter label differs from its real memory footprint.

## Model and license boundaries

The [official model card](https://thinkingmachines.ai/model-card/inkling-small/) is the best source for architecture facts:

| Property | Provider disclosure | Operational implication |
| --- | --- | --- |
| Architecture | 42-layer decoder-only sparse MoE | Runtime must support the custom multimodal/MoE architecture |
| Parameters | 276B total, 12B active | Capacity-plan for the full checkpoint, not 12B |
| Routing | 6 of 256 routed experts plus 2 shared experts | Sparse compute does not imply sparse storage |
| Inputs | UTF-8 text, pixel images, 16 kHz WAV audio | Build modality-specific validation and size limits |
| Output | UTF-8 text | No native audio or image output |
| Context | Up to 1M tokens | Treat 1M as a limit to test, not a default request size |
| Numerics | BF16, MXFP8 and NVFP4 support | Released BF16 and NVFP4 paths have different hardware requirements |
| Legal and policy terms | Hugging Face repositories label materials Apache-2.0; the model card separately links the Model AUP | Review both the license metadata and the current, mutable use restrictions |

The Hugging Face pages identify the repositories as Apache-2.0, but that metadata is not the whole review surface. The provider's model card separately links the [Thinking Machines Model Acceptable Use Policy](https://thinkingmachines.ai/model-acceptable-use-policy/), which says it governs access to and use of the model weights, parameters, associated materials and modified versions and may be updated. Do not state unqualified Apache rights or assume the AUP is irrelevant; counsel and policy owners should review the pinned repository materials together with the AUP in effect for the intended use.

The Hugging Face interface may also show generated parameter metadata that disagrees with the provider's disclosed count. Use **276B total / 12B active** from the model card as authoritative. Interface metadata is useful for discovery, not for overriding a provider architecture statement.

"Open weights" also has a narrower meaning than "the complete system is reproducible." Thinking Machines describes training data as a mixture of public, third-party and synthetic or augmented sources. The release does not provide a complete training corpus or a recipe that lets a third party recreate the checkpoint bit-for-bit.

## Choose a deployment path

The first choice is not SGLang versus vLLM. It is whether your evaluation needs weight custody.

| Path | Best for | Main evidence you gain | Main risk |
| --- | --- | --- | --- |
| Hosted playground/API | Prompt, modality and workflow screening | Whether the model is promising on representative tasks | Hosted behavior may differ from the checkpoint/runtime you later deploy |
| NVFP4 self-hosting | Cost-conscious controlled serving | Real throughput, memory and quantized quality on owned infrastructure | Kernel and architecture compatibility |
| BF16 self-hosting | Precision-sensitive evaluation | Closest available baseline for measuring quantization effects | Large cluster cost and operational complexity |
| Fine-tuning through Tinker | Task adaptation experiments | Whether supervised or reinforcement tuning changes task success | Platform-specific training and data-governance boundary |

A sensible sequence is hosted screening, NVFP4 validation, then BF16 only if quantization deltas matter enough to justify the cluster. Keep the prompt set, tool schema and grading rubric stable across those stages.

The model card names **SGLang, vLLM, TokenSpeed, Unsloth and Hugging Face** as supported deployment frameworks or recipes. That list establishes provider-described routes; it does not establish equivalent feature coverage, performance or security defaults across them.

## Hardware and memory without the 12B trap

Thinking Machines publishes minimum aggregate VRAM figures rather than a promise about usable context or batch size:

| Checkpoint / mode | Provider minimum | Example hardware | Important boundary |
| --- | ---: | --- | --- |
| BF16 | 600 GB aggregate VRAM | 4x NVIDIA B300 | Leaves an implementation-dependent margin above raw weight storage |
| BF16 | 600 GB aggregate VRAM | 8x NVIDIA H200 | Requires multi-GPU communication and a matching parallelism plan |
| NVFP4 W4A4 | 180 GB aggregate VRAM | 1x NVIDIA B300 | Requires SM100 or newer architecture |
| NVFP4 W4A16 | 180 GB aggregate VRAM | 2x NVIDIA H200 | Activation precision and kernels differ from W4A4 |

These are lower bounds for a supported configuration, not a batch-and-context guarantee. Loaded weights compete with KV cache, modality preprocessing, communication buffers, CUDA graphs, allocator fragmentation and the inference engine itself.

Use this worksheet before provisioning:

```text
checkpoint mode:
GPU model and count:
aggregate physical VRAM:
free VRAM after driver/runtime:
target input tokens:
maximum generated tokens:
concurrent sequences:
image/audio inputs:
tensor/expert parallelism:
measured peak VRAM:
failure margin:
```

Start with a short text-only request and one sequence. Add tools, modalities, context and concurrency one axis at a time. A server that loads successfully but OOMs during a long tool trajectory is not a valid deployment.

## Serving framework boundaries

The official [vLLM Inkling-Small recipe](https://recipes.vllm.ai/thinkingmachines/Inkling-Small) requires a nightly vLLM build with Inkling support. Its B300 NVFP4 W4A4 starting configuration is:

```bash
# Official B300 W4A4 starting configuration; not executed for this article.
export VLLM_USE_V2_MODEL_RUNNER=1
export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1

vllm serve thinkingmachines/Inkling-Small-NVFP4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --tensor-parallel-size 1 \
  --kernel-config.enable_flashinfer_autotune=False \
  --trust-remote-code
```

This command was not executed for the article. The same recipe says H200 uses TP2 in W4A16 mode, with the NVFP4 weights dequantized to BF16 on the fly. Before production, pin the model revision and nightly runtime build, bind the service to a private interface, require authentication, and set explicit limits for input size, output tokens and concurrency. Confirm that the chosen runtime correctly preserves reasoning/tool state and supports the modality mix you need.

| Check | Why it matters | Evidence to retain |
| --- | --- | --- |
| Immutable model revision | Upstream files can change | Commit hash and artifact digests |
| Runtime and kernel versions | NVFP4 support is hardware- and version-sensitive | Container digest and GPU driver inventory |
| Chat template and tool parser | Tool-use quality can change without weight changes | Rendered prompt fixture and schema |
| Reasoning-effort control | Published evals generally use effort 0.99 | Request configuration and returned usage |
| Context/concurrency limits | "Up to 1M" is not a capacity plan | Load-test matrix and peak VRAM |
| Modal preprocessing | Images and audio create separate failure modes | Accepted formats, limits and rejection tests |

## Read the agent benchmarks as system results

Thinking Machines reports its own evaluations and also imports some external scores. The key coding results are system measurements, not model-only constants.

| Model | Benchmark | Score | Provider configuration | What not to infer |
| --- | --- | ---: | --- | --- |
| Inkling-Small | SWE-bench Verified | 80.2% | Bash-only harness; coding evals use a 256K trajectory; effort generally 0.99 and temperature 1.0 | Not a reproduction in the official or your production agent harness |
| Inkling-Small | SWE-bench Pro public | 55.9% | Published launch evaluation settings | Does not predict your repository mix or review quality |
| Inkling-Small | Terminal-Bench 2.1, best harness | 64.7% | Internal coding harness; contaminated web-search solutions assigned zero | Not directly comparable to a different scaffold or tool policy |
| Inkling-Small | SciCode | 48.7% | Launch table, with source-specific evaluation details | Not proof of scientific software correctness |

The provider says all its evals run at effort `0.99` and temperature `1.0`, and coding evals allow a `256K` maximum-token trajectory. Long trajectories can materially change success, latency and cost. The Terminal-Bench footnote also says a small number of solutions found to be contaminated through web search were scored zero. That is useful disclosure, but it does not turn the overall result into an independent audit.

For a stronger internal study, use the validation structure in the [coding agents for scientific software guide](/blog/coding-agents-scientific-software-validation-guide). For security-oriented agent claims, the [Fugu orchestration and security benchmark guide](/blog/fugu-cyber-orchestration-security-benchmarks) explains why model, scaffold, permissions and containment must be reported together.

## Run a bounded smoke test

The following is a **verification plan**, not tested output from this article.

1. Pin the exact checkpoint, tokenizer, processor and serving container.
2. Start with text-only generation at low context and one concurrent request.
3. Record startup time, idle VRAM, first-token latency, tokens per second and peak VRAM.
4. Submit a deterministic repository-navigation task with no write tools.
5. Add a read-only shell tool and verify arguments against an allowlist.
6. Run an image-understanding fixture, then a short 16 kHz WAV fixture.
7. Increase reasoning effort and record output/reasoning-token changes.
8. Run the same task at 32K, 128K and your intended context; do not jump straight to 1M.
9. Introduce a recoverable tool failure and check retry, timeout and state preservation.
10. Only then allow writes in a disposable repository with tests and diff review.

Use a compact result sheet:

| Case | Expected result | Failure signal | Decision |
| --- | --- | --- | --- |
| Text generation | Valid, relevant text | Template tokens leak or output is empty | Fix template/runtime |
| Tool schema | Valid allowed call | Hallucinated tool or unsafe arguments | Add schema/policy gate |
| Image input | Correct fixture facts | Unsupported size or preprocessing drift | Bound formats |
| Audio input | Correct short transcript/reasoning | Sampling or duration failure | Normalize audio |
| Long context | Retains required earlier fact | Recall falls or VRAM spikes | Lower operational limit |
| Interrupted tool | Recovers without duplicate side effect | Replays mutation | Add idempotency |

## Deploy with bounded authority

The model card reports internal safety evaluations and external red teaming. Those provider-reported results do not replace deployment controls, especially when the model can call tools.

- Run inference and tools in separate, least-privilege security boundaries.
- Keep credentials out of prompts, model-visible files and tool output.
- Default network egress to denied; allow only task-required destinations.
- Require human approval for destructive, financial, identity or production actions.
- Validate tool arguments independently of model text.
- Cap steps, wall time, output tokens, concurrency and spend per run.
- Make mutation tools idempotent or attach unique operation keys.
- Log prompts, tool decisions, executed actions and results with appropriate redaction.
- Test prompt injection through repository files, web results, images and audio.
- Provide a server-side kill switch that revokes active credentials and stops workers.
- Re-run the smoke suite after any checkpoint, runtime, prompt or tool-schema change.

Open weights move some controls into your hands. They do not remove the risks of unsafe permissions, untrusted inputs or poorly contained tools.

## Adopt, evaluate or wait

**Evaluate now** if you have a documented 180 GB or larger GPU path, need multimodal input plus agentic tool use, and can run a controlled comparison against an existing system.

**Adopt after evidence** if the pinned checkpoint meets your task-success, latency, memory, safety and operational-cost thresholds under the same harness you will deploy.

**Wait or use hosted access** if your hardware is smaller than the provider minimum, you need an independently reproduced leaderboard result, or your runtime cannot yet support the required NVFP4 mode. A model that fits only by relying on an undocumented quantization is a separate artifact with a separate quality and licensing review.

## FAQ

### Is Inkling-Small open source?

Inkling-Small is an open-weights model whose Hugging Face repositories label the released materials Apache-2.0. The provider model card separately says model use is subject to the mutable Thinking Machines Model Acceptable Use Policy, so review both surfaces for the intended use. Open weights also do not disclose the full training data or reproduce the training process.

### How much GPU memory does Inkling-Small need?

Thinking Machines says the BF16 checkpoint needs at least 600 GB of aggregate VRAM, with 4x B300 or 8x H200 as examples. The NVFP4 checkpoint needs at least 180 GB: W4A4 can run on one B300 with SM100 or newer, while W4A16 is documented for two H200 GPUs.

### Can Inkling-Small run on a single H200?

Not in either provider-documented minimum configuration. Thinking Machines documents two H200 GPUs for the NVFP4 W4A16 path and eight H200 GPUs for BF16. A one-H200 claim would need a separately documented quantization and runtime recipe.

### Does Inkling-Small support images and audio?

Yes. The model card lists text, image and 16 kHz WAV audio as inputs and text as output. It recommends image dimensions between 40 and 4096 pixels and audio clips ideally under two minutes.

### Are the Inkling-Small coding benchmarks independently reproduced?

No independent reproduction is established here. The 80.2% SWE-bench Verified result uses Thinking Machines' bash-only harness, and the 64.7% Terminal-Bench 2.1 result uses its internal coding harness with stated contamination handling.

### Which parameter count should I use when sources disagree?

Use the provider model card's 276 billion total and 12 billion active parameters. Hugging Face interface metadata can be derived or incomplete, so it should not override the provider's architecture disclosure.


---

[Join the Agentpedia newsletter](https://agentpedia.codes/blog)
[Browse related Agentpedia articles](https://agentpedia.codes/blog)

## Official sources

This guide uses first-party launch, model-card and model-repository sources. Benchmark values remain provider-reported unless the source row says otherwise.

### Thinking Machines Lab

- [Introducing Inkling-Small](https://thinkingmachines.ai/news/inkling-small/) -- July 30 release, architecture summary, capabilities, evaluation settings and benchmark table
- [Inkling-Small model card](https://thinkingmachines.ai/model-card/inkling-small/) -- Apache license label, linked Model AUP, modalities, architecture, minimum hardware, distribution and safety disclosures
- [Thinking Machines Model Acceptable Use Policy](https://thinkingmachines.ai/model-acceptable-use-policy/) -- separate use restrictions and update terms for model materials
- [Inkling-Small on Hugging Face](https://huggingface.co/thinkingmachines/Inkling-Small) -- BF16 repository, Apache-2.0 metadata and framework starting examples
- [Inkling-Small NVFP4 on Hugging Face](https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4) -- quantized repository and framework starting examples
- [Official vLLM Inkling-Small recipe](https://recipes.vllm.ai/thinkingmachines/Inkling-Small) -- nightly requirement and architecture-specific B300, B200 and H200 configurations

### Related AgentPedia guides

- [Poolside Laguna S 2.1 serving and benchmark guide](/blog/poolside-laguna-s-2-1-developer-guide)
- [Validating coding agents for scientific software](/blog/coding-agents-scientific-software-validation-guide)
- [Fugu orchestration security and benchmark guide](/blog/fugu-cyber-orchestration-security-benchmarks)


---

- [All articles](https://agentpedia.codes/blog)