A Go struct the model can't break
Give goinfer a Go struct and the model can only write JSON that fits it, so json.Unmarshal parses it. It fixes the shape, not whether the values are right.
Limit It doesn't check what the values say.
Read →A Go library · pure Go · no cgo
Run an open-weight LLM inside your Go program.
Import it and call it, then cross-compile it like anything else: no cgo, no daemon, no C toolchain. Constrain the output to a Go struct, and a token that would break the struct is never sampled.
go get github.com/townsendmerino/goinfer/decoder@latest github.com/townsendmerino/goinfer/tokenizer@latesttype Person struct {
Name string `json:"name"`
Age int `json:"age"`
Langs []string `json:"langs"`
Email string `json:"email,omitempty"` // omitempty (or a pointer) makes a field optional
}
g, err := constrain.GrammarFromStruct(Person{}) // struct -> JSON Schema -> grammar
mask := constrain.NewMasker(g, constrain.TokenBytes(m.Config().VocabSize, tok.TokenText), stop).StopWhenComplete()
ch, gen := m.Generate(context.Background(), ids, 128, decoder.SamplingParams{LogitProcessor: mask.Process, StopIDs: stop})
var p Person
check(json.Unmarshal([]byte(raw), &p), "unmarshal") // the shape is guaranteed; a value can still be wrongWhat will run well on your machine?
Pick the computer closest to yours. goinfer says what fits, how fast it measured on that hardware, and how that compared with Ollama in the same session.
On MacBook Pro, M1 Pro, 16 GB
On RTX 2070 SUPER, 8 GB
On Ryzen 7 3700X, CPU only
With your memory and graphics
goinferOllama, same sessiontokens per second
-
Code completion and small editsNo speed measured1.18× OllamaFrom the 2026-09-25 pre-registered sweep. The Metal decode kernels changed on 2026-09-26. 1.40× Ollama 0.91× Ollama no measurement for this machine
goinfer-chat pull qwen2.5-coder-0.5b -
Qwen2 and Qwen2.5, including the Coder models most of goinfer's numbers are measured on.No speed measured1.06× OllamaAfter the R18b kernel change, a same-session A/B against Ollama. Not the pre-registered sweep. 1.47× Ollama level with OllamaEvery pair within 3% of Ollama (1.007 to 1.015). no measurement for this machine
goinfer-chat pull Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF:q4_k_m -
Box class SNo speed measuredNo speed measuredNo speed measuredNo speed measurednot measured not measured not measured no measurement for this machine
goinfer-chat pull qwen3.5-0.8b -
General instruction following at a size that still loads in secondsNo speed measuredNo speed measurednot measured 0.90× OllamaAt its default, --quant q4k. The earlier 1.14× compared a raw completion with broken output and is withdrawn. 0.76× OllamaAt its default, --quant q4k. Ollama is CPU-forced (num_gpu 0) so both run on the CPU. no measurement for this machine
goinfer-chat pull phi3-mini-4k -
Box class MNo speed measuredNo speed measuredNo speed measuredNo speed measurednot measured not measured not measured no measurement for this machine
goinfer-chat pull qwen3-vl-2b -
A Mamba-2/attention hybrid MoE, if you want to exercise that pathNo speed measuredNo speed measuredNo speed measuredNo speed measurednot measured not measured not measured no measurement for this machine
goinfer-chat pull granite-4.0-h-tiny -
The 20-35B-class MoE this project actually validates and measuresNo speed measuredNo speed measuredgoinfer aloneOllama ran out of Metal memory at its first decode on this 16 GB Mac, two nights running, so there is no Ollama figure. goinfer's default int4mix runs as int4 on Metal. Context capped at 2,048. not measured 0.70× OllamaAt its default, int4mix. Ollama is CPU-forced (num_gpu 0) so both run on the CPU. no measurement for this machine
goinfer-chat pull gpt-oss-20b -
The 20-35B-class MoE this project has the most measurements onNo speed measuredNo speed measuredgoinfer aloneOllama's load (about 14.3 GB, mmap off) did not finish on this 16 GB Mac, so there is no Ollama figure. goinfer pages the experts from a one-time sidecar with -moe-cache-experts; its first run is slower while that cache warms. Context capped at 2,048. 1.76× Ollama, graded ambiguousEvery pair was above 1.67×, but goinfer's own runs spread 5.3%, over the 5% cap set before the run, so it is not graded ahead. goinfer keeps every expert on the card; Ollama offloads to the CPU. Different checkpoints, so this compares approaches, not kernels. Context capped at 2,048. not measured no measurement for this machine
goinfer-chat pull gemma-4-26b-a4b
Metal decode. Ollama v0.32.5 at its defaults, same machine, same session. 128 tokens of context unless noted. Measured 2026-09-25 to 2026-10-08. Against Ollama here goinfer was ahead in 2 of the 2 graded cells and behind in 0.CUDA decode. Ollama v0.32.5 at its defaults, same machine, same session. 128 tokens of context unless noted. Measured 2026-09-29. Against Ollama here goinfer was ahead in 2 of the 3 graded cells and behind in 1.CPU decode. Ollama v0.32.5 at its defaults, same machine, same session. 128 tokens of context unless noted. Measured 2026-09-29 to 2026-10-07. Against Ollama here goinfer was ahead in 0 of the 3 graded cells and behind in 3.A coarse fit estimate from download sizes, with no speeds. goinfer has not been measured on this machine, so nothing is claimed.
Every model and how it was checked
What are you here to do?
Embed it
A Go library. Ask a question, get a filled-in struct back. The Use it from Go page walks through the smallest program, struct and schema constraints, and per-field confidence.
go get github.com/townsendmerino/goinfer/decoder@latest github.com/townsendmerino/goinfer/tokenizer@latest
Try it
One file with a model inside. Download, run, ask. Nothing to install.
./goinfer-chat-0.5b-<os>-<arch>
Connect your tools
OpenAI- and Anthropic-compatible. Setup notes for Claude Code, opencode and jevx.
goinfer-serve -web -model ~/models/model.gguf
Run bigger models
Ask how a model would be placed on your machine, and whether it fits, without starting a chat.
goinfer-chat fit <model>
If you read one chapter of the book
Chapter 11 — Knowing you're right
How measurements in this repo have gone wrong, why profiling Go is not enough here, and what the codebase does instead. Written for someone who knows Go and not machine learning.
What's different
Each of these has a writeup: the problem, what goinfer does, what was measured, and what it doesn't do.
Tool calls that can't come out malformed
When a model starts a tool call, goinfer holds it to valid JSON, a tool you supplied, and that tool's argument schema. It does not pick the tool for the model.
Limit It doesn't choose the right tool.
Read →Stop means stop
Cancel one generation by id, or halt them all from an HTTP call, a file or a signal. Control sits on its own Unix socket, away from the API.
Limit It doesn't sandbox what a client does with text it already has.
Read →How sure was it?
Each enum, boolean and integer field of a constrained JSON answer comes back with the model's own probability over the answers the schema allowed.
Limit It isn't the probability that the answer is right.
Read →Decisions without generating
goinfer answers a choice among a few options from one read of the prompt, with no generation. It ships; its accuracy on a real model is not yet measured.
Limit It hasn't been graded on a real model.
Read →Checked against the reference
Each model family is compared with HuggingFace's own implementation, the result is printed on the Models page, and the label says how strong the check was.
Limit It doesn't cover every quantization on every machine.
Read →The book
New to how this works?
11 chapters on how a language model actually runs, for someone who knows Go and not machine learning. Read it here.