Accuracy, with the scope attached

This page measures how close the engine’s calculations land. Whether the inputs are right is a separate question: every figure on a model page names the publisher’s own file it was read from, at a pinned revision, so it can be checked without taking this site’s word for it. A perfect formula fed a wrong layer count is still a wrong answer, so both have to be published.

“How much memory does this need?” is four questions stacked on each other, and a score for one of them is not a score for the next. This page measures the bottom two. The top two are calculated under stated assumptions and are labelled that way everywhere they appear.

The distance between the second row and the fourth is the honest gap in this product. Closing it needs runs on physical hardware, not a better formula, and until those exist no number here is presented as one.

320 of the 327 catalogued models can be sized, including the ones with no published GGUF; the other 7 publish neither a file nor a usable parameter count, and are refused rather than guessed. To find out how much that costs in accuracy, the reconstruction is replayed against every published file we do have and compared byte for byte.

The worst column is where the honesty lives. Q2_K and Q3_K_M are worse than the rest: their mixture was wrong until published headers corrected it during the last data pass, and llama.cpp still varies those two formats with model properties this rule does not carry. Every large individual outlier is a small model, where the embedding matrices are a big share of the file and one unknowable layout choice — whether the published file ties its output tensor — moves the size by tens of percent. That case is not hidden behind a point estimate: the range widens and the interface shows the range.

Excluding those tie-ambiguous models, the remaining 110 cases have a median error of 0.008% and a worst case of 26.30%.

• The corpus is the catalogued models whose own publisher released a GGUF. It spans eleven publishers, which is what makes it evidence that the reconstruction generalises rather than fitting one publisher's conventions, but it is still not a random sample of the catalogue.
• The GGUF metadata header is modelled from the tokenizer vocabulary at forty bytes per entry, rounded outward from the 39.3 bytes per entry measured on the one artifact family whose headers are parsed.
• Q2_K and Q3_K_M carry a visibly larger error. Their mixture was corrected from published headers during this pass, but llama.cpp also varies those two formats with model properties this rule does not carry, notably on routed models.
• A model that does not reconcile with the published parameter count is never sized this way; it falls back to the block-geometry bounds.

Memory is arithmetic, and every serious tool lands close. Decode speed is where they separate — and it is the number the nearest comparable calculator publishes as its own weakest. Both figures below are each site’s own self-assessment on its own corpus, read from its own accuracy page on 1 September 2026.

What this site’s column was measured on: Measured on 39 held-out benchmarks of one model, Llama 2 7B, 87% of them on Apple hardware. It is not evidence about a mixture-of-experts model, a long context, a batch of concurrent requests, or a device generation the corpus does not contain.

Read this narrowly. Different corpora, different case counts, and each number chosen by the site reporting it — so this is not a controlled comparison and nothing here says their engine is worse. What it does show is that the figure they publish as their weakest is the figure this site holds itself to hardest, and that the cases here are scored leave-one-out: every case is predicted by a calibration fitted without it, which is the only way a self-reported error rate means anything at all.

Prefill utilisation is solved from a single published benchmark: 9.6% of the 191 TFLOPS dense matrix peak its vendor publishes, from 1024 prompt tokens at 3009.02 tok/s on one device.

That is one measurement, on one backend, so the site does not present a point estimate. It shows the physical roofline floor and the utilisation-adjusted figure as a range.

Read the range, not its upper end. The benchmark behind the utilisation is a routed mixture-of-experts model on a Vulkan build, and routing prefill through eight experts per token issues many small matrix multiplications where a dense model issues few large ones. A dense model on a CUDA or ROCm build keeps the matrix units far busier than 9.6% and should be expected to land nearer the floor than the calibrated figure. Both ends are stated for exactly that reason; a single number here would be the one that is wrong.

Prefill is compute bound and a runtime dispatches its matrix multiplications to the tensor or matrix units, so the roof has to be priced from those units and not from the headline vector figure — on an RTX 3090 the two differ by a factor of two, on an RTX 6000 Ada by a factor of four. The catalogue now carries a dense matrix peak, quoted from the vendor's own specification table, for 21 of 135 devices, and every time to first token on the site is priced against it.

What remains withheld. The other 114 devices get no time to first token, and the calculator says why on each one. Two reasons account for all of them. Some vendors publish matrix throughput only with structured sparsity — NVIDIA quotes H200 at 1,979 FP16 teraFLOPS footnoted “With sparsity”, which is exactly twice the dense rate and describes a workload with half its weights pruned; halving it here would be this site's arithmetic rather than the vendor's publication, so the estimate is withheld. Others, Apple silicon among them, publish no matrix throughput for the GPU at all. The memory roof alone is a few milliseconds; publishing it as a “lower bound” would be dressing a vacuous claim as an answer.

Qwen3-30B-A3B Q4_K_M on Radeon AI PRO R9700, llama.cpp Vulkan build 8233: 183.47 tok/s at 128 and 171.30 tok/s at 4,096.

The benchmark did not publish a GGUF hash or repository link. Filename and exact byte size match the pinned official artifact; identity is therefore founded, not verified. This single device/runtime observation is not extrapolated.

• No measured accuracy for total runtime VRAM. The runtime reserve is an uncalibrated prior and every result says so on its face.
• No measured accuracy for offloaded decode, multi-user serving, or routed decode speed. Those are models resting on published physics, not fitted to a corpus of their own.
• No time to first token on the 114 devices whose vendor publishes no dense matrix throughput — either because the only figure offered is sparsity-doubled, or because none is offered at all. No dense figure is ever inferred from a sparse one.
• No accuracy claim for the 116 of 327 catalogued models whose architecture the engine cannot reconstruct. Those are sized with block-geometry bounds and shown as a range on purpose.

Every table on this page is generated by a script that fails the build when a threshold is missed, so a regression cannot quietly widen these numbers. The method behind them is on the methodology page.