Best local LLMs for 24 GB VRAM

For models under 36B parameters, yes: all 211 in the catalogue fit entirely at Q4_K_M, the format most people download. None of the 51 models in the 36B and larger band fits. All 214 that fit answer at 30 tokens a second or more on the NVIDIA GeForce RTX 4090, which is faster than most people read. A longer conversation needs more memory than the 8,192 tokens counted here, and a smaller format than Q4_K_M needs less; the calculator sizes either.

The same models fit on every one of them. What changes is speed, which follows memory bandwidth: a card with twice the bandwidth decodes roughly twice as fast.

Which card to buy, at every memory size →

Store links may pay us a commission. They never decide which models are listed — the memory arithmetic does.

Widely downloaded models that need more than 24 GB at Q4_K_M, and the smallest memory size that holds each one entirely. Each link shows what 24 GB can still do with it: a smaller format, or part of the model in system RAM at a lower speed.

3 more models fit in 32 GB. Best local LLMs for 32 GB →