Architecture
Retrieved 2026-09-01 at pinned commit bf73d03a43bd.
- Architecture
- deepseek_v3
- Layers
- 26
- Hidden size
- 1,536
- Attention heads
- 32
- KV heads
- 32
- Head dimension
- 64
- Feed-forward width
- 8,960
- Vocabulary
- 128,256
- Context ceiling
- 262,144
- RoPE theta
- 100,000
- Experts
- 64
- Experts / token
- 4
- Expert width
- 1,280
- Shared experts
- 1
- Latent KV rank
- 512
Where the memory goes
Latent attention projects keys and values into a compressed rank before caching them, which is why its KV cache is a fraction of a comparable dense model.