Architecture
Retrieved 2026-09-11 at pinned commit dba1be0a40aa; the native packed/fused checkpoint metadata is not used as a logical parameter total.
- Architecture
- deepseek_v41_text
- Layers
- 40
- Hidden size
- 5,120
- Attention heads
- 64
- KV heads
- 1
- Head dimension
- 512
- Vocabulary
- 129,280
- Context ceiling
- 1,048,576
- Sliding window
- 128
- RoPE theta
- 10,000
- Experts
- 384
- Experts / token
- 6
- Expert width
- 2,304
- Shared experts
- 1
Where the memory goes
Grouped-query attention shares each key/value head across 64 query heads, so the KV cache is 2% of what multi-head attention would need at the same context.
This is a multimodal checkpoint (vision); the diagram and memory sizing above cover the text decoder. Encoder/audio/image activations are not included in the KV or weight figures.