Architecture
Retrieved 2026-09-01 at pinned commit 58e77dd48a65.
- Architecture
- llama
- Layers
- 40
- Hidden size
- 5,120
- Attention heads
- 40
- KV heads
- 40
- Feed-forward width
- 13,824
- Vocabulary
- 32,000
- Context ceiling
- 4,096
Where the memory goes
Every attention head caches its own keys and values, so the KV cache grows at the full multi-head rate with context.