Architecture
Retrieved 2026-10-03 at pinned commit b4c84d42903a; parameter count from the safetensors index at the same revision.
- Architecture
- qwen3_5_text
- Layers
- 64
- Hidden size
- 5,120
- Attention heads
- 24
- KV heads
- 4
- Head dimension
- 256
- Feed-forward width
- 17,408
- Vocabulary
- 248,320
- Context ceiling
- 262,144
- RoPE theta
- 10,000,000
Where the memory goes
Grouped-query attention shares each key/value head across 6 query heads, so the KV cache is 17% of what multi-head attention would need at the same context.
This is a multimodal checkpoint (vision); the diagram and memory sizing above cover the text decoder. Encoder/audio/image activations are not included in the KV or weight figures.