Qwen3-Next: Hybrid Attention + Ultra-Sparse MoE for 10x Faster Long-Context LLMs

Read Articleadded Sep 12, 2025

Qwen3-Next introduces a hybrid attention stack and ultra-sparse MoE that activates only ~3B parameters of an 80B model, achieving strong accuracy with far lower compute. It adds stability-friendly designs and native MTP for faster, more reliable inference, delivering >10x throughput beyond 32K context and competitive results up to 256K. Post-trained Instruct and Thinking variants rival or surpass larger models and are production-ready across major serving stacks.

Key Points

  • Hybrid attention (Gated DeltaNet + Gated Attention at 3:1) yields better accuracy and long-context efficiency than either approach alone, with attention enhancements (output gating, larger head dim, partial RoPE).
  • Ultra-sparse MoE: 80B total parameters with ~3B (3.7%) activated per step using 512 experts (10 routed + 1 shared) and global load balancing, improving efficiency and loss scaling.
  • Training stability optimizations (zero-centered RMSNorm with weight decay, normalized router init, attention output gating) mitigate attention sink and massive activations.
  • Native Multi-Token Prediction improves speculative decoding acceptance and end-task performance via multi-step training aligned with inference.
  • Qwen3-Next-80B-A3B matches or beats much larger models at a fraction of cost: <80% of Qwen3-30A-3B’s GPU hours and 9.3% of Qwen3-32B’s compute, with >10x throughput beyond 32K context and strong Instruct/Thinking results (including up to 256K context).

Sentiment

Cautiously positive: strong enthusiasm for the architecture and efficiency gains, tempered by concerns about generalization, consistency, and context-length trade-offs.

In Agreement

  • Qwen3-Next’s hybrid attention (Gated DeltaNet + Gated Attention) and native MTP are clever design choices that materially improve long-context throughput and decoding speed.
  • Avoiding an extra un-embedding/output head for MTP saves several GBs and is a big deal for inference efficiency.
  • MoE progress is impressive: with ~3B active params, the model can rival or beat older dense models while running substantially faster.
  • Native 256K context and validated RoPE scaling toward ~1M make it strong for long-context tasks, and it excels on ultra-long benchmarks.
  • The model can be run fully offline, with feasible performance using CPU+RAM and partial expert offload to modest GPUs.
  • Better architectures, not just bigger parameter counts, are a promising path forward; this model shows near-flagship quality at much lower cost.

Opposed

  • Some users experience hallucinations and odd dialog, suggesting quality inconsistency.
  • Critics argue Qwen models are overfit and weak on out-of-distribution generalization, struggling with guided exploration in math and code interpretation/reversal.
  • For certain workflows, 1M native context (e.g., Qwen2.5-Turbo) is preferred over Qwen3-Next’s 256K plus scaling, due to trade-offs that can hurt short-text performance.
  • Debate over MoE practicality: while partial offload can work, skeptics say swapping experts from RAM/SSD can be uselessly slow in real-world use.
  • Dense models still have advantages in knowledge depth; dismissing models like Llama 3.1 405B ignores their past near-frontier performance.
  • Observed ASCII outputs suggest memorization and instability, raising questions about robustness and UI-aware formatting.
Qwen3-Next: Hybrid Attention + Ultra-Sparse MoE for 10x Faster Long-Context LLMs