Model Quantization

Techniques for reducing the numerical precision of AI model weights to shrink memory footprint and accelerate inference, enabling large models to run on consumer hardware.

Reading List

Under the Hood

The Hidden Technical Hazards Sabotaging Local LLM Performance

Aug 23, 2026422

Local LLM performance is sabotaged by mathematical divergences in software kernels and quantization that cause models to make incorrect token choices and logic errors.

LLM Inference Model Quantization AI Reliability On-Device AI

Under the Hood

Quantization: How to Run Massive LLMs on Your Laptop

Mar 25, 2026248

Quantization is a compression technique that makes LLMs significantly smaller and faster for local use with minimal impact on their intelligence.

On-Device AI LLM Inference AI Infrastructure Model Quantization