Modern AI "Quantization": How to Run a 100GB Model in 16GB of VRAM?

When you see that a model with 70 billion parameters requires 140GB of VRAM just to load, you might assume it can only run on top-tier data centers. However, thanks to a technique called Quantization, we can now smoothly run these behemoths on ordinary consumer-grade Macs or PCs.

What is Quantization?

In simple terms, quantization is the process of reducing numerical precision.

In deep learning, model weights are typically stored as FP32 (32-bit floating-point numbers). Each FP32 number occupies 4 bytes. If a model has 70 billion parameters, loading the weights alone would require $70 \times 4 = 280\text{GB}$ of VRAM.

The goal of quantization is to convert these high-precision floating-point numbers into lower-precision formats, such as FP16 (2 bytes), INT8 (1 byte), or even INT4 (0.5 bytes). By quantizing a model to 4-bit, the same 70B model would require only about $35\text{GB}$ of VRAM to run.

Three Main Approaches to Quantization

1. Post-Training Quantization (PTQ)

This is the most commonly used method. After the model is fully trained, its weights are directly "truncated" or "mapped" from FP32 to INT8/INT4.
- Pros: Extremely fast; no retraining required.
- Cons: Significant loss in precision, especially when quantizing below 4-bit, which can lead to logical inconsistencies or degraded language capabilities.

2. Quantization-Aware Training (QAT)

This approach simulates quantization errors during the training process, allowing the model to adapt to low-precision environments while learning.
- Pros: Minimal precision loss; performance nearly matches full-precision models.
- Cons: Extremely high computational cost; requires the full training dataset and substantial computing power.

3. Advanced Compression Algorithms (e.g., GGUF / EXL2 / AWQ)

To achieve extreme compression on consumer hardware without breaking the model, various optimized algorithms have emerged:
- AWQ (Activation-aware Weight Quantization): Protects critical weights from excessive quantization by analyzing the importance of activation values.
- GGUF (llama.cpp): A format supporting hybrid CPU/GPU inference, allowing users to flexibly decide how many model layers to place in GPU memory based on available VRAM.

The Cost of Quantization: Precision vs. Size

Quantization is not a free lunch. When you reduce precision from FP16 to INT4, two things happen:
1. Weight Rounding Errors: A weight originally valued at $0.123456$ becomes $0.12$. These tiny deviations accumulate over hundreds of billions of calculations, leading to perceptible quality degradation (e.g., reduced mathematical ability, increased hallucinations).
2. Shifted Computational Overhead: Although VRAM usage decreases, the GPU must first decompress INT4 back to FP16 $\rightarrow$ perform calculations $\rightarrow$ write back results. If decompression efficiency is low, inference speed may actually decrease.

Recommendations for Practitioners

If you are choosing a model version for local deployment (such as GGUF or EXL2 versions on Hugging Face):
- Prioritizing Quality $\rightarrow$ Q4_K_M or Q5_K_M: This is currently the "sweet spot," offering significant size reduction with almost imperceptible performance loss.
- Prioritizing Speed/Minimal Memory Usage $\rightarrow$ Q3 or Q2: Recommended only when memory is extremely limited, as the model may exhibit obvious logical gaps.
- Production Environments $\rightarrow$ FP16 or BF16: If computing power is sufficient and high accuracy is critical (e.g., in healthcare or finance), stick with full-precision or half-precision versions.

Quantization technology has "liberated" AI from expensive server clusters, bringing it to everyone's desktop. It demonstrates that in the field of AI engineering, "good enough" often holds more commercial value and broader accessibility than "absolute perfection."