Model Quantization Is Not Compression; It’s an Engineering Trade-off of Precision for Bandwidth

Many people understand quantization as "making large models smaller," which is only half right. For teams sensitive to inference costs, the more pertinent question is: with the same set of GPUs, how many additional requests per hour can be processed after quantization?

Calculate the Costs Before Choosing a Solution

Running a 70B model in FP16 on an A100 80G requires approximately 140GB just for weights, necessitating at least two cards. Quantizing the same model to INT8 reduces the size to about 70GB, halving the bandwidth pressure across two cards; quantizing to INT4 brings it down to roughly 35GB, allowing it to fit on a single 48GB card. The reduction in memory usage yields three direct benefits:

1. A single card can accommodate larger models, or more replicas on the same card;

2. The volume of data moved from VRAM to compute units decreases, resulting in lower latency for small batches;

3. More VRAM is available for the KV cache, enabling concurrent processing of longer contexts.

The third point is often overlooked. In production environments, the bottleneck is frequently not "insufficient compute power," but "insufficient cache space."

Trade-offs Between INT8 and INT4

INT8 typically uses per-channel or per-tensor scale factors, resulting in negligible accuracy loss, making it safe for most scenarios. INT4 commonly employs methods like GPTQ or AWQ, selecting which sensitive channels to preserve based on weight distribution. Empirical rules of thumb include:

- General chat, classification, and extraction tasks: INT4 performance is virtually indistinguishable from higher precision;

- Mathematical reasoning and code generation: INT4 shows more noticeable accuracy degradation; for sensitive tasks, it is recommended to keep at least key layers (such as attention projections and the final layer) in INT8;

- Fine-tuning scenarios: Perform LoRA (QLoRA) on the quantized format first, then decide whether to merge back to FP16 for deployment after training.

A practical testing method: Use a high-quality evaluation set of 200 samples covering your core business logic, run FP16, INT8, and INT4 respectively, and compare task metrics and first-token latency. Do not rely solely on general benchmarks like MMLU, as your business data distribution often differs significantly from benchmark distributions.

Common Pitfalls

- The first inference after quantization incurs kernel compilation overhead; ensure warm-up before stress testing;

- Different inference frameworks (vLLM, TGI, TensorRT-LLM) have inconsistent support for INT4 formats; verify compatibility for GGUF, GPTQ, AWQ, and FP8 before switching frameworks;

- FP8 requires Hopper architecture or newer hardware (H100/H200) and cannot be used on A100s; do not follow this trend blindly;

- Re-exporting a quantized model with a different quantization tool may result in a 1-2 point drop in metrics; lock the versions of your export toolchain.

The conclusion is straightforward: clearly calculate VRAM usage and concurrency capacity before quantization, and let real-world business evaluation sets speak for themselves after quantization. Converting saved VRAM into higher throughput is the correct way to balance this engineering equation.