Reducing LLM Inference Costs: It’s Not About Layoffs, It’s About Queuing

When many people think about cost reduction, their first instinct is to lay off staff. For model teams, the immediate reaction is often to slash compute bills. These two approaches overlap by about 80%. Today, I’ll discuss the three most practical levers on the inference side—strategies that small teams like ours have actually implemented and can proudly report in monthly reviews.

Lever 1: KV Cache – Turning Redundant Computation into Lookups

When generating the next token, the model needs to attend to the Keys and Values of all preceding tokens. A naive implementation recalculates these at every step, causing costs to scale quadratically with sequence length. KV Cache stores these precomputed results, allowing subsequent steps to simply look them up. This reduces the per-step overhead from O(n²) to O(n).

In practice, keep two points in mind. First, simply adding more VRAM isn’t always the solution. For a 70B model with an 8K context window, a single 80GB GPU might not hold the entire KV Cache. In such cases, prioritize reducing concurrency rather than cutting context length. Second, the cache hit rate for multi-turn conversations depends on your session management strategy. Setting the session key to `session_id` instead of `request_id` can boost hit rates from 30% to over 80%, directly slashing your bill.

Lever 2: Quantization – The Explicit Price Tag of Trading Precision for Bytes

INT8 uses half the VRAM and bandwidth of FP16, and INT4 cuts it in half again. While this sounds straightforward, there’s a pitfall in actual deployment: quantization loss is uneven. Weight layers are less affected, but areas near the softmax in attention mechanisms suffer more. At low bit-widths, performance drops on long-tail tasks (such as math and coding) are often more severe than average metrics suggest.

Our approach involves layer-specific decisions: we use INT4 for the main weights but retain INT8 for the output layer and attention projections. This allows a 70B model to run on 48GB GPUs. With this hybrid scheme, performance degradation stays within 0.5 percentage points, making it imperceptible to business users. The savings aren’t just about using fewer cards; it’s about fitting twice the concurrency.

Lever 3: Batching – Assembling Fragmented Requests into Bulk Units

Low GPU utilization during inference is usually due to fragmented requests. Individual requests occupy GPU time slices for short durations, resulting in high proportional overhead for startup and context switching. Continuous batching allows new requests to be inserted into running batches dynamically, keeping the GPU fully utilized.

There’s a simple criterion to judge whether this is worthwhile: your P95 latency requirement. If your business can tolerate responses within 2 seconds, batching is pure profit. If you require responses within 200 milliseconds, batching might push up your P95 latency, in which case adding more GPUs is a better option. In our customer service scenario, with a P95 requirement of 1.5 seconds, enabling batching increased GPU utilization from 34% to 71%, reducing the cost per token by 40%.

How to Combine These Three Levers

The order matters: enable KV Cache first (zero loss, pure gain), then apply quantization (some loss, requires evaluation), and finally tune batching (incurs latency costs, depends on business tolerance). After implementing all three, small teams can typically reduce their inference bills to 30% of the original cost. Don’t rush to reinvest the savings into hardware; instead, check if any requests can be handled by smaller models or rule-based systems. That’s where the bigger opportunities lie.