How to Calculate LLM API Costs: Getting the Token Math Right

When building LLM applications, the most frequently asked question is, "How much will this feature cost per day?" The answer hinges on one variable: how many tokens each request consumes. This article breaks down the calculation and highlights common pitfalls where mistakes are easily made.

What Exactly Is a Token?

Models don't charge by word count; they charge by token. A token is the smallest unit of text processed by the model. As a general rule:

- In English, 1 token is approximately 3/4 of a word.

- In Chinese, due to different tokenization methods, 1 token typically corresponds to 1 to 1.5 Chinese characters.

The exact ratio depends on the tokenizer used by your specific model, but these approximations are sufficient for estimation: 1,000 Chinese characters roughly equal 700 to 1,000 tokens.

One easily overlooked point: everything in the context counts as tokens. User messages, system prompts, tool function definitions, retrieved document snippets, and conversation history all contribute to the input token count. In many applications, the actual input token count is 5 to 10 times larger than just the "user's message."

How to Calculate the Cost of a Single Request

The billing formula is straightforward:

Cost = (Input Tokens + Output Tokens) × Corresponding Unit Price

Let’s clarify two common pitfalls. First, the unit prices for input and output are usually different; output is generally more expensive, often by a factor of 2 to 5. Second, unit prices can vary based on tiered pricing or monthly commitments. The "cost per request" derived from small-scale testing may not hold true when volume increases.

Here is a concrete example. Consider a customer service bot:

- System prompt: 800 characters ≈ 900 tokens

- Retaining the last 10 turns of conversation, averaging 150 characters per turn ≈ 1,000 tokens

- Injecting 3 retrieved product document snippets ≈ 800 tokens

- Model response: ~200 characters ≈ 250 tokens

A single request involves approximately 2,700 input tokens + 250 output tokens. Assuming unit prices of $0.50 per million input tokens and $1.50 per million output tokens (prices vary significantly among vendors, but we’ll use these figures for illustration):

Single Request Cost ≈ (2700 / 1,000,000) × 0.5 + (250 / 1,000,000) × 1.5 ≈ $0.0017

With 10,000 requests per day, the daily cost is ~$17, resulting in a monthly cost of around $500. This scale is realistic: a mid-sized customer service operation handling 500,000 requests per week could easily see monthly bills exceeding $10,000.

Only Four Paths to Cost Optimization

1. **Compress the Context**. For conversation history, a common approach is to implement expiration: discard turns older than 15 minutes or replace original text with summaries. Before injecting retrieved documents, perform relevance filtering to keep only the top 1–2 most relevant snippets. Every 500 input tokens removed saves 50,000 tokens per day at 10,000 requests.

2. **Cache Stable Prefixes**. Long texts that repeat in every request, such as system prompts and tool definitions, can leverage prefix caching supported by most vendors. Cached portions are billed at a discounted rate. The longer the system prompt and the higher the request volume, the more significant the savings from caching. This is the quickest win with minimal code changes.

3. **Route by Difficulty**. Use rules or smaller models for tasks like classification, matching, or formatting. Reserve large models only for requests requiring multi-step reasoning. This isn’t just a slogan; it’s often a simple `if` branch that can drastically reduce large model calls.

4. **Limit Output Length**. Set a reasonable `max_tokens` limit and explicitly instruct the model in the prompt to "answer within 3 sentences." If a 50-character response suffices, do not allow the model to ramble on for 500 characters. Since output tokens are more expensive, verbosity essentially doubles your costs.

Combining these four strategies commonly reduces the cost per request by 30% to 50% without compromising response quality.

How to Know Your Actual Spending

Don’t rely on estimates; rely on logs. Each API response includes a `usage` field containing `prompt_tokens` and `completion_tokens`. Log these two values in your access logs and aggregate them daily to generate a real cost curve. Don’t wait until the end-of-month bill arrives to discover you’ve overspent.

Be able to detect two common anomalies immediately:

- **Input tokens suddenly double**: This likely indicates a bug in context assembly, such as repeatedly injecting the same document snippet in a loop or failing to truncate conversation history.

- **Surge in "context length exceeded" errors**: Your context is nearly full, meaning your compression strategy has failed. You must either trim history or switch to a model with a larger context window (which is more expensive).

Summary

Treat token usage as an engineering metric just like response time: monitor it, set budgets, and configure alerts. Before launch, create a static estimate; after running for a week, refine it with real usage data. Whenever you modify prompts, adjust context objects, or switch models, re-validate your costs. Cost management isn’t a one-time calculation; it’s a process of continuous calibration.