AI API Costs: How Tokens Work and How to Spend Less

Most AI APIs charge by tokens, small chunks of text (roughly a few characters each in English, often more tokens per word in other languages such as Persian). You pay for the tokens you send in and the tokens the model writes back, usually at different rates, and output tokens typically cost more than input tokens. Always check your provider's current price list; prices and model line-ups change often.

Context is the hidden multiplier

In a chat or an agent loop, every turn usually resends the whole conversation so far. A long session with large files pasted in therefore costs much more than the last message suggests. Tool results, such as long logs or full web pages, are the usual culprits.

Ways to spend less

1. Match the model to the job. Classification, extraction and routing rarely need the most capable model. Use a smaller, cheaper one and reserve the big one for hard reasoning. Measure quality before switching (see evaluating model output).
2. Trim the context. Send the relevant excerpt, not the whole document. Summarise old turns. Cap tool output length.
3. Use prompt caching where offered. Many providers discount repeated prefixes such as long system prompts or reference documents. Put stable content first and variable content last so the prefix can be reused.
4. Batch non-urgent work. Some providers offer lower prices for asynchronous batch processing.
5. Limit output. Ask for the format and length you need. Structured output is usually shorter than prose.
6. Avoid retries by design. Validate inputs before calling the model; fix prompts that fail often rather than retrying.
7. Start new sessions for new tasks. A fresh context is cheaper and often better than a long, cluttered one.

Measure before optimising

Log tokens in and out, model name and cost per request, tagged by feature. You will usually find that one or two features account for most of the bill. Optimise those first.

Budgets and guardrails

Set spending limits and alerts at the provider level, and put per-user or per-feature rate limits in your own code. An agent stuck in a loop can spend real money quickly, so cap the number of steps and the total tokens per task.

Cost is a quality question too

The cheapest call is the one you did not need. Often a deterministic function, a database query or a cached answer does the job better than a model call. Use the model where judgement or language understanding is truly needed.