Rate limits, tokens, and costs: how to keep your LLM API usage under control

When building web applications that rely on Large Language Models, you quickly run into the technical reality of API integrations: tokens are not free, and bandwidth is not infinite. Especially with advanced implementations, such as a RAG (Retrieval-Augmented Generation) architecture with vector search, costs and latency can quickly add up unnoticed. In this guide, we dissect the mechanisms and offer concrete tools for optimization.

What is a token (and context vs. output)?

Roughly speaking, one token equals three-quarters of a word. LLM providers, whether you use models like Claude Pro variants or DeepSeek, bill based on two streams:

Input tokens (Context): The text, instructions, and context you send to the model.
Output tokens (Completion): The response generated by the model.

Output tokens are significantly more expensive than input tokens with most providers (sometimes by a factor of 3 to 5). When designing your prompts, it is therefore essential to guide the model toward concise, structured output (such as JSON) to reduce completion costs, even if this means your input prompt becomes slightly longer.

Be careful with RAG: If you use vector search to retrieve document chunks, you will quickly blow up your context window. Retrieve only the top-k most relevant results instead of sending entire documents.

Rate limits: Understanding TPM and RPM

Providers protect their infrastructure with limits. This usually happens along two axes:

RPM (Requests Per Minute): The number of individual API calls you are allowed to make.
TPM (Tokens Per Minute): The total volume of processed tokens (input + output combined).

If you hit these limits, the API returns an HTTP 429 Too Many Requests error. Simply ignoring this leads to broken applications and poor user experiences.

Caching, Batching, and Routing

To reduce costs and bypass rate limits, the architecture needs to be built smarter:

Semantic Caching: Store answers to frequently asked, similar questions. If a user asks a question that semantically matches a previous question by 95%, serve the cache instead of making a new LLM call.
API Orchestrators: Use a gateway or orchestrator (for example, a setup with OpenClaw) to route requests smartly. Simple classification tasks can go to a cheaper, faster model, while complex reasoning goes to the most expensive flagship model.

Pseudocode: Exponential Backoff & Caching

Below is an abstract example of how to robustly handle rate limits and caching in your application layer:

function call_llm_api_with_backoff(prompt, max_retries=3):
# 1. Check local cache first
cache_result = check_semantic_cache(prompt)
if cache_result:
return cache_result

# 2. Prepare API call
delay = 1

for attempt in range(max_retries):
response = execute_request(prompt)

if response.status == 200:
save_to_cache(prompt, response.data)
return response.data

if response.status == 429: # Rate limit hit
wait(delay)
delay = delay * 2 # Exponential increase (1s, 2s, 4s...)
else:
abort_with_error(response)

throw Error("Rate limit persistently exceeded after retries")

Cost Control Checklist

Set Max Tokens: Always define a hard limit for max_tokens in your API call to prevent infinite generation loops.
Smart Model Routing: Use fast, cheap models for simple NLP tasks (such as entity extraction) and reserve complex models exclusively for heavy reasoning.
RAG Optimization: Reduce your chunk size during vectorization and set strict thresholds (similarity scores) so that irrelevant context is not sent.
Batch Processing: Do you have asynchronous tasks (e.g., background summaries)? Make use of the Batch API endpoints that many providers offer at reduced rates (often a 50% discount).
Monitor your Tiers: Upgrade to higher usage tiers with your provider in time by pre-funding your account, which often immediately raises your TPM/RPM ceilings.

Need help setting up a cost-effective architecture or optimizing your RAG pipelines? Check out our LLM Consultancy services for tailored technical advice.