API Documentation: Costs & Budgets

Why Monitor LLM Costs?

When scaling generative AI or RAG (Retrieval-Augmented Generation) applications, API costs can quickly add up. Without close monitoring of input (prompt) and output (completion) tokens per session, you risk budget overruns. A robust monitoring system breaks down costs per model, user, or project.

For an efficient setup, it is also essential to know which models offer the best price-to-performance ratio. To do so, consult our comprehensive LLM Benchmark.

Example Dashboard (Concept)

A good dashboard displays the "burn rate" at a glance. Below you can see a structural layout of the metrics aggregated by our API.

Implementation: Quotas and Logging (Pseudocode)

To control costs per user, the API gateway validates current usage before forwarding the request to the external LLM provider. After the response, the actual tokens consumed are logged.

# Pseudocode: LLM Request Middleware
def handle_llm_request(user_id, prompt_data, model="deepseek-chat"):
    # 1. Check budget limit
    current_spend = get_user_spend(user_id, current_month)
    if current_spend > USER_BUDGET_LIMIT:
        return {"error": "Budget exceeded. Please contact the administrator."}
        
    # 2. Execute the request via the LLMnet.nl aggregator
    response = llm_aggregator.call(model, prompt_data)
    
    # 3. Calculate costs based on token pricing
    cost = calculate_cost(
        model, 
        prompt_tokens=response.usage.prompt_tokens, 
        completion_tokens=response.usage.completion_tokens
    )
    
    # 4. Update the database & trigger any alerts
    log_usage(user_id, cost)
    check_and_trigger_alerts(user_id, current_spend + cost)
    
    return response.content

Best Practices for Budgeting

• Soft & Hard Caps: Set a 'soft cap' (e.g., 80%) where the administrator receives an email, and a 'hard cap' (100%) where the API temporarily blocks requests for that specific user.
• Caching: Avoid duplicate API costs by intercepting identical queries via a local (vector) cache before querying the LLM.
• Distinguish between internal and external users: Apply stricter quotas for public end-users than for your internal development team.