Buy, build, or custom algorithm: the three routes to LLM routing

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

An LLM gateway centralizes requests to different models, providers, and endpoints. But how do you choose the right approach? There are three routes: buying an off-the-shelf aggregator, building a gateway yourself, or developing a custom routing algorithm. Each route comes with its own costs, failure modes, and operational consequences. In this article, we compare the three options across six criteria: cost, throughput, flexibility, failure behavior, security, and maintenance. At the end, you'll find a decision tree tailored to your scale and risk appetite.

This article belongs to pillar A2 (Gateway, routing & throughput management) and links in the first paragraph to the anchor article how to self-host an LLM gateway, which explains the baseline architecture.

1. The three routes at a glance

Each route solves the same problem: your application needs to see a single interface, while the gateway distributes requests across multiple models, endpoints, or providers. The differences lie in who manages the logic and who runs the infrastructure.

Route 1: Buying an off-the-shelf aggregator

You integrate with an existing service that already has routing, fallback, and cost control built in. Examples include LLM API aggregators such as Fireworks, Together AI, or the open-source variant LiteLLM. You send a single API call to the aggregator, which forwards the request to the right provider and returns the response. The aggregator bills you based on usage and often charges a markup percentage.

Pros: you get a working solution within a day without managing infrastructure yourself. Cons: you pay a margin on top of provider costs, and you depend on the uptime and pricing models of the aggregator itself.

Route 2: Building a gateway yourself

You develop a custom proxy that receives, validates, and routes requests, then returns the responses. The gateway runs on your own infrastructure (cloud or on-premise), and you manage the keys, rate limits, and fallback logic yourself. Examples of open-source building blocks include llm-gateway (Python), bentoML or a custom implementation with FastAPI and Redis for rate limiting.

Pros: full control over routing, costs, and security. Cons: you must manage the infrastructure yourself, deploy updates, and resolve outages. Building a custom gateway only becomes cost-effective starting at several thousand requests per day.

Route 3: Developing a custom routing algorithm

You are not only building the infrastructure, but also the decision-making mechanism that determines which model or provider receives a request. The algorithm can take into account cost, latency, availability, task type, and even the content of the prompt. Examples include a cost-based router that always selects the cheapest available model, or a latency-based router that directs requests to the fastest endpoint.

Pros: maximum flexibility and optimization for your specific use case. Cons: complex development, maintenance, and the risk of the algorithm causing unintended side effects (for example, one provider consistently becoming overloaded).

2. Costs: what do you pay per route?

The costs of LLM routing consist of three components: provider costs, gateway costs, and operational costs. Each route has a different cost profile.

Costs with an off-the-shelf aggregator

You pay the aggregator a markup percentage on top of the provider costs. For example: if the provider charges €0.01 per 1000 tokens, the aggregator charges €0.012. Additionally, there may be fixed costs for premium features such as dedicated endpoints or priority support. The aggregator invoices you monthly based on usage, so there are no upfront costs.

Example calculation: at 1 million tokens per month and a 20% markup, you pay €12 instead of €10 directly to the provider. The aggregator does, however, take the burden of infrastructure and maintenance costs off your hands.

Costs with a custom-built gateway

You only pay the provider costs, but you do incur your own infrastructure costs. Running a gateway on a cloud provider like AWS or GCP costs between €50 and €200 per month, depending on the volume of requests and the required resources. On top of that, there are development costs: building the gateway takes several weeks of development time, and maintenance (updates, monitoring, bug fixes) requires ongoing effort.

Example calculation: at 1 million tokens per month, you only pay the provider costs (€10), but you also incur €100 in cloud costs and €500 in development time. From roughly 5 million tokens per month, building your own gateway becomes cheaper than using an aggregator.

Costs with a custom routing algorithm

The costs are comparable to a custom-built gateway, but with additional development costs for the algorithm. Building a routing algorithm takes more time than a simple gateway, and maintenance is more complex because the algorithm requires continuous optimization. Additionally, there may be extra costs for monitoring and logging, since you need to continuously track the algorithm's behavior.

Example calculation: at 1 million tokens per month, you pay €10 in provider costs, €100 in cloud costs, and €1000 in development costs for the algorithm. A custom algorithm only becomes cost-effective at very high volumes (10+ million tokens per month) or for complex use cases.

3. Throughput: how many requests can you process?

The throughput of your routing solution determines how many requests you can handle per second. This depends on the infrastructure and the efficiency of the routing logic.

Throughput with an off-the-shelf aggregator

Aggregators are built for scale and can handle thousands of requests per second. Throughput is typically limited only by the rate limits of the underlying providers. For example: if you use an aggregator that distributes requests across three providers, each with a rate limit of 100 requests per second, you can theoretically process 300 requests per second. In practice, this is lower due to overhead and network latency.

Pros: you don't have to scale anything yourself. Cons: you rely on the aggregator's uptime, and delays can occur during peak loads.

Throughput with a custom-built gateway

The throughput of a custom-built gateway depends on the infrastructure it runs on. A gateway running on a single server can process several hundred requests per second, while a distributed gateway on Kubernetes or AWS Lambda can handle thousands of requests per second. You can increase throughput by adding more resources, but this also increases costs.

Example: a gateway built with FastAPI and Redis for rate limiting can, on a t3.mediuminstance (AWS), process approximately 200 requests per second. By scaling horizontally to three instances, you can increase this to 600 requests per second.

Throughput with a custom routing algorithm

A custom routing algorithm adds complexity to the gateway, which can reduce throughput. The algorithm must decide for each request which provider to route it to, introducing extra compute and latency. In the best-case scenario, the impact is minimal (a few milliseconds per request), but with complex algorithms, latency can climb to tens of milliseconds.

Example: a cost-based routing algorithm that queries prices from all providers for every request can increase latency by 50 ms. At 100 requests per second, this translates to 5 seconds of cumulative delay per second.

4. Flexibility: how quickly can you adapt?

Flexibility determines how quickly you can respond to changes, such as new models, price adjustments, or evolving use cases. Each route offers a different level of flexibility.

Flexibility with an off-the-shelf aggregator

Aggregators offer limited flexibility. You can usually only choose from a predefined set of providers and models, and the routing logic is often restricted to simple fallback mechanisms. If you need a specific model or a custom routing strategy, you depend entirely on the aggregator's support.

Example: if an aggregator only supports OpenAI and Anthropic, you cannot route requests to a locally hosted model. You have to wait until the aggregator adds the model or switch to a different aggregator.

Flexibility with a custom-built gateway

A custom-built gateway offers complete flexibility. You can integrate any provider and any model, and fully tailor the routing logic to your use case. For example, you can build a fallback mechanism that routes requests to a local model when cloud providers are unavailable, or implement a priority queue for critical tasks.

Example: if you want to add a new model, you only need to add a new endpoint configuration to your gateway. You can also adjust rate limits per provider or add new validation rules.

Flexibility with a custom routing algorithm

A custom routing algorithm offers the highest degree of flexibility. You can fully tailor the algorithm to your use case—for instance, by factoring in prompt content, user history, or real-time performance metrics. You can also dynamically switch between models based on cost, latency, or availability.

Example: you can build an algorithm that routes customer service requests to an inexpensive model, while sending legal advice requests to a more expensive, highly accurate model. You can also adapt the algorithm to accommodate new pricing tiers or provider limits.

5. Failure behavior: what happens during outages?

Failure behavior determines how your routing solution responds to disruptions, such as provider downtime, network issues, or exceeded rate limits. Each approach comes with its own failure modes and mitigation strategies.

Failure behavior with an off-the-shelf aggregator

Aggregators typically feature built-in fallback mechanisms. If a provider becomes unavailable, the aggregator automatically forwards the request to another provider. While this works well for straightforward use cases, complex failures (such as all providers hitting rate limits simultaneously) can cause the aggregator itself to become unreachable.

Example: if OpenAI goes down, the aggregator forwards requests to Anthropic. However, if both providers hit rate limits, the aggregator itself may return a 503 Service Unavailable error. You have no control over the fallback logic and cannot designate which provider takes priority.

Failure behavior with a custom-built gateway

A custom-built gateway gives you full control over failure handling. You decide which fallback strategies to implement, such as retry logic, circuit breakers, or graceful degradation. You can also implement priority queues for critical tasks to ensure they are always processed first.

Example: you can implement a circuit breaker that automatically switches to a fallback provider after three failed requests to a primary provider. You can also build a priority queue ensuring that customer service requests always take precedence, even when the gateway is under heavy load. Read more about these patterns in the article on graceful degradation during LLM outages.

Failure behavior with a custom routing algorithm

A custom routing algorithm can further optimize failure handling by dynamically switching between providers based on real-time performance metrics. For example: if a provider responds slowly, the algorithm can automatically fail over to a faster provider. The algorithm can also take the prompt's content into account, ensuring that critical requests always route to the most reliable provider.

Example: you can build a latency-based algorithm that measures response times across all providers every second. If a provider slows down past 500 ms, the algorithm automatically switches to a faster provider. This does require comprehensive monitoring and logging, as detailed in the article on observability and logging for LLM applications.

6. Security: who manages the keys?

Security is crucial in LLM routing, as you are handling API keys, user data, and sensitive prompts. Each approach carries its own risks and mitigating measures.

Security with a turnkey aggregator

With an aggregator, you don't manage provider keys yourself, but you do need to manage a key for the aggregator. The aggregator manages the provider keys and handles encryption of data in transit. The risk is your dependency on the aggregator's security practices. If the aggregator is breached, your data and keys could fall into the wrong hands.

Example: when using an aggregator, you don't have to store an OpenAI key in your own codebase. However, you do need to manage a key for the aggregator, and if that key leaks, an attacker can make requests to the aggregator on your behalf.

Security with a custom-built gateway

With a custom-built gateway, you manage all provider keys yourself. This gives you complete control over security, but it also means you are solely responsible for securely storing and managing those keys. You must implement measures such as encryption of keys at rest, restricted access to the key store, and audit logging of key usage.

Example: you can store keys in a secrets manager such as AWS Secrets Manager or HashiCorp Vault. You can also implement rate limiting and IP whitelisting to prevent abuse. Read more about secure key management in the article on how to securely manage API keys for LLMs.

Security with a custom routing algorithm

A custom routing algorithm introduces additional security complexity. The algorithm requires access to provider keys and potentially to user data or prompts. You must ensure the algorithm does not leak sensitive data, for instance through logging or error messages. Additionally, you need to protect the algorithm against tampering, such as prompt injection or denial-of-service attacks.

Example: if your algorithm takes the content of the prompt into account, you must ensure that the prompt is not stored or logged. You should also implement validation rules to prevent an attacker from manipulating the algorithm, for instance by sending a prompt that forces the algorithm to select an expensive model. You can read more about input validation in the article on input validation and output filtering.

7. Maintenance: who resolves issues?

Maintenance encompasses everything needed to keep your routing solution operational: updates, monitoring, bug fixes, and documentation. Each route comes with its own maintenance requirements.

Maintenance with an off-the-shelf aggregator

With an aggregator, you do not have to maintain anything yourself. The aggregator handles updates, monitoring, and bug fixes. You only need to maintain your own integration, for example by implementing new API versions or adapting your code to changed rate limits.

Advantage: you carry no operational burden. Disadvantage: you depend on the aggregator's response time. If there is an outage, you must wait for the aggregator to resolve it.

Maintenance with a custom-built gateway

With a custom-built gateway, you are responsible for maintenance yourself. You must keep the gateway up to date with security patches, new provider APIs, and changes in your own infrastructure. Additionally, you need to implement monitoring and logging to quickly detect and resolve issues.

Example: if OpenAI releases a new API version, you must update your gateway to support that version. You must also regularly check the rate limits of all providers and adapt your gateway when they change. You can read more about monitoring in the article on observability and logging.

Maintenance with a proprietary routing algorithm

A proprietary routing algorithm requires the most maintenance. You not only have to maintain the gateway, but also the algorithm itself. The algorithm must be continuously optimized based on new data, for example when a provider's performance changes or when new models become available. In addition, you must monitor the algorithm to prevent unintended side effects, such as overloading a single provider.

Example: if a provider introduces a new pricing model, you must update your algorithm to account for the new costs. You must also regularly evaluate the algorithm's performance, for example by running A/B tests between different routing strategies.

8. Decision tree: which route suits you?

The right route depends on your scale, budget, risk tolerance, and use case. Use this decision tree to choose the best option:

Example 1: a startup with 100,000 requests per month and a limited budget opts for an aggregator. Costs are predictable and there is no operational burden.

Example 2: a SaaS company with 5 million requests per month and an in-house development team builds its own gateway. Costs are lower than with an aggregator, and the company has full control over routing and security.

Example 3: an enterprise organization with 50 million requests per month and a complex use case (such as legal advice) develops a proprietary routing algorithm. The algorithm optimizes for cost, latency, and accuracy, and the company has the resources to continually improve the algorithm.

9. Conclusion: routing is not a one-time decision

LLM routing is not a static decision, but a dynamic process that evolves with your use case. Start with an aggregator if you want to get up and running quickly, build your own gateway if you need scale and control, and develop a custom algorithm if you want maximum flexibility and optimization. Every route comes with its own costs, failure modes, and operational implications, so choose deliberately and regularly re-evaluate whether your current solution still fits your needs.

Remember that routing is not just about selecting a model, but also about managing costs, throughput, security, and failure behavior. A well-architected routing solution makes the difference between a reliable, cost-efficient LLM integration and an unpredictable, expensive system. If you want to learn more about the technical details of routing, check out the article on how to orchestrate multiple models.

For those looking to dive deeper into embeddings and retrieval: embedding APIs come with their own rate limits and costs that must be managed separately. Learn more about this in the article on how to evaluate embedding models for search and RAG applications.