Managing API Keys for LLMs Securely: A Complete Guide

If you work with Large Language Models (LLMs) through services like OpenAI, Anthropic, Google Gemini, or Mistral, your API key is the gateway to unprecedented computing power. But this key is much more than just a password; it is essentially an unsecured credit card. In the world of web development, securing these keys is one of the most critical tasks you have.

A leaked LLM API key can lead to thousands of euros in unintended costs in just a few hours. Malicious actors continuously scan the internet looking for unprotected keys. They use them not only to access AI services for free themselves, but also resell them or deploy them for large-scale, automated spam campaigns. In this article, we discuss how to securely manage LLM API keys in a real production environment, far out of reach from hackers and careless mistakes.

Why API Keys Never Belong in the Frontend or Git

The most common mistake made by beginner developers is calling an LLM provider directly from frontend code. Whether it is a React application, a Vue dashboard, or a mobile app in Flutter: if the code runs on the end user's device, it is insecure.

The Danger of Frontend Code and Client-Side Execution

Everything you send to a user's browser is public. Even if your code is minified or obfuscated, an experienced attacker can retrieve your API key within seconds using a browser's developer tools. As soon as the application makes a network request (HTTP request) to, for example, api.openai.com, the key is visible in the Authorization header of the network tab.

The same principle applies to mobile apps. Decompiling an APK or IPA file is trivial nowadays. If the string containing the API key is in the source code, it will inevitably be found by automated scripts.

The Pitfall of Git Repositories and Commit History

Another classic mistake is hardcoding an API key in the codebase and then pushing it to a version control system like Git (for example, GitHub or GitLab). Many developers think: "I'll just remove the key from the file in the next commit." This is a fundamental misunderstanding of how Git works.

Git remembers the entire history. A key that was in a previous commit remains visible in the commit history forever. Malicious bots scan public repositories within milliseconds of a push. Even in private repositories, it is a major risk: a compromised developer account or a rogue employee will have direct access to your production keys.

The Basics: Environment Variables and Secret Managers

If the key should not be in the code, where should it be? The universal answer in software development is using the environment in which the application runs.

Working with .env Files

During local development, you use so-called .env files. These are simple text files in which you store configurations in the form of key-value pairs, such as OPENAI_API_KEY=sk-12345.... It is crucial to explicitly add the .env file to your .gitignore file so that it is never sent to your repository.

In your backend code, you then read this variable. In Node.js, for example, you do this via process.env.OPENAI_API_KEY, and in Python via os.environ.get("OPENAI_API_KEY"). This way, the code remains flexible and separated from sensitive data.

Advanced: Cloud Secret Managers

For a robust production environment, .env files on a server are not sufficient. They are difficult to manage centrally and can leak if someone accidentally exposes your server's folder structure (for example, via a misconfigured web server).

The standard for production is using a Secret Manager. Services such as AWS Secrets Manager, Azure Key Vault, Google Cloud Secret Manager, or HashiCorp Vault provide a secure vault for your API keys. They offer huge advantages:

• Encryption at rest: Keys are stored encrypted.
• Fine-grained access control (IAM): You can specify exactly which virtual machine or serverless function is allowed to read the key.
• Audit logging: You have a log showing who (or which service) requested the key and when.

Structuring: Scoping and Environments

Not every API key should be usable for everything. The principle of least privilege states that a system should only have the permissions strictly necessary to perform its task.

Separate Keys per Environment

Never use the same API key for your development (Dev), testing (Staging), and production (Prod) environments. Create separate keys with your LLM provider. If a developer accidentally leaks the Dev key during testing, you can revoke it without your production application going offline (downtime). Furthermore, this makes it much easier to see which environment is generating the most costs.

Restricting Permissions (Scoping)

Modern LLM providers increasingly offer "scoped API keys". This means you can limit the permissions of a key. For example:

• A key may only generate text (chat/completions), but not upload files for fine-tuning.
• A key only has access to a specific, cheaper model (such as gpt-4o-mini) and is blocked for expensive, heavy models.
• A key only has "read-only" permissions to request usage statistics (useful for dashboards).

By strictly scoping keys, you greatly limit the potential damage should a leak occur.

Cost Management: Limiting Spending per Key

In addition to preventing theft, you must also protect yourself against errors in your own code. An infinite loop that accidentally calls an LLM 10,000 times per minute can be just as destructive as a hacker. Setting budgets is therefore essential.

In the portal of almost every major provider, you can set limits. We distinguish two types:

1. Soft limits: When this financial amount is reached, the API continues to work, but you immediately receive a warning email. This gives you the opportunity to intervene or investigate whether the traffic is legitimate.
2. Hard limits: Once this amount is reached, the provider blocks all new API requests. Your application will start returning error messages (usually an HTTP 429 or 403 status code), but your wallet is protected.

Structurally monitoring these expenses is a discipline in itself. For an in-depth guide on this, we recommend reading our article on monitoring costs, in combination with properly configuring rate limits and costs.

Security Architecture: The Proxy or API Gateway

As established earlier, a client (the browser or mobile app) must never communicate directly with the LLM provider. The only secure architecture is placing your own server, proxy, or API gateway between the client and the LLM.

The flow looks like this:

Client (React) → Your API Gateway → LLM Provider (OpenAI)

In this setup, the following happens:

• The client sends a request to your domain (e.g., api.jouwapp.nl/v1/chat). The client uses a normal authentication method for this (such as a JWT token linked to the user's login credentials).
• Your API gateway receives the request, checks the user's identity, and validates whether they are authorized to call the LLM.
• Only after everything is approved does your server retrieve the secret LLM API key from the Secret Manager.
• Your server inserts the API key into the Authorization header and forwards the request to the LLM provider.
• The response from the LLM returns to your server, which then securely forwards it to the client.

This "Backend-for-Frontend" (BFF) or proxy architecture ensures that the API key never leaves your secure network. Furthermore, this gives you the perfect place to add caching, custom rate limiting, and logging, which is crucial for robust integrations. For a broader view of secure system setups, you can also consult the basic principles of AI security on our learning platform.

Best Practices: Rotating API Keys Without Downtime

Just like passwords, API keys must be replaced periodically. This process is called "rotation". Many compliance frameworks (such as ISO 27001 or SOC2) require secrets to be rotated every 90 days. Rotating keys reduces the risk of an unnoticed leak leading to long-term abuse.

Rotating a key in a production environment must be done without the service going offline. Always follow this step-by-step process:

1. Generate a new key: Log in to your LLM provider and create a second, active key. Do not delete the old key yet.
2. Update the Secret Manager: Add the new key to your environment variables or Secret Manager. Depending on your setup, this may require restarting or redeploying the application.
3. Verify the new key: Check your monitoring dashboards. Ensure that successful API calls (HTTP 200) are being made with the new configuration.
4. Wait for the transition: Give the system a few minutes (or hours, depending on active sessions and queue workers) to ensure no more requests are running through the old configuration.
5. Revoke the old key: Only when you are sure the old key is no longer being used by your own systems should you delete or deactivate it in the provider's portal.

Emergency Plan: What to Do in Case of a (Suspected) Leak?

Even with the best processes, things can go wrong. An employee falls for a phishing attack, or a misconfigured firewall exposes an internal service. If you suspect an API key has been leaked, every minute counts. Follow this checklist immediately:

1. Revoke the key: This is absolute priority 1. Delete the key immediately at the provider. Do not ask for permission first, do not fix the code first. Stop the bleeding. This will cause downtime for your application, but it stops the financial drain.
2. Identify the source of the leak: Find out how the key became public. Was it in a GitHub commit? Was it in a frontend bundle? Was the server hacked? You must close the gap before placing a new key.
3. Rotate and restore: Generate a new key and place it back into the application securely (via the Secret Manager). Deploy the application to bring it back online.
4. Assess the damage (Audit): Log in to your provider's billing dashboard. Look at which models were called and how many tokens were consumed during the period the key was leaked.
5. Report: Inform internal stakeholders (management, security team) about the incident. Check if the leaked API key could theoretically have been used to access sensitive customer context (for example, if the attacker could retrieve prompt history, which is possible with some APIs).

Conclusion

Managing LLM API keys requires mature software engineering practices. Simply copying a string into your code is acceptable for a ten-minute local hobby project, but completely unacceptable for anything that touches the internet. By consistently using Secret Managers, proxy servers, strict limits, and regular rotation, you protect your organization from both financial damage and reputational loss in the AI era.