Adding content moderation via a moderation API
When building software based on Large Language Models (LLMs), controlling input and output is an essential part of system design. Although language models have built-in safety filters, these are rarely sufficient for business-critical or publicly accessible applications. An external moderation API provides an independent verification layer to detect and block unwanted, illegal, or harmful content before it can cause harm to the user, the system, or the organization.
Content moderation via an API is not merely about policing conduct, but functions as a security perimeter. By systematically scanning input and output streams for policy violations, the risks of reputational damage, compute abuse, and legal liability are effectively mitigated.
The difference between technical validation and content moderation
In a robust application architecture, technical input validation and substantive content moderation are often chained together, yet they serve fundamentally different purposes. Implementing both layers is essential to keeping an application both stable and secure.
Technical validation verifies whether a request meets the system's structural requirements. Examples include checking the MIME type of uploaded files, enforcing character limits, validating a JSON schema, or parsing parameters. For the technical implementation of these structural checks, you can consult the guide on input validation and output filtering .
Substantive moderation, by contrast, examines the meaning and intent of the text. A prompt may be 100% technically valid JSON and fit neatly within the token limit, yet contain hate speech, confidential personal data, or attempt to bypass model instructions. Moderation APIs assess this semantic context using predefined policy rules and classification models.
Input and output moderation: Two lines of defense
A comprehensive moderation strategy rests on two pillars: gatekeeping at entry (input moderation) and vetting the final output (output moderation). Omitting either pillar leaves a clear vulnerability in the system.
1. Moderation before generation (Input)
Input moderation occurs immediately after the user submits a request, but before that request is forwarded to the primary language model. The primary goals of input moderation are:
- Flagging policy violations: Identifying text that incites violence, hate speech, harassment, or self-harm.
- Detection of indirect and direct prompt injection: User attempts to override or circumvent the AI's system instructions.
- Detection of personally identifiable information (PII): Preventing sensitive information such as BSN numbers, credit card details, or medical data from being inadvertently sent to external model providers.
- Cost and capacity savings: By blocking harmful or unauthorized prompts early on, you prevent expensive LLM tokens from being consumed for unwanted generations.
2. Post-generation moderation (Output)
Even if the user's input seems completely neutral, a language model's output can still contain undesirable elements. Output moderation checks the model's response before it is displayed to the end user. This focuses on:
- Unintended toxicity or bias: Under specific circumstances, models can respond unexpectedly inappropriately or offensively.
- Hallucinated personal data: Language models can generate plausible-sounding yet fictitious or scraped personal data that should not be made public.
- Prohibited topics: If the model exceeds its assigned role boundaries and provides advice on medical or legal topics the system is not intended for, this can be intercepted in the output layer.
- Brand damage and corporate information: Checking for leakage of internal system prompts or confidential knowledge base information in the generated text.
- Layer 1 (Local/Fast): Run regular expressions and keyword checks on the input. Known patterns or obvious PII (such as phone numbers or email addresses) are immediately blocked or masked without network latency.
- Layer 2 (External API): If the input meets the basic criteria, send the text to the external moderation API for an in-depth semantic evaluation.
- Layer 3 (Probability-based handling): Based on the returned probability scores, determine whether the text is approved immediately, definitively rejected, or, in case of doubt, forwarded for human review.
Strategic choices: Custom rules versus external moderation APIs
When setting up a moderation layer, developers face the choice between building an in-house filter or using a specialized external moderation API. Both approaches have specific pros and cons in terms of cost, latency, and maintenance.
Custom rule sets and local classifiers
Building an in-house filtering system is usually done using keyword lists (blacklists), regular expressions (regex), or lightweight, locally hosted open-source classification models.
Advantages: This approach offers extremely low latency and involves no additional costs per API call. Furthermore, no data leaves the internal network, which is advantageous from a privacy perspective.
Limitations: Static rules are easy to bypass through creative language, punctuation variations, or synonyms. Manually maintaining keyword lists is labor-intensive and scales poorly. Lightweight local classifiers often lack the contextual linguistic nuance required to detect subtle policy violations.
External moderation APIs
Specialized moderation APIs use larger, continuously updated classification models trained to recognize context, intent, and multilingual patterns. More background on the architecture of these models can be found in the knowledge base on moderation and safety models.
Advantages: High accuracy in understanding context, continuous updates by the vendor to address new circumvention techniques, and immediate availability of standardized categories.
Limitations: Each request requires an external network call, adding extra milliseconds to the total response time. Additionally, usage involves variable API costs per request, and third-party data processing must be taken into account.
The layered integration model (Tiered architecture)
In practice, a tiered setup often proves to be the most efficient solution. It combines fast, low-cost internal checks with thorough external API checks:
Placement of moderation in the processing pipeline
The way a moderation API is integrated into the processing pipeline directly impacts the user experience and system architecture. There are three common integration patterns.
1. Sequential before the model call (Pre-call)
In this pattern, the application waits for the moderation API's result before sending the request to the language model. This is the safest pattern for input moderation. Although total latency for the user increases by the duration of the moderation call, it is guaranteed to prevent the LLM from processing unwanted prompts.
2. Parallel to the model call
To minimize latency, the moderation call can be initiated simultaneously with the language model's processing. If the moderation API reports a violation before the language model finishes generating, the LLM process is aborted and the user receives an error message. This pattern reduces perceived waiting time, but it does consume LLM capacity if the request is stopped midway.
3. Moderation during streaming output (Chunking)
When an application streams responses to the user, waiting for full text generation before starting moderation is undesirable. To solve this, an intermediate approach using text chunking is applied. Developers implementing this pattern can read more about the technical background of streaming responses.
With chunking, the application collects generated tokens in a temporary buffer (for instance, per 50 to 100 words or per completed sentence). As soon as a buffer is full, it is sent in parallel to the moderation API while the streaming process to the user continues with a slight delay. If a chunk contains a violation, the stream is immediately aborted and replaced with a generic message.
Common categories in moderation APIs
Moderation APIs categorize the analyzed text into specific risk classes and assign a score or a binary flag (flagged: true/false) to each class. The table below provides an overview of the most common categories and their corresponding mechanisms.
| Category | Mechanism | Typical application |
|---|---|---|
| Hate & Discrimination | Detects colloquial speech, slurs, or derogatory remarks targeting protected characteristics such as race, religion, gender, or sexual orientation. | Input and output moderation for public chatbots. |
| Violence & Threats | Recognizes descriptions of physical violence, glorification of violent acts, or direct threats against individuals or groups. | Filtering user input on forums and in assistants. |
| Self-harm | Identifies text that incites, provides instructions for, or expresses self-harm and suicide. | Preventing harmful advice in medical or wellness applications. |
| Sexually explicit | Detects inappropriate sexual content, explicit descriptions, or non-consensual topics. | Enforcing terms and conditions in B2B and B2C software. |
| PII & Confidentiality | Analyzes text for patterns of directly identifiable personal data such as BSN (citizen service number), credit cards, passwords, and addresses. | Data Loss Prevention (DLP) for internal enterprise gateways. |
| Harmful instructions | Detects requests for assistance with illegal activities, weapon manufacturing, or executing cyberattacks. | Securing general LLM integrations against abuse. |
Handling misclassifications: False positives and false negatives
No classification model is flawless. When setting up content moderation, two types of errors must be taken into account:
- False positives: Legitimate input is incorrectly flagged as a policy violation. This leads to user frustration, as they receive a notification stating the request cannot be processed.
- False negatives: Harmful or unwanted text goes undetected and passes through the check. This poses a residual risk for the organization.
Tuning thresholds
Most moderation APIs return probability scores (for example, a value between 0.00 and 1.00) per category. Developers must configure the desired threshold per use case. A financial assistant requires stricter settings regarding PII and legal claims than a creative writing aid.
Striking the right balance between user convenience and risk tolerance requires close coordination with the organization. Guidelines for this can be found in the guide to Drafting an AI policy.
Human-in-the-loop review
For situations where the moderation API score falls into a 'gray area' (for example, between 0.40 and 0.70), a review queue can be implemented. Instead of rejecting the request immediately, it is flagged for review by a human moderator. This is particularly valuable when training and refining company-specific moderation rules.
Compliance, GDPR, and the EU AI Act
Implementing a moderation API is not just a technical and ethical choice, but also directly intersects with laws and regulations regarding data protection and artificial intelligence.
GDPR and the processing of personal data
When a moderation API is used to scan text, this third-party service processes the content of the request. If this text contains personal data, the following applies:
- Data Processing Agreement: A valid Data Processing Agreement (DPA) must be in place with the moderation API provider.
- Data transfers: Foreign API vendors must comply with GDPR-compliant data transfer mechanisms (such as the EU-US Data Privacy Framework or Standard Contractual Clauses).
- Log retention periods: Moderation APIs sometimes store requests to improve their own models. For compliant use, the option for 'zero data retention' (no data retention by the provider) must be enabled.
Relationship with the EU AI Act
Under the European AI Regulation (EU AI Act), certain applications fall into the high-risk category. For these applications, demonstrably managing risks related to bias, harmful output, and fundamental rights is mandatory. Incorporating an independent moderation layer constitutes an important technical measure within the risk management systems of these applications. A comprehensive explanation of the obligations can be found in the overview of the EU AI Act legislation.
Practical integration patterns in the gateway layer
To prevent each individual microservice from having to maintain its own connection to a moderation API, moderation functionality is preferably centralized in an API gateway or LLM proxy.
By incorporating moderation into a central gateway, a single, uniform point is established where all incoming and outgoing LLM traffic is inspected and logged. More information on setting up such an architecture is available in the guide on self-hosting an LLM gateway.
Caching moderation results
Frequently asked prompts or static content can lead to unnecessary repeated API calls. By applying a hashing mechanism (such as SHA-256) to the sanitized input text, moderation results can be temporarily cached in a fast in-memory datastore such as Redis. This reduces average latency and saves API costs.
Timeouts and fallback strategies
Like any external network service, a moderation API can experience delays or outages. Therefore, the gateway must explicitly define how the system responds when the moderation API fails to respond or times out:
- Fail-Open: In the event of an outage or timeout, the system allows the request through to the LLM. This ensures application availability, but accepts the risk of processing unfiltered content. This is primarily used for low-risk applications.
- Fail-Closed: In the event of an outage, the system blocks the request and returns an error message to the user. This guarantees that no unmanaged content enters or leaves the system, at the expense of availability. This is the standard for enterprise and high-risk applications.
