Health checks and syntax probes for LLM endpoints

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

For traditional web applications and microservices, a simple HTTP GET call to a /healthzendpoint is enough to determine whether a process is running and can handle network traffic. But once a system depends on external language models, this superficial check falls fundamentally short. An LLM provider can return a valid HTTP 200 status code on a status or model-list endpoint while the underlying inference cluster is heavily overloaded, cuts tokens off midway, or generates corrupt payloads that crash downstream parsers. Anyone who wants to run reliable AI applications in production must extend the architecture with layered status checks and active syntax probes. This article covers the operational implementation of these detection mechanisms within pillar A1 (Reliability & failure behavior).

The overview on building robust integrations explains why failure modes in language models differ structurally from conventional software components. Where a database crashes or refuses connections, an LLM often shows subtle degradation: latency creeps up gradually, specific parameter variants (such as response_format or tool instructions) fail silently, or context limits are quietly exceeded. By setting up liveness, readiness, and syntax probes in a controlled way, failing upstream dependencies are isolated immediately, before end users run into stalled workflows or corrupted data.

The three levels of endpoint validation

A robust verification system for LLM dependencies consists of three strictly separated layers. Each level answers a specific operational question and has its own execution frequency, cost profile, and error handling.

The distinction between liveness and readiness prevents so-called cascading restarts. If an orchestration platform like Kubernetes restarts a gateway container because the external OpenAI or Anthropic API is having an outage, the restart fixes nothing. The gateway actually becomes overloaded during the restart phase, internal queues are lost, and the restart cycle masks the real cause. The liveness probe therefore checks only the local process status and memory usage. The readiness probe determines whether the instance is allowed to receive upstream traffic from the load balancer, and disables individual routes as soon as authentication or network access fails.

Failure modes in language model endpoints

To build effective probes, you first need to systematically map out the ways language models fail in practice. Unlike deterministic software, AI models exhibit various failure modes that remain completely invisible to traditional monitoring:

• Silent schema corruption: The provider rolls out a sub-version update or a changed quantization in the background. As a result, the model responds differently to complex instructions, required fields are suddenly omitted, or braces don't close correctly in the generated JSON.
• Token exhaustion and capacity issues: The provider accepts the incoming HTTP request and immediately returns a 200 OK status, but stops streaming the response halfway through because the GPU cluster is hitting capacity limits.
• Parameter-specific failures: The general chat-completion route functions properly, but specific features — such as forced json_schemasettings or advanced tool definitions — return internal server errors (HTTP 500) or deserialization errors.
• Extreme tail latency: The endpoint remains technically operational, but the Time to First Token (TTFT) rises from the usual 400 milliseconds to 25 seconds, causing downstream microservices to time out en masse.
• Silent content filter overreactions: An automated safety filter on the provider's side unjustly blocks mundane operational prompts, causing the application to receive an empty response or an error message without a payload.

Syntax probes: active validation of structured output

A syntax probe is an active, synthetic call sent periodically to the upstream model to check whether the output passes the required parsing and schema validation logic. Many business-critical applications rely on strict data contracts, as described in the guide on reliable structured output. If a model suddenly delivers invalid JSON or deviates from the data model, the entire downstream processing chain breaks.

An effective syntax probe uses a minimal prompt budget to spare token costs and rate limits, but does enforce exactly the same schema validation mechanism used in production. Below is a generic TypeScript example of a syntax probe that verifies the network connection, the response time, and strict JSON schema validation via a standard fetch interface.

interface ProbeResult {
  healthy: boolean;
  latencyMs: number;
  statusCode?: number;
  error?: string;
}

async function runSyntaxProbe(
  endpointUrl: string,
  apiKey: string,
  modelName: string,
  timeoutMs = 4000
): Promise<ProbeResult> {
  const start = Date.now();
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), timeoutMs);

  // Dynamische nonce voorkomt dat gateways of providers een gecachte response serveren
  const dynamicNonce = Math.floor(Math.random() * 1000000);

  const payload = {
    model: modelName,
    messages: [
      { role: "system", content: "Retourneer uitsluitend het gevraagde JSON-schema." },
      { role: "user", content: `Statuscheck ID ${dynamicNonce}. Bevestig status met 'ok'.` }
    ],
    response_format: {
      type: "json_schema",
      json_schema: {
        name: "health_probe_payload",
        strict: true,
        schema: {
          type: "object",
          properties: {
            status: { type: "string", enum: ["ok"] },
            nonce: { type: "integer" }
          },
          required: ["status", "nonce"],
          additionalProperties: false
        }
      }
    },
    max_tokens: 30,
    temperature: 0
  };

  try {
    const response = await fetch(endpointUrl, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${apiKey}`
      },
      body: JSON.stringify(payload),
      signal: controller.signal
    });

    const latencyMs = Date.now() - start;

    if (!response.ok) {
      return {
        healthy: false,
        latencyMs,
        statusCode: response.status,
        error: `HTTP ${response.status}: ${response.statusText}`
      };
    }

    const data = await response.json();
    const rawContent = data.choices?.[0]?.message?.content;
    if (!rawContent) {
      return {
        healthy: false,
        latencyMs,
        statusCode: response.status,
        error: "Geen geldige payload ontvangen in de response"
      };
    }

    const parsed = JSON.parse(rawContent);
    if (parsed.status !== "ok" || parsed.nonce !== dynamicNonce) {
      return {
        healthy: false,
        latencyMs,
        statusCode: response.status,
        error: "Syntaxis-validatie mislukt: schema-inhoud komt niet overeen"
      };
    }

    return { healthy: true, latencyMs, statusCode: response.status };
  } catch (err: any) {
    const isTimeout = err.name === "AbortError";
    return {
      healthy: false,
      latencyMs: Date.now() - start,
      error: isTimeout ? `Timeout na ${timeoutMs}ms` : err.message
    };
  } finally {
    clearTimeout(timeoutId);
  }
}

Integration with circuit breakers and fallback paths

Detecting a failing syntax probe only has operational value if the outcome is directly linked to the routing and failover layer of the gateway. When two or three consecutive syntax probes fail or exceed the configured deadline, the model provider in question must be removed from the active routing pool immediately.

In the article on implementing circuit breakers for unstable LLM APIs the state transitions between Closed, Open, and Half-Open are covered in detail. The syntax probe acts here as the ideal test pulse in the Half-Open state. Only once a synthetic probe successfully returns a valid JSON structure within the set time limit may the circuit breaker return to the Closed status and allow regular production traffic through.

When a provider is marked unhealthy, the gateway switches over to a secondary model or an alternative cloud region. This prevents users from seeing error messages while the primary provider is experiencing a partial outage.

What does active endpoint probing cost?

Periodically sending synthetic requests brings structural API costs, rate-limit load, and network traffic with it. The frequency of the probes must therefore be carefully tuned to the application's risk profile.

When using compact, efficient models, the direct token costs of active probes remain limited to a few cents per month, even under an aggressive schedule. The main point of attention is rate limits (Requests Per Minute). When an account runs on a low API tier with a model provider, an overly frequent probe can eat into capacity that real end users need. In such situations, an interval of 3 to 5 minutes is the best operational balance.

Measurement methods and metrics for analysis

A binary 'healthy or unhealthy' measurement is not sufficient for in-depth operational analysis. To recognize degradation in time, before a total outage occurs, gateways should record the following telemetry:

• Time to First Token (TTFT): The time between sending the HTTP request and receiving the first token chunk. A structural rise in TTFT almost always points to queuing and GPU congestion at the provider.
• Token generation speed (tokens/sec): The speed at which the model then generates output. If this value suddenly drops, it often indicates resource throttling by the provider's infrastructure.
• JSON deserialization error rate: The percentage of synthetic probes where the received string cannot be successfully parsed as valid JSON.
• Schema drift & violation rate: Cases where the JSON is syntactically valid but required fields are missing or enum values don't match the specification.

For advanced production systems where model behavior is evaluated over longer periods, acceptance testing offers a solution. The overview on setting up acceptance tests for non-deterministic output goes into detail on systematically evaluating model responses beyond strict syntax checks.

Weaknesses and pitfalls of active probes

Although syntax probes are indispensable for proactive outage detection, they have specific operational limitations that must be explicitly weighed in the architecture design:

• False confidence with short contexts: For cost reasons, a probe sends a minimal prompt of roughly 30 to 50 tokens. A language model can handle a simple prompt without any problem, but can still fail on memory or attention limits once a user submits a heavy RAG prompt of 40,000 tokens. A successful syntax probe therefore only guarantees that the inference engine and the JSON generation layer work, not that the model stays stable under heavy context load.
• Cache pollution and false positives from semantic caching: If the exact same static prompt is repeated every minute, an intermediate gateway or provider cache can serve the response directly without actually calling the language model. As a result, the probe only measures the cache's response time, not the health of the underlying GPUs. Using a dynamic nonce or variable timestamp in the prompt is therefore mandatory.
• Jitter and false positives from overly tight timeouts: If a probe's timeout is set to 1,000 milliseconds while the normal P95 latency is around 800 milliseconds, a temporary network fluctuation will immediately trigger an unwarranted failover. Always use a conservative threshold of at least 2 to 3 consecutive failures before a circuit breaker trips.
• System load during provider incidents: When an upstream provider is experiencing a global outage and returning error messages, hundreds of distributed gateways keep firing probes undiminished. This can lead to unnecessary overhead. Ensure exponential backoff on the probe interval once a circuit breaker has reached the 'Open' status.

Step-by-step plan for production implementation

Follow this step-by-step plan when designing and rolling out health checks in an LLM gateway:

1. Segregate the monitoring endpoints: Configure /healthz/liveness for internal process monitoring, /healthz/readiness for upstream API reachability, and decouple the active syntax probe as an asynchronous background task.
2. Minimize the payload: Keep the probe prompt extremely concise and cap max_tokens at a maximum of 25 to 30 tokens. This avoids unnecessary costs and protects your rate limits.
3. Enforce strict timeouts: Set the syntax probe's timeout to the model's P99 latency limit plus a safe margin (typically between 3,000 and 5,000 milliseconds).
4. Add dynamic entropy: Always include a random number or timestamp in the request to force responses out of caches.
5. Link directly to the circuit breaker: Make sure the result of the syntax probe directly updates the routing table, so that failing endpoints are avoided immediately.

By consistently applying these verification layers, monitoring changes from a reactive, after-the-fact process into an automated, preventive safety net that safeguards the reliability of the entire LLM integration.