Skip to content
NLEN
Illustration: Streaming with tool calls: partial output and interrupted calls

Streaming with tool calls: partial output and interrupted calls

By Ivo Donker — compiled with AI assistance · August 7, 2026

The duality of streaming and function calling in production

Applying Server-Sent Events (SSE) and function calling together introduces a unique architectural challenge in production environments. Separately, both patterns are transparent: streaming delivers directly processable text tokens for user interaction, while function calling generates a structured JSON data envelope for backend systems. See the guide on streaming responses in LLM APIs for the basic principles of SSE connections and token buffering. When you combine these two techniques, the behavior of the API stream changes fundamentally. A large language model (LLM) doesn't stream the arguments of a function call as one valid JSON object, but as a series of fragmented text slices (deltas) over the duration of the request. For the fundamental workings of function calls and schema definitions, see the overview of function calling and tool use.

The combination of streaming and tool calls differs from regular text streaming because the intermediate data stream is unusable for direct processing by external application logic. As long as the last token of an arguments array hasn't been sent by the server, the serialized JSON fragment remains in a syntactically incomplete state. Parsing an accumulated buffer too early inevitably leads to runtime exceptions. Moreover, a stream can be interrupted at any point by a network error, client disconnect, or provider timeout. In a standard text stream, an interruption merely results in a cut-off sentence on the user's screen. With streaming tool calls, an interruption can result in a half-received payload, leaving backend systems uncertain about the integrity and intent of the request.

In this article, we analyze the specific failure modes of streaming tool calls in production environments. For each failure mode, we cover systematic detection, concrete mitigation strategies, and the associated operational costs in terms of latency, financial budget, and system complexity.

Failure mode 1: Fragmentation and accumulation errors in argument deltas

Failure mode: The model splits the JSON arguments of a tool call across dozens of separate SSE chunks. When multiple tool calls are invoked in parallel, the API sends the argument deltas interleaved, correlated via an index field. If the processing application simply concatenates these chunks without accounting for the specific tool index, a corrupted string results that cannot be parsed.

Detection: Detection happens at the level of the stream-parsing mechanism. As soon as JSON.parse() produce a SyntaxError is thrown while processing an accumulated buffer, or when required fields are missing after schema validation, this indicates an accumulation error or a missing delta.

Mitigation: Implement an explicit buffering layer that splits incoming chunks into two separate channels: direct text output for the user interface, and indexed argument buffers for tool calls. Every incoming tool delta must be assigned to an internal state table based on the supplied index and id. Only once the stream sends the explicit end signal may the buffer be merged per index and handed to a JSON parser and schema validator.

The provider-agnostic pseudocode below illustrates how to process an SSE stream with strict separation of text and tool deltas, including a timeout on the stream and an explicit error path.

async function processLLMStream(streamReader, timeoutMs) {
  const toolCallBuffers = new Map();
  let textBuffer = "";
  
  const timeoutPromise = new Promise((_, reject) => 
    setTimeout(() => reject(new Error("STREAM_TIMEOUT")), timeoutMs)
  );

  try {
    await Promise.race([
      (async () => {
        for await (const chunk of streamReader) {
          if (chunk.type === "text_delta") {
            textBuffer += chunk.content;
            emitUIUpdate(chunk.content);
          } else if (chunk.type === "tool_delta") {
            const index = chunk.index;
            if (!toolCallBuffers.has(index)) {
              toolCallBuffers.set(index, { id: chunk.id, name: chunk.name, argsText: "" });
            }
            toolCallBuffers.get(index).argsText += chunk.args_delta;
          }
        }
      })(),
      timeoutPromise
    ]);
  } catch (error) {
    logError("Streamverwerking mislukt of getime-out", error);
    throw new StreamProcessingException("Stream onderbroken voor accumulatie", error);
  }

  const validatedToolCalls = [];
  for (const [index, call] of toolCallBuffers.entries()) {
    try {
      const parsedArgs = JSON.parse(call.argsText);
      validatedToolCalls.push({ id: call.id, name: call.name, args: parsedArgs });
    } catch (parseError) {
      throw new InvalidToolPayloadException(`Ongeldige JSON in tool call index ${index}`, parseError);
    }
  }
  return { text: textBuffer, toolCalls: validatedToolCalls };
}

Cost of the mitigation:

Failure mode 2: Incomplete streams and interrupted connections

Failure mode: The SSE connection is terminated prematurely by a dropped network connection, exceeding the maximum response time, or an active cancellation by the client. The received buffer does contain a partial JSON structure, but the stream's end marker is missing.

Detection: Inspect the stream's metadata when the transport channel closes. A correctly closed stream contains an explicit status indicator, such as finish_reason. If the stream closes without finish_reason being equal to tool_calls or stop (for example, with length or when the field is entirely absent), the signal is incomplete. Additionally, an attempt to parse the accumulated string will fail with an unexpected end of input.

Mitigation: Apply a strict all-or-nothing principle to processing tool calls. Partially received argument strings must never be forcibly repaired or partially executed. If the stream is interrupted before the closing event is received, the entire payload of that tool call must be discarded. The system must roll the session state back to the last known valid state. For setting up robust control mechanisms and automated validations, also read the guide on testing LLM integrations.

The fragment below shows how an application evaluates the final status of a stream and, on an interrupted connection, throws a controlled exception instead of forwarding the incomplete data to business-critical systems.

function finalizeStreamPayload(streamState) {
  if (!streamState.isCompleted) {
    throw new AbortedStreamException("Stream verbroken voordat de server het eindsignaal stuurde.");
  }

  if (streamState.finishReason === "length") {
    throw new TokenLimitExceededException("Model heeft de max_tokens limiet bereikt tijdens tool call.");
  }

  if (streamState.finishReason !== "tool_calls" && streamState.finishReason !== "stop") {
    throw new InvalidFinishReasonException(`Onverwachte finish_reason: ${streamState.finishReason}`);
  }

  return streamState.toolCalls.map(call => {
    const parsed = safelyParseJSON(call.rawArgs);
    if (!parsed.success) {
      throw new MalformedPayloadException(`Onvolledige JSON voor tool ${call.name}`);
    }
    return { id: call.id, name: call.name, args: parsed.data };
  });
}

Cost of the mitigation:

Failure mode 3: Duplicate execution under uncertain execution states

Failure mode: The model sends the full arguments of a tool call via the stream. The client receives the arguments, the stream closes, and the client starts executing the external function (for example, a payment order or a database mutation). If the network connection drops right after execution — but before the client can report the function's result back to the LLM — an automatic retry handler may resubmit the request. This creates the risk that the external action gets executed twice.

Detection: This problem doesn't manifest as a direct API error, but as inconsistency in external systems (such as duplicate bookings or duplicate resource creation). In the application logs, this shows up as a request being reinitiated with the same business context but a new stream ID.

Mitigation: Never blindly retry tool calls on a network error or interrupted follow-up stream. Always use a unique idempotency key for every external action that causes mutations. This key must be derived from the unique tool call ID supplied by the LLM provider in the stream, combined with a hash of the validated arguments. For setting up distributed unique keys and processing them in backend systems, see the article on idempotency in LLM calls.

It's also necessary to bound automatic retry mechanisms on the stream itself. For the precise configuration of retry strategies for dropped HTTP connections, we refer to the overview on retries and exponential backoff.

async function executeToolCallWithIdempotency(toolCall, executionContext) {
  const idempotencyKey = `exec_${toolCall.id}_${hashPayload(toolCall.args)}`;
  
  const lockAcquired = await executionContext.idempotencyStore.lock(idempotencyKey);
  if (!lockAcquired) {
    return await executionContext.idempotencyStore.getResult(idempotencyKey);
  }

  try {
    const result = await Promise.race([
      executeExternalService(toolCall.name, toolCall.args),
      new Promise((_, reject) => setTimeout(() => reject(new Error("EXECUTION_TIMEOUT")), 5000))
    ]);
    
    await executionContext.idempotencyStore.save(idempotencyKey, result);
    return result;
  } catch (error) {
    await executionContext.idempotencyStore.clearLock(idempotencyKey);
    throw new ToolExecutionException(`Fout tijdens uitvoering van ${toolCall.name}`, error);
  }
}

Cost of the mitigation:

Failure mode 4: Provider-side cancellation, AbortControllers, and token costs

Failure mode: The client aborts the HTTP stream using an AbortController because a configured client timeout expires or the user clicks "cancel." The assumption that stopping the stream immediately halts cost and processing on the provider side is often incorrect.

Detection: Monitoring at the API gateway shows a discrepancy between the number of tokens received by the client and the number of tokens billed in the provider's usage reports. For an in-depth analysis of response times, active cancellations, and setting up maximum processing times, we refer to the article on timeouts, cancellations, and deadline budgets.

Mitigation: When a client triggers an AbortController , the underlying TCP connection closes. LLM providers, however, process the stopping of a stream asynchronously. The model can keep generating internally until the next network write fails. For accurate cost control, the application system should register the dropped request in an observability module. For setting up such monitoring, see the document on observability and logging in LLM integrations.

To avoid unexpected costs at large volumes, system architects must also account for the rates charged for interrupted generated tokens. Information on rate structures and limits is available in the guide on rate limits and cost management.

The pseudocode example below demonstrates the correct handling of an AbortController in combination with a timeout budget and catching the cancellation event.

async function fetchStreamWithCancellation(apiEndpoint, payload, clientSignal) {
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), 10000);
  
  if (clientSignal) {
    clientSignal.addEventListener("abort", () => controller.abort());
  }

  try {
    const response = await fetch(apiEndpoint, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(payload),
      signal: controller.signal
    });
    clearTimeout(timeoutId);
    return response.body;
  } catch (error) {
    clearTimeout(timeoutId);
    if (error.name === "AbortError") {
      logMetric("stream_cancelled_by_client_or_timeout");
      throw new RequestCancelledException("API-verzoek geannuleerd via AbortSignal.");
    }
    throw new NetworkException("Transportfout tijdens API-verzoek", error);
  }
}

Cost of the mitigation:

Failure mode 5: Tension between streaming and strict structured output

Failure mode: Many LLM providers offer the ability to enforce strict JSON schemas via proprietary or open standards (structured output). When this is combined with streaming, an architectural tension arises. To guarantee that the output complies 100% with the supplied schema, the provider engine must constrain the generation process at the token level with a grammar mask (constrained sampling). Intermediate text deltas by definition never satisfy the full schema until the closing token has been processed.

Detection: The application receives streaming chunks that aren't valid JSON on their own, or the provider returns an API error (such as HTTP status 400) because the combination of stream: true and certain strict schema options isn't supported by that particular engine.

Mitigation: Accept that when streaming tool arguments, syntactic validation can only take place at the end of the stream. Use the strict schema as a tool to force the provider's model to generate valid JSON, but perform the final schema validation (for example, via Zod or JSON Schema validators) only in your own application code after the stream has fully accumulated. For an in-depth look at schema enforcement, read the overview of structured output and JSON schemas.

If it's critical for your business that failures from invalid JSON are kept to an absolute minimum, it may be necessary to fall back on robust processing strategies. For further continuity solutions, consult the guide on graceful degradation during LLM outages.

Cost of the mitigation:

Decision tree and integration checklist for streaming tool calls

Combining streaming and tool calls is a powerful pattern, but it introduces significant overhead. Use the decision tree and considerations below to determine whether streaming is the right choice for your specific tool-call scenario.

When to use streaming with tool calls?

When not to use streaming with tool calls?

Production checklist for rollout

  1. Set up buffers: Is there a separate buffer present per tool_call.index that's only released at the stream's closing element?
  2. End-signal check: Does the parser explicitly check whether the stream is closed with finish_reason: "tool_calls" or finish_reason: "stop"?
  3. Idempotency secured: Is the tool call ID supplied by the provider passed to the executing backend service as the idempotency key?
  4. Timeout budgets: Is a hard time limit set for the entire duration of the stream, tied to an AbortController?
  5. Error path handling: On a fragmented or failed JSON parse, is the action fully canceled without partial execution?

This article does not cover handling retries and backoff, setting up a central proxy architecture, or selecting suitable models for tool use; see the articles on retries and exponential backoff for LLM calls, self-hosting an LLM gateway, and choosing the right model for function calling.