Streaming responses in LLM APIs

When building an application that uses large language models (LLMs) like Claude or DeepSeek, you quickly notice that text generation takes time. Because models predict text token by token, a long response can take seconds to complete. Without streaming, your user is left staring at a blank loading indicator. Streaming responses solve this problem.

How does it work? (Server-Sent Events)

Almost all modern LLM APIs use Server-Sent Events (SSE) for streaming. Instead of closing the HTTP connection as soon as a single large JSON payload is calculated, the server keeps the connection open.

As the model generates tokens, the API pushes them directly to the client as individual chunks of text. Once the stream is complete, the server sends a specific terminator, often [DONE], after which the connection closes.

When is streaming essential?

For simple, short data transformations (such as JSON extraction or classification) on the backend, it is often better to wait for the full response. However, streaming is essential in the following scenarios:

• Chat interfaces: For a natural, human-like response experience.
• RAG applications with vector search: Because the retrieval step (searching your vector database) already adds latency, you want to make the subsequent LLM generation visible immediately to mask the waiting time for the end user.

Implementation Pitfalls

Building a robust streaming client comes with a few challenges:

• Markdown Rendering: Because tokens arrive incrementally, your Markdown structure is often incomplete in real-time (e.g., an open code block without a closing tag). Use a parser that is forgiving of incomplete syntax to prevent UI flickering.
• Error Handling: In the event of a timeout mid-stream, SSE does not always return a standard HTTP 500 error via the headers. You must catch error messages within the data stream.
• Token Billing: In the past, streaming APIs did not return a total token count. Nowadays, with most providers, you can include the final usage statistics in the last chunk via a parameter (such as stream_options: {"include_usage": true}).

Pseudocode: Consuming an SSE Stream

Below you can see the basic logic to query a streaming endpoint in the frontend. We use the standard Fetch API to decode the ReadableStream.

async function fetchLLMStream(prompt) {
const response = await fetch('https://api.example.com/v1/chat/completions', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: "your-chosen-model",
messages: [{ role: "user", content: prompt }],
stream: true // Enable streaming
})
});

const reader = response.body.getReader();
const decoder = new TextDecoder("utf-8");

while (true) {
const { value, done } = await reader.read();
if (done) break;

const chunk = decoder.decode(value);
const lines = chunk.split('\n').filter(line => line.trim() !== '');

for (const line of lines) {
if (line === 'data: [DONE]') return;

if (line.startsWith('data: ')) {
const data = JSON.parse(line.slice(6));
const token = data.choices[0]?.delta?.content || "";

// Update the UI incrementally
appendToChatInterface(token);
}
}
}
}

With this setup, you ensure smooth, real-time interaction, regardless of the total processing power the model requires.