Streaming and conversations

Streaming responses

Streaming returns content in pieces as it is generated. It changes delivery, not whether a request is billed.

For a model supporting Chat Completions streaming:

{
  "model": "REPLACE_WITH_AVAILABLE_MODEL_ID",
  "messages": [{"role": "user", "content": "Hello"}],
  "stream": true
}

Parse the response correctly

OpenAI-compatible chat streams typically use SSE events beginning with data:. Text increments are usually in choices[0].delta.content, with data: [DONE] marking completion.

• Do not parse the entire stream with a single response.json() call.
• Handle events without text, including roles, empty increments, tool calls and errors.
• Network chunks are not necessarily complete SSE events; buffer and parse at event boundaries.
• An interrupted response or missing completion marker is not a complete success.

A suitable client or SDK can handle this for you. Responses and Claude use different event structures; do not mix parsers.

Include conversation history

With Chat Completions, include the messages you want the model to remember in the next request:

[
  {"role": "user", "content": "Introduce the solar system."},
  {"role": "assistant", "content": "The solar system includes the Sun and the objects orbiting it."},
  {"role": "user", "content": "Which planet is Earth from the Sun?"}
]

Longer history can increase input usage. Reduce or summarize history when approaching the model's context limit.

Retry after an interruption?

Check the client result and usage logs first. A disconnected client does not prove that the server did not process the request. Retrying can create another billed request.

For rate limits such as 429, wait and use bounded retries. Fix invalid keys, model names and parameters before retrying.