Stream a response

Enable streaming

Set stream to true on a chat-completions request. Router sends ordered server-sent events containing content deltas, followed by a final finish reason, usage, and [DONE].

1import os2from uuid import uuid43from openai import OpenAI4
5client = OpenAI(6    base_url=os.environ["ROUTER_BASE_URL"],7    api_key=os.environ["ROUTER_API_KEY"],8)9
10request_id = str(uuid4())11stream = client.chat.completions.create(12    model="gemma3:1b",13    messages=[{"role": "user", "content": "Hello!"}],14    stream=True,15    extra_headers={"Idempotency-Key": request_id, "X-Max-Charge-Credits": "5.000000"},16)17
18for chunk in stream:19    if chunk.choices:20        print(chunk.choices[0].delta.content or "", end="")

Handle completion and failure

An opened connection is not a completed answer. Track content as it arrives and distinguish a finished response from a failed or interrupted stream.

A normal stream contains:

1. Zero or more chunks with choices[0].delta.content.
2. A final chunk with finish_reason set to stop or length and a usage object.
3. The terminal data: [DONE] marker.

Event by event

Three common outcomes, event by event:

data: {"id":"$REQUEST_ID","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello"},"finish_reason":null}]}

The owner can inspect GET /v1/credits/requests/{request_id} with an application key or signed-in account token even if the stream disconnects. Accepted provider-reported partial usage can still be charged; token counts are not independently verified. The receipt shows the final amount when settlement completes.

Router does not replay a stream. Sending the same Idempotency-Key again returns request_already_exists (409); use the receipt to resolve the original request.

Stop receiving a stream

Close or abort the client request when the user stops receiving text. Release your stream reader and show that receiving stopped. Accepted paid work may continue to completion and settlement after disconnect; HTTP abort is not guaranteed durable cancellation. Check the saved credit receipt to resolve the result and charge before starting another paid request.

See errors and limits for recovery guidance.