Stream a response

Enable streaming

Set stream to true on a chat-completions request. Router sends ordered server-sent events containing content deltas, followed by a final finish reason, usage, and [DONE].

1import os2from openai import OpenAI3
4client = OpenAI(5    base_url=os.environ["ROUTER_BASE_URL"],6    api_key=os.environ["ROUTER_API_KEY"],7)8
9stream = client.chat.completions.create(10    model="gemma3:1b",11    messages=[{"role": "user", "content": "Hello!"}],12    stream=True,13)14
15for chunk in stream:16    if chunk.choices:17        print(chunk.choices[0].delta.content or "", end="")

Handle completion and failure

An opened connection is not a completed answer. Track content as it arrives and distinguish a finished response from a failed or interrupted stream.

Stop a request

Close or abort the client request when the user stops generation. Release your stream reader and show a stopped state. Keep cancellation and request errors distinct in your interface.

See errors and limits for recovery guidance.