Retries, timeouts & cancellation
Recover from temporary failures without duplicating uncontrolled work.
Set a request deadline
Give every Router call an application-level timeout. Choose a deadline that fits the model and prompt, then show a clear waiting state. A network connection staying open does not guarantee that generation will finish.
Retry only temporary failures
Good retry candidates include 429 responses and selected 503 responses such as no_provider_available or request_capacity. Respect Retry-After when present. Use exponential backoff, random jitter, and a small attempt limit.
const retryable = new Set([429, 503]);
async function waitBeforeRetry(response: Response, attempt: number) { const header = Number(response.headers.get("retry-after")); const seconds = Number.isFinite(header) ? header : Math.min(2 ** attempt, 8) + Math.random(); await new Promise((resolve) => setTimeout(resolve, seconds * 1000));}Do not retry malformed requests, unsupported parameters, invalid credentials, or an unknown model until you change the request.
Treat a retry as a new generation
The standard chat-completions request does not carry an idempotency key. A retry can generate a different answer and may start new work. Keep the attempt count bounded and avoid automatic retries for actions where repeated output could cause side effects in your application.
Cancel when the user leaves
Abort the HTTP request if the user stops generation, navigates away, or exceeds your deadline. Router observes the closed connection and cancels the active request lifecycle. Already-running provider work may take a short time to stop.
const controller = new AbortController();const timeout = setTimeout(() => controller.abort(), 45_000);
try { const response = await fetch(`${process.env.ROUTER_BASE_URL}/chat/completions`, { method: "POST", signal: controller.signal, headers: { Authorization: `Bearer ${process.env.ROUTER_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ model: "gemma3:1b", messages: [{ role: "user", content: "Hello!" }], }), }); // Inspect response.ok and the error body before reading a success payload.} finally { clearTimeout(timeout);}Use the X-Request-Id response header when correlating an attempt with usage history.