How CArouter protects your AI conversations at the provider
Every request CArouter routes runs on a provider's machine, and some provider failures look exactly like a successful answer. Here is what production showed us this autumn, and the platform features that protect what a provider receives from your conversation and what it sends back.
CArouter sends each request to one of several providers serving the model you asked for. Most provider failures announce themselves with a timeout or an error code, and our platform then moves the request to the next provider before the first byte reaches you.
Some failures arrive as a perfectly normal response instead. Our production traffic showed several of them this autumn, and each one led to a change in the platform. This post covers what a provider sends back, what it receives from your conversation, and where your conversation runs.
An answer made of exclamation marks
On 5 November, for about five hours, one endpoint serving Qwen3.6 35B-A3B answered some long prompts with a single character, the exclamation mark, repeated until the token limit. Each response carried a success status and a full token count, and a chat turn lost one to four minutes on it.
The character is the clue. When a model's arithmetic overflows, its next-token probabilities become invalid and the sampler picks token 0 at every step. In Qwen's vocabulary, as in many others, token 0 is the exclamation mark.
Our platform now treats that pattern as a provider failure:
- A response with 128 exclamation marks in a row, in the answer, the reasoning or a tool call's arguments, counts as a failed attempt.
- A non-streamed response is retried on the next provider serving the same model, exactly as after a timeout.
- A stream is held back while its text is all exclamation marks, and its first real token releases it, so a healthy answer arrives at the same speed. A stream that collapses from its first token is retried elsewhere, and you receive the healthy answer only.
- A stream that collapses after you have received part of it ends with an error frame your application can retry, because restarting a half-delivered answer is your decision.
- Either way the attempt is free, and your Usage page labels it "Degenerate answer".
On the endpoint concerned, we also corrected the cause: the model's recurrent memory now runs at the numeric precision its authors specified.
A route one provider lacks
Providers often serve the same model on Chat Completions and differ on other routes, such as the Responses API. A call to a route one provider lacks used to come back as a 404 that read like a missing model.
The catalog now records the routes each offer lacks. The router sends a request only to offers that serve its route, a "route not supported" answer moves it to the next provider, and the documentation lists for each model the routes that work.
An endpoint that slows down before it is full
Each provider endpoint has a concurrency ceiling, and an engine slows down well before it reaches it. In October, long prompts waited more than 90 seconds on busy endpoints while another endpoint for the same model had room.
An endpoint now counts as busy from half of its ceiling. A busy endpoint stays available and moves behind that provider's endpoints that have room, so a long prompt goes where it will be answered sooner.
Knowing whose problem a failure was
Every failed request on your Usage page states whose problem it was: the platform's, the provider's, your own request's, or a capacity limit. A failure of unknown cause counts as ours, because a red row we have to explain costs less than an outage filed under someone else's name.
What a provider receives from your conversation
- With the PII filter switched on for an API key, detected personal information such as names, email addresses, phone numbers, Social Insurance Numbers and health numbers is replaced with placeholders before the prompt reaches the provider, and restored in the answer.
- Detection is best effort, and pseudonymized text is still personal information. The placeholder map exists only in memory for the duration of the request, and a key that requires the filter is refused whenever the filter is unavailable.
- On the engines we run, prompt caching is private to your workspace: a cached prompt prefix is reused by your own requests only. Settings > Privacy & data switches that reuse off entirely.
- A provider's error message can quote your request. The same settings page lets you keep only each error's classification and erases the text already stored.
- When a provider's agreement asks for a customer identifier to trace abuse reports, it receives a keyed hash of your workspace. Your own users' identifiers stay with you.
- A provider is marked zero-retention only after someone has read its published policy and recorded the citation. Every other provider shows "retention not verified", so you choose with the facts in hand.
- Every connection to a provider verifies its TLS certificate, including endpoints that customers add themselves.
Where your conversation runs
- An offer served in Canada by a Canadian-owned provider is tried first, whatever a foreign offer costs.
- Failover stays in Canada. A request whose Canadian providers are all unavailable is refused, and it crosses a border only for an API key that explicitly allows foreign providers.
- Failover stays on the model you named. A fine-tuned model and its base have different weights, so each one is served under its own name only.
- Each response states its serving region in the X-CArouter-Region header, read from the provider's own report where it gives one. An unrecognised region is reported as unverified rather than as Canada.
Check it from your own logs
The X-CArouter-Provider and X-CArouter-Region headers name who answered and where. Your Usage page lists every failed request with its cause, so you can verify each of these rules on your own traffic.
Compare available models and providers. Read the technical routing rules.