How we cut the wait on short requests by up to 41%
A short request spent up to 41% of its wait opening a connection to the provider. Our platform now keeps those connections open.
Every request you send to CArouter is passed on to a provider serving the model you named. Until this week, our platform opened a new connection to that provider for each request.
It now keeps its connections to the providers open between requests. In production, a short request reaches its answer up to 41% sooner.
The short version
- What a client sees. The median time to the answer of a short request went from 597 to 352 ms through a partner in Europe, and from 752 to 663 ms on our own servers.
- What we changed. Our platform keeps its connections to the providers open and reuses them. A request on a kept connection takes one round trip to the provider instead of three.
- What else changed. A request has a fixed number of attempts. A provider that is full no longer uses one.
- What it costs. The first request after a quiet period still opens a connection.
Three round trips for one request
A request to a provider travels over an encrypted connection. Opening one takes two exchanges before the provider reads a word of the request: the first opens the connection, the second proves who the provider is and sets up the encryption.
The request itself is the third exchange. Each one is a round trip between our platform and the provider, and a round trip across the Atlantic takes about a tenth of a second.
A long answer hides that cost, because writing it takes many seconds. A short request shows it in full. Agents send short requests by the hundreds: choose a tool, classify a message, answer yes or no.
What a new connection adds
We measured it from our platform in production, with a request that asks a provider for its list of models. No model works on that request, so the figure is the path alone.
A new connection added 240 to 300 ms to a request on our own servers, about 80 ms at a Canadian partner, and 420 to 750 ms at our partners in Europe.
Every request paid that time, and so did every retry at another provider.
What changed in our platform
Our platform now opens a connection to a provider once and sends the following requests on it. Each request in progress has a connection of its own, so no request waits for another to finish.
A connection stays open for a short while after its last request. While it waits, the platform checks that the path to the provider is still alive, and it closes a connection whose path has died.
A request that finds its connection already closed is sent again once, on a new connection, as long as the provider has answered nothing. Any other failure goes to the failover you already know: the next provider serving the same model.
What we measured in production
We compared the day before the change with the day after, over the same hours, for requests answered in 16 tokens or fewer. The figure is the median time between sending the request to the provider and receiving its complete answer.
Through a partner in Europe, the median went from 597 to 352 ms, which is 41% less. On our own servers it went from 752 to 663 ms, 12% less.
On our own servers the median moved less than the connection measurement suggests. An engine's load differs from one day to the next, and that difference can weigh as much on the median as a connection does.
A full provider no longer uses an attempt
When a provider fails, our platform sends the request to the next one serving the same model, before the first byte reaches you. A request has a fixed number of such attempts.
Each provider endpoint also has a limit on the requests it holds at once. Until this week, an endpoint that was full used one attempt, although nothing had been sent to it. In a busy minute, a request could spend all its attempts on full endpoints and be refused while another endpoint had room.
An attempt is now a request actually sent. Full endpoints are passed over at no cost, and the request goes to the first endpoint with room.
The routing rules are the same
- A kept connection passed the same certificate check as a new one, when it was opened, and it serves that provider only.
- An offer served in Canada by a Canadian-owned provider is still tried first. A request crosses a border only for an API key that allows foreign providers.
- Passing over a full endpoint follows the same order. A key that keeps its requests in Canada is refused when every Canadian endpoint is full, exactly as before.
- An endpoint that a customer added themselves gets a new connection for every request, because its address is checked before each one.
What we checked before production
We tested the change against the real providers first, from a separate environment, with about 80 small requests.
A complete streamed answer, a caller hanging up in the middle of a stream, several requests at once, and answers a minute long with nothing sent in between all behaved as before. Every stream was billed once and gave back its place at the provider.
What it costs
The first request after a quiet period opens a new connection and pays the three round trips.
The saving is a fixed number of milliseconds for each request. A long answer gains the same milliseconds, which is a small share of the many seconds it takes to write.
For a short request, the path to the provider is where a router saves time.
Compare available models and providers. Read the technical routing rules.