Smart routing with Kev: our benchmark

Today every request to CArouter names the model that answers it. We want to offer one that names none: write auto, and CArouter picks the model best suited to the request, then serves it the way you prefer, cheapest, fastest or balanced.

That takes a decision model, a small model whose only job is to read a request and say what kind of work it is, in a few milliseconds. We tested three candidates on 1,020 real requests, in English and in French, to find out whether one is good enough to trust with that call and what it would cost to run.

The short version

• Accuracy. Jev, a hosted commercial model, identifies the kind of request correctly 81% of the time in English and 76% in French. Kev-4B, an open model we can run ourselves, reaches 76% and 70%. Its smaller sibling Kev-0.8B reaches 70% and 64%. Laya, the third candidate, stays under 60% out of the box.
• Confident mistakes. Jev is wrong while claiming to be sure on one request in eight. Kev-4B is on one in thirty, and Kev-0.8B never was. For a system that should only act when it is sure, that matters more than the accuracy gap.
• Bigger is not better. The largest Kev, at 9 billion parameters, scores no better than the 4-billion one and needs twice the memory. The 4B is the sweet spot.
• Speed. On a datacenter GPU, one decision takes 43 milliseconds on Kev-0.8B and 58 on Kev-4B, an overhead a customer will not notice. Without a GPU, only Laya answers fast enough to be usable, and it is the least accurate.
• The best candidate. Kev-4B: accurate enough, honest about its doubts, open, and fast on a GPU. Whether it is good enough to select models for our customers is a question this test cannot settle, because no public dataset contains the kinds of requests they send. That is the next test to build.

Why pick the model for you

CArouter gives access to a catalog of open models, all served from Canada, through one API. Most customers choose one model when they integrate and never revisit the choice. But a request to fix a piece of code, a request to translate a paragraph and a request to summarize a contract do not need the same model, and the best one for each is rarely the most expensive.

Today a customer can already tell us how to serve a given model: at the lowest price, at the highest speed, or balanced between the two. That preference chooses among the providers that serve the model the customer named. It cannot choose the model itself, and it can never move a request outside Canada. Automatic selection would add the first of those two things while keeping the second.

What a decision model is

Ask a chat model to classify a support ticket and it writes a sentence back, which then has to be read and interpreted. A decision model does not write. It takes a text and a typed question, which of these categories, how much on this scale, yes or no, and returns the answer with a confidence score, in one pass.

There is nothing to interpret and nothing it can make up, because it never generates. That makes it the right tool to sit in front of every request: fast, predictable, and honest about how sure it is.

Jev is the commercial version of this idea, sold by TypeSafe as a hosted service. Kev is an open reproduction of it, in three sizes, that anyone can run on their own hardware. Laya is a second open model that answers the same kinds of questions with a different design.

How we tested

We asked each model one question about 1,020 requests: what kind of task is the user asking for, among nine kinds such as writing code, solving a math problem, summarizing a passage or brainstorming ideas. The requests and their correct answers come from three public datasets that people wrote and labelled by hand, not from us.

The same requests exist in English and in French, so the two languages are compared on identical material. Neither Kev nor Laya had ever seen these datasets. Jev's training data is not public, so it may have.

Speed was measured separately, on three machines: a laptop's processor alone, the same laptop's small GPU, and a rented datacenter GPU. Every request was new, so no shortcut or cache could help.

What we found: accuracy

Jev leads, Kev follows five to six points behind, Laya trails. Two things are true for every model. Code and math are easy: Jev and Kev-4B get them right on every English request, and Kev-4B keeps a perfect score on math in French. And French costs everyone five to eight points, Jev included, so it is a property of the task rather than of one model.

The scores count an answer as right when it lands in the right family of tasks. The datasets themselves label the same kind of request in more than one way, a question about a passage being called a summary in one place and a reading question in another, so the finer nine-way score is lower for every model and in the same order.

The second chart is the one that decides. A system that picks a model on a customer's behalf should only do so when it is sure, and otherwise fall back to a sensible default. That means setting a confidence bar. A wrong answer given with high confidence sails over that bar, so what matters is not only how often a model is wrong, but how often it is wrong while sure.

Jev is the most accurate and the most overconfident: 12% of its answers are confident mistakes. Kev-4B is at 3% and Kev-0.8B at 0%. A confidence bar catches Kev's mistakes and lets Jev's through.

What we found: speed, and where each one can run

On a datacenter GPU every model is fast enough. Kev-0.8B answers in about 45 milliseconds whatever the request length up to a thousand words. Kev-4B needs 58 milliseconds on a short request and about a second on a very long one. A laptop GPU gives the same answers two to seven times more slowly, and cannot hold Kev-4B at all.

Without a GPU, the picture changes. Kev-0.8B takes over a second per decision on a laptop processor, and Kev-4B close to a minute. Only Laya answers in a quarter of a second, about four decisions a second per machine, and it is the least accurate of the six.

Memory decides the rest. Kev-0.8B needs 3.4 GB of GPU memory, Kev-4B 17 GB and Kev-9B 32 GB. On a machine that also serves a language model, that memory is taken from the customers' own requests.

What deploying each one would mean

• As a decision service, on a GPU with room to spare: Kev-4B. Five to six points behind Jev, a quarter as often confidently wrong, free to run under an open licence, and compatible with Jev's API, so a team already using Jev could switch by changing an address. 17 GB of GPU memory and 58 milliseconds would be the price, and the memory is the real cost on a machine that also serves language models.
• As a decision service where memory is tight: Kev-0.8B. 3.4 GB, 45 milliseconds, never confidently wrong in our test. It would give up six points of accuracy against the 4B, and that is the whole trade.
• Behind automatic model selection: Kev, under a confidence bar, once one more test is passed. The 45 milliseconds it would add before an answer starts are acceptable, and its mistakes are the kind a confidence bar catches. What our test does not show is accuracy on the kinds of requests our customers actually send, multi-step agent work and very long documents among them, because no public dataset contains them. That test has to exist, and be passed, before anyone's requests are routed on Kev's word.
• Running without a GPU: Laya, once trained for the job. It is the only candidate that answers quickly on an ordinary processor. Out of the box it reads the task right barely half the time, and its authors say its accuracy comes from training it on your own examples, which we have not done.
• Jev: the reference, not a candidate. It is the most accurate model in the test and a hosted service from a San Francisco company that makes no statement about where requests are processed, which settles it for a product whose whole promise is that requests stay in Canada. It is also the most overconfident, the one flaw a confidence bar cannot fix.

Why the choice has to stay put

A conversation is a series of requests, each one carrying everything said before it. Most providers keep a short-term memory of that shared beginning, so that the next turn is cheaper and faster if it reaches the same provider. CArouter already keeps a conversation with the provider that holds that memory, for as long as it lasts.

If the model chosen for a conversation changed from one turn to the next, that saving would be lost every time, and the tone of the answers would change mid-conversation. So automatic selection has to hold its choice: it remembers the model it picked for a conversation and only changes it when the decision model disagrees with confidence.

That is why the confident-mistake rate is the number this whole test turns on. One confidence bar decides two things, whether a choice is trusted at all and whether a later turn may change it. And if the decision model is ever unavailable, requests go to a default model rather than being refused.

What this does not prove

One question, one run, one set of requests. A differently worded question or a different set could reorder the middle of the table. It could not bring Laya into contention or move Jev off the top.

The processor figures come from a laptop, and a server processor would do better. They also lack the optimized code Kev was written for, which only exists for GPUs. So they are a ceiling on how slow Kev is without a GPU, not a prediction for a given server.

Jev was timed from Canada over the public internet, at 185 milliseconds per decision. That is what a customer here would experience, not the model's own speed.

Where this leaves us

Kev-4B is the candidate we would deploy, and the two open questions are practical ones: which machine can spare 17 GB for it, and whether it holds up on a test built from the requests our customers really send. A decision service customers call directly, the way they call embeddings or reranking today, needs only the first answered; automatic model selection needs both. Every measurement behind this article is kept, and we will share it with anyone who asks.