OpenJev vs Kev-4B: benchmark on multiple CPU/GPU hardware
OpenJev asks an ordinary Qwen3.5-4B to pick an option letter, with no training for the job. On the two benchmarks we used to choose Kev-4B, it named the kind of task right on 95.7% of our 328 requests against 88.7% for Kev-4B, though it was confidently wrong far more often on public prompts.
Kev-4B is the decision model CArouter serves at /v1/decisions. We chose it on 1,020 public prompts, then measured it on 328 requests written like the work our customers send.
OpenJev takes another route and trains nothing. It asks an ordinary chat model to pick an option letter, then reads how likely each letter was. We ran it on both benchmarks, scored by the same script as Kev, to find out whether it would have been the better choice.
Read the first benchmark. Read the second, with Kev's launch.
The short version
- On our requests. OpenJev's Qwen3.5-4B named the kind of task right on 95.7% of our 328 requests, against 88.7% for Kev-4B, with the same score in English and in French.
- Kev's weak spots. It recognised casual chat 97% of the time and a single step of an agent's work 88% of the time. Kev-4B gets those right a third and half of the time.
- On public prompts. It led there too, at 79% in English and 81% in French, level with the hosted Jev. But it was confidently wrong on 19% of them, against 3% for Kev-4B.
- Confidence on our requests. It was at least 90% sure of 294 answers and wrong on two of them. Kev-4B was that sure of 183 answers and never wrong.
- The browser file. The 3 GB file openjev.com downloads scores as well as the full model. The two smaller models the site offers are far behind and often confidently wrong.
- Speed. On this laptop's GPU, the browser file makes a three-question decision in 267 milliseconds on a short request. On a server processor alone it takes 9 to 12 seconds, slower than both Kev models, where Kev-4B answers in 143 milliseconds on the GPU that serves it.
What OpenJev is
OpenJev is an independent open-source project, renamed SemIf by its author, with no tie to TypeSafe, the company behind Jev. Its code is under the MIT licence, and openjev.com runs it inside the browser. openjev.com · Source code.
It puts the text and the question into a prompt, lists the options as letters A, B, C, and asks the model to reply with one letter. The model never writes anything. OpenJev reads the probability the model gave each letter, and those become the probabilities of the answer.
Its best model is Qwen3.5-4B. Kev-4B was trained from the base version of that same model, so this test also asks a plain question: does Kev's training beat what Qwen's own instruction tuning already knows?
How we tested
We asked OpenJev exactly what we had asked Kev. On the public prompts, that is one question with nine kinds of task. On our requests, it is three questions: the kind of task, how hard it is, and whether the answer will be long.
One script scored every model. Before scoring anything new, it recomputes the figures of both previous articles from their raw answers and stops if a single one differs.
OpenJev ran with its own code, at a pinned version. As a check, we first ran its own 144 test decisions. Our setup gave the answer its authors published on all 144 of them, and the same 0.813 score.
We ran each OpenJev model twice: at full precision, where its published scores come from, and as the smaller file the website actually downloads. OpenJev takes one question at a time with a list of options, so a difficulty scale became five options and a yes-or-no question became Yes and No.
We also ran Kev-0.8B and both Laya models on our 328 requests, which only Kev-4B had seen. Jev and Kev-9B are missing there: Jev is a paid hosted service we did not call again, and Kev-9B does not fit on the laptop we used.
What we found: accuracy
On our requests, OpenJev's Qwen3.5-4B is the most accurate model we have measured: 95.7% at full precision and 97.3% as the browser file, against 88.7% for Kev-4B. English and French score the same.
On the public prompts, it matches Jev and passes Kev-4B by four points in English and eleven in French. French cost every model in the first benchmark five to eight points, and it cost OpenJev nothing.
| Model | Public prompts, English | Public prompts, French | Confidently wrong, public, English | Our 328 requests | Confidently wrong, our requests |
|---|---|---|---|---|---|
| Jevhosted | 80.9% | 76.0% | 11.7% | not run | not run |
| Kev-9B | 74.3% | 66.5% | 4.8% | not run | not run |
| Kev-4B | 75.7% | 69.8% | 3.3% | 88.7% | 0.0% |
| Kev-0.8B | 69.6% | 63.7% | 0.0% | 82.9% | 0.0% |
| Laya | 57.6% | 46.9% | 0.0% | 50.9% | 0.0% |
| Laya multilingual | 55.0% | 47.1% | 3.7% | 42.4% | 3.0% |
| OpenJev Qwen3.5-4Bfull model | 79.3% | 80.8% | 19.3% | 95.7% | 0.6% |
| OpenJev Qwen3.5-4Bbrowser file | 80.4% | 80.3% | 19.0% | 97.3% | 0.9% |
| OpenJev Qwen3.5-4Bbrowser file, page's wording | 79.3% | 77.4% | 20.5% | 92.7% | 1.8% |
| OpenJev MiniCPM5-2Bfull model | 76.3% | 75.0% | 20.2% | 75.0% | 13.7% |
| OpenJev MiniCPM5-2Bbrowser file | 73.0% | 74.6% | 13.0% | 77.1% | 9.1% |
| OpenJev Qwen3-0.6Bfull model | 59.8% | 45.6% | 35.6% | 59.1% | 23.5% |
| OpenJev Qwen3-0.6Bbrowser file | 59.6% | 45.8% | 34.1% | 61.9% | 19.5% |
The site's smaller models change the picture. MiniCPM5-2B, the default on a desktop, holds up on the public prompts but falls to 75% on ours. Qwen3-0.6B, offered for phones, trails every model except Laya.
Where the difference comes from
The gap sits in two kinds of request. OpenJev recognised casual chat 97% of the time and single agent steps 88% of the time. Kev-4B reads a general-knowledge question as research, and an agent's next step as code.
Kev-4B is better with documents: 93% against 82% on research and 100% against 85% on summaries. OpenJev tends to read a long research request as business writing. Both are perfect on code, math, business writing and translation.
What we found: confidence
A model that decides on someone's behalf should act only when it is sure. The first article measured how often a model is wrong while at least 90% sure, and here the two benchmarks disagree about OpenJev.
On our requests, OpenJev was that sure of 294 answers out of 328 and wrong twice, both times calling a research request business writing. Kev-4B was that sure of 183 answers and never wrong. A bar at 90% would let OpenJev decide nine requests in ten on its own, and Kev-4B just over half.
On the public prompts, OpenJev was confidently wrong on 19% of the English requests, against 3% for Kev-4B and 12% for Jev. Many of those labels are debatable.
The dataset files "How many U.S. Presidents have served in the military?" under brainstorming. OpenJev called it a factual question with 99.8% confidence, and we counted it wrong, as we did for every model.
Not all of them are the labels' fault. Counted by family of task, where such disputes stop mattering, OpenJev is still confidently wrong on 15% of the public prompts. Its confidence held on requests like ours, and it needs checking on anything else.
The browser file and the smaller models
The 3 GB file openjev.com downloads is a 4-bit copy of Qwen3.5-4B. It scored 97.3% on our requests and 80% on the public prompts in both languages, as well as the full model.
Two things change inside the browser. The page asks its question in its own words, and with them the same file scores 92.7% on our requests instead of 97.3%, still ahead of Kev-4B. It also gives the model a window of 2,048 tokens, and 14 of our 328 requests do not fit in it.
The two smaller models are overconfident as well as less accurate. MiniCPM5-2B was confidently wrong on 14% of our requests and Qwen3-0.6B on 24%, where Kev-4B and Kev-0.8B were never confidently wrong.
Difficulty and long answers
On the other two questions the two 4B models are close. OpenJev places a request on our exact difficulty level 36% of the time, against 41% for Kev-4B, and within one level 87% against 88%.
Neither is good at predicting a long answer. Only 10 of our 328 requests need one, and OpenJev flagged 4 of them against 3 for Kev-4B.
Speed and memory
OpenJev asks one question per pass over the text. Three questions take three passes, or one pass over the text and three short ones, which is the faster mode its code offers. Kev reads the text once and answers every question from that single pass.
On this laptop's GPU, the browser file of OpenJev's Qwen3.5-4B makes a three-question decision in 267 milliseconds on a short request and 2.2 seconds on a 4,000-token one. Kev-4B does not fit on this GPU, so the two 4B models were never timed side by side. On the first benchmark's datacenter GPU, Kev-4B took 58 milliseconds and 1 second.
Kev-4B does fit on a MacBook Air with Apple's M4 chip and 24 GB of memory, which its GPU shares with the processor. Through MLX, Apple's machine-learning library, which Kev supports, the same decision took 1.3 seconds on a short request and 14 seconds on a 4,000-token one.
Through PyTorch, the library our own server runs Kev with, it took 3.6 seconds on a short request and 20 seconds on a 1,000-token one. On a Mac, PyTorch has no fast version of Kev's recurrent layers and reads the text again for each question, and at 4,000 tokens it needed more memory than the Mac has. On six check prompts, both gave the same answers as the datacenter GPU.
The full Qwen3.5-4B needs 8.7 GB for its weights, more than this laptop's GPU holds. We measured its accuracy with part of it kept in system memory, and did not time it.
The browser file takes about 3 GB on the GPU, everything included. Kev-4B used up to 17 GB on the first benchmark's GPU, and holds under 9 GB on the machine that serves it.
On a processor alone
We timed the same three-question decision on two server processors, with no GPU in reach. One is a virtual machine with four AMD EPYC (Milan) cores and eight threads. The other is eight of the ten fast Arm cores of the GB10 machine that serves Kev, run at the lowest priority beside the models it serves. Kev ran the way our server runs it, reading the text once and then each question, and OpenJev ran its browser file in the faster of its two modes. The dotted lines are Kev on the GB10's GPU, on the same requests.
On a processor, OpenJev's 4B file is the slowest of the three. On the GB10's cores it took 8.9 seconds on a short request and 87 seconds on a 4,000-token one, against 6.1 and 53 seconds for Kev-4B and 1.3 and 15 seconds for Kev-0.8B. On the EPYC cores, OpenJev and Kev-4B were close, at 12 and 10 seconds on a short request, and Kev-0.8B took 1.8 seconds.
None of this works while a caller waits. On the GPU of the same machine, Kev-0.8B answers in 52 to 293 milliseconds and Kev-4B in 143 milliseconds to 1 second, about 25 to 50 times faster than the same model on its fastest processor.
Two details matter on a processor. The EPYC has no fast arithmetic for the 16-bit numbers Kev is stored in, so Kev-0.8B ran seven to eight times slower in that format than in 32-bit numbers. The GB10's Arm cores have that arithmetic, and there the two formats ran at about the same speed. The EPYC machine's eight threads are also only four cores: with four threads, Kev was about 20% slower.
On the GB10, moving a model to the processor would not free memory either, since its processor and GPU share one pool and a model takes the same room on both. Kev-4B also reads its weights as 32-bit numbers, about 16 GB, to fold its adapter in exactly. That does not fit beside the models the machine serves, so on its processor we timed Kev-4B in 16-bit with the adapter kept apart.
What this does not prove
- This is one run per model on two sets of requests. On a kind with 20 requests, one mistake moves the figure by five points.
- We wrote and labelled the 328 requests ourselves. On OpenJev's own 144 test decisions, Kev's model card reports the reverse, 0.896 for Kev-4B against 0.813 for OpenJev's Qwen3.5-4B, so the winner depends on the requests.
- The browser file ran through llama.cpp on the laptop's GPU, not inside a browser. Its accuracy carries over, but its speed does not: OpenJev's authors measured 3.3 seconds for one decision in Chrome on an RTX 3090.
- The laptop timings come from two laptops, and the processor timings from one EPYC machine and one GB10, each run once with five requests per length. The GB10's cores were shared with the models it serves.
What this means for CArouter
We chose Kev-4B because it was the most accurate open model on public prompts and the most honest about its doubts. On requests like the ones our customers send, an untrained Qwen3.5-4B reads them better, including the two kinds Kev-4B misses most.
Kev-4B stays behind /v1/decisions for now, on the GPU: a processor is no place for any of these models while a caller waits. Before changing anything, we will measure the prompted Qwen3.5-4B on the GPU that serves Kev, at the speed a customer would see, and run the full selection test that automatic model selection needs. Every measurement behind this article is kept, and we will share it with anyone who asks.