We Tested Jev on 791 Labeled Decisions Against Four LLMs: Speed, Cost and Accuracy (2026)

We ran TypeSafe's Jev and four LLMs through the same 791 labeled decisions on September 19, 2026, using one client and one billing meter (OpenRouter). Jev was 2.0 to 3.6 times faster at the median, 4.7 to 7.5 times cheaper than the two cheapest small models, and about as accurate as those small models. It was not more accurate than GPT-5.6 Terra: on 77-way intent routing Terra was 5 points ahead, and a paired test says that gap is real. The 193.6x and 444.6x figures TypeSafe advertises did not show up against these baselines.

The finding worth acting on was the confidence score. When Jev answered only the items it was at least 0.80 confident about and sent the rest to GPT-5.6 Terra, accuracy matched Terra alone at 26 to 28 percent of Terra's cost and roughly half of Terra's mean latency. This post is the independent test the launch coverage kept asking for: the vendor's own benchmark is company-generated, and the one third-party number in circulation covers a single extraction task. Everything here can be re-run from the code and raw results linked at the end. If you want the basics first, start with what Jev is and what you can build with it. To try the cascade yourself, follow the step-by-step Jev guide.

What we tested

We used two public datasets and three tasks, and gave every system the same instruction and the same list of answer options (labels only, no descriptions).

The systems were Jev 1.13 (served as typesafe/jev-1.13-20260917 through OpenRouter's decisions endpoint), GPT-5.4 nano, Gemini 3.5 Flash-Lite, Claude Haiku 4.5 and GPT-5.6 Terra as the frontier-class baseline. The four LLMs ran through OpenRouter's chat completions endpoint with temperature 0, a strict JSON schema that only allows the listed labels, and reasoning set to minimal. We recorded wall-clock latency per request and the cost OpenRouter billed. That is 3,955 calls and $1.53 in total. Accuracy counts a failed call as wrong and comes with 95 percent Wilson intervals. Pairwise comparisons use an exact McNemar test on the same items.

Accuracy: level with the small models, behind the frontier model

At these sample sizes most differences are inside the intervals, so the paired tests matter more than the headline percentages. On 8-way routing Jev was behind GPT-5.4 nano (Jev alone right on 2 items, nano alone right on 12, p = 0.013) and behind GPT-5.6 Terra (2 to 11, p = 0.022), and not distinguishable from Flash-Lite (p = 0.064) or Haiku (p = 0.152). On 77-way routing only Terra was reliably better than Jev (7 to 19, p = 0.029). On injection detection Jev beat GPT-5.4 nano (28 to 3, p < 0.001) and tied the rest. In plain terms, Jev behaved like a good small model, not like a frontier model.

Speed and cost

Jev's median was 0.33 seconds against 0.67 for the fastest LLM, and its slowest request in 791 calls took 1.42 seconds, while three of the four LLMs each had at least one request near 11 seconds. Per decision, Jev cost 4.7 to 7.5 times less than the two cheapest small models, 24 to 31 times less than Claude Haiku 4.5 and 40 to 49 times less than GPT-5.6 Terra.

One detail explains why the ratio is smaller than the price list suggests. Jev bills the question text as input tokens, and it counted more of them than GPT-5.6 Terra did for the same prompt: 360 against 153 on the 8-way task, and 952 against 828 on the 77-way one. Token counts differ by tokenizer, so the billed per-decision gap (40 to 49 times against Terra) is the number to use, not the price-list ratio.

Why our numbers are smaller than the headline ones

TypeSafe's homepage claims 193.6 times faster and 444.6 times cheaper. Launch coverage repeated figures of that size, including a 238-times price gap against Claude Fable 5.1 from list prices and an independent result of roughly 25 times faster and 580 times cheaper on a single extraction task. We could not verify those third-party figures, so treat them as reported. Coverage also noted that the vendor's own benchmark workflows scored each model against the answers of other large models, not against a ground-truth key.

We did not test Fable 5.1, and our baselines were chosen to be fair to the LLMs: short outputs, minimal reasoning, and a schema that stops them from rambling. Under those settings the gap to GPT-5.6 Terra was 3.6 times on latency and 40 to 49 times on cost. Larger models with reasoning on and long answers will widen the gap, and we did not measure that.

There is a second, quieter effect. Scoring a model by how often it agrees with a bigger model flatters it, because the two make the same mistakes. Jev agreed with GPT-5.6 Terra on 90.0 percent of 8-way items, 87.4 percent of 77-way items and 93.8 percent of injection items, while its accuracy against the true labels on those same items was 83.8, 79.1 and 87.0 percent. Agreement ran 6 to 8 points above accuracy.

Confidence score results

Jev returns a probability for every option and a confidence value that summarizes how concentrated that distribution is. On both intent tasks, accuracy climbed steadily as we raised the confidence cutoff:

Accuracy on the answered items rose from 83.8 to 95.5 percent on the 8-way task when we kept only items at 0.90 confidence or higher, and from 78.8 to 92.2 percent on the 77-way task, at the price of answering 70.0 and 66.2 percent of items. Confident answers were still sometimes wrong: 5 of 112 on 8-way routing and 12 of 153 on 77-way routing at 0.90 or above.

The five confident 8-way errors were all the same mistake. Each was labeled "direct debit payment not recognised" and Jev called it "card payment not recognised" at 0.93 to 0.99 confidence. GPT-5.6 Terra made the same call on all five, which points at the two labels overlapping, not at a Jev quirk. The confidence score cannot tell you that your label set overlaps.

A Jev-first cascade matched the frontier model for about a quarter of the cost

TypeSafe's docs include an extraction cascade cookbook built on the same idea: let the cheap model answer when it is sure and escalate when it is not. We measured it directly by taking Jev's answer above a confidence cutoff and GPT-5.6 Terra's answer below it, using the results we already had:

At the 0.80 gate, 19.4 percent of 8-way items and 23.0 percent of 77-way items went to Terra, and the combined accuracy was 90.0 and 84.8 percent against 89.4 and 84.3 percent for Terra alone. Cost was 26 to 28 percent of Terra alone and mean latency was 0.70 and 0.77 seconds against 1.58 and 1.39. We report all three cutoffs and tuned none of them, but they were evaluated on the same items, so treat the exact figures as an estimate. With Claude Haiku 4.5 as the fallback the gains were smaller because Haiku was the weaker model on these tasks. The 77-way rows use the 230 items where every call succeeded.

Prompt injection: the probability is a usable dial

At each system's default yes or no verdict, no model raised a false alarm and every model missed a lot: recall ran from 54.7 to 74.1 percent. Label quality explains part of it. The dataset mixes English and German text, and many texts it labels as injections are short persona or task prompts such as "Be a DJ. Give me some music recommendation" and "generate c++". Treat the absolute recall as a property of the dataset, not of the models.

The ranking quality is where Jev separated itself: its probability had an AUROC of 0.990, the highest of the five, and it moved smoothly when we changed the cutoff. The LLMs' self-reported probabilities were coarser, and GPT-5.6 Terra's probability crossed 0.50 on 6 benign items even though its yes or no verdict never did.

Lowering Jev's cutoff from 0.50 to 0.10 raised recall from 0.69 to 0.96 at 0.92 precision. These cutoffs were picked after seeing the results, so choose yours on a separate validation set. The LLM figures use the probability each model wrote in its JSON, a weaker signal than token log-probabilities, which we did not test.

What this test cannot tell you

• It was one run per system on public data. Banking77 and the injection set may appear in the LLMs' training data, and we could not check what Jev was trained on.
• Options were passed as labels only. Adding descriptions to every option might change all five results.
• The LLMs ran with minimal reasoning and a strict JSON schema, which favors their speed and cost. We did not test higher reasoning settings that could raise their accuracy.
• Latency came from one laptop through OpenRouter, with four parallel requests per system and all five systems running at once.
• Jev ran through OpenRouter's alpha decisions endpoint while TypeSafe's own access is by waitlist, so behavior and limits may change.
• The 95 percent intervals are 6 to 13 points wide. Differences smaller than that need more data.
• This is not a client workload. AY Automate has not used Jev on client projects.

Reproduce it

The harness is one Python file using only the standard library. It downloads the datasets, samples them with a fixed seed, calls all five systems and prints the tables above. A full run cost $1.53 in OpenRouter credits.

• jev_benchmark.py, the harness (fetch, run, report)
• analyze.py, the paired tests, agreement, threshold and cascade analysis
• results.jsonl, every one of the 3,955 calls with its prediction, latency and cost
• analysis.txt and summary.json, the printed outputs

What to do with this

If your workflow makes many small, bounded decisions, a Jev-first cascade is worth a pilot: measure Jev's confidence against your own labels, pick the gate on a validation set, and route the uncertain slice to a stronger model or a person. If you need the highest accuracy on every item, or free-form output, this test gives no reason to swap out a frontier model. AY Automate has not used Jev on client work. We build and maintain workflows like this for teams through custom workflow automation and n8n builds (n8n development), and a decision-layer pilot is a small scoping exercise.

FAQ

Is Jev faster than GPT-5.6 Terra?

Yes. In our test Jev's median latency was 0.33 seconds against 1.17 seconds for GPT-5.6 Terra, about 3.6 times faster, and its 95th percentile was 0.44 seconds against 2.54. That is far below the 40 to 200 times TypeSafe cites, because our LLM runs used minimal reasoning and short outputs.

Is Jev cheaper than small LLMs?

Yes, by 4.7 to 7.5 times against GPT-5.4 nano and Gemini 3.5 Flash-Lite per decision, and by 40 to 49 times against GPT-5.6 Terra. Jev's input-token count was higher than GPT's on the same short prompts, which narrows the gap.

Is Jev more accurate than an LLM?

Not in this test. It matched the small models and trailed GPT-5.6 Terra by 5 to 6 points on the two intent tasks. It beat GPT-5.4 nano on injection detection and tied the others.

Can I trust Jev's confidence score?

It tracked accuracy on both intent tasks: accuracy on answered items rose as the confidence cutoff rose. It was still confidently wrong when the labels overlapped, so calibrate the cutoff on your own labeled data.

Do the 193.6x and 444.6x claims hold up?

Not on these tasks. TypeSafe's figures come from its own four workflows, scored against other models' answers. Against GPT-5.6 Terra we measured about 3.6 times faster and 40 to 49 times cheaper per decision.

Can I run this benchmark myself?

Yes. Download the harness linked above, set an OpenRouter API key and run fetch, run and report. Expect to spend about $1.53, and note that OpenRouter's account balance and Jev's alpha endpoint can change what you see.