Jev vs Claude Haiku 5.5: We Re-Ran Our 791-Decision Benchmark

Claude Haiku 5.5 lists at a tenth of Haiku 4.5's price per token, so we re-ran our Jev vs LLM benchmark to see whether the new Haiku closes the gap with TypeSafe's Jev. Same harness, same 791 labeled decisions, same prompts, one billing meter (OpenRouter), run on October 24, 2026. Short version: Haiku 5.5 cost 7.1 to 7.7 times less per decision than Haiku 4.5, not 10 times, because it bills 34 to 39 percent more input tokens for the same text. Jev was still 3.3 to 4.4 times cheaper than Haiku 5.5, about 3.8 times faster at the median, and as accurate or more accurate on all three tasks.

We added Claude Sonnet 5.5 as the strong baseline. It was the most accurate system on two of the three tasks and cost 64 to 88 times more than Jev per decision. Sending only Jev's low-confidence answers to Sonnet 5.5 came within 0.6 points of Sonnet alone on one task and edged past it on the other, at 22 to 24 percent of the cost.

TL;DR

• Haiku 5.5 was 7.2x, 7.1x and 7.7x cheaper per decision than Haiku 4.5 on our three tasks. The list price is 10x lower, but the newer tokenizer counted 34 to 39 percent more input tokens for identical prompts, and Haiku 5.5 billed a few more output tokens per answer.
• Jev was still 3.3x (8-way), 4.4x (77-way) and 3.3x (injection) cheaper than Haiku 5.5. In September Haiku 4.5 was 24 to 31 times more expensive than Jev. The gap shrank a lot, but it did not close.
• Jev's median latency was 0.36 s, Haiku 5.5's 1.36 s, Haiku 4.5's 1.53 s and Sonnet 5.5's 1.63 s, all measured in the same run.
• Haiku 5.5 tied Jev on 8-way routing, trailed it on 77-way routing (Jev alone right on 20 items, Haiku 5.5 alone on 8, p = 0.036), and was worse than Haiku 4.5 on injection detection (p = 0.008).
• Sonnet 5.5 was the accuracy leader on 8-way routing (90.0%) and injection detection (90.8%) and tied Jev on 77-way routing (80.5% each).
• The whole run was 3,164 calls for $1.90.

What we tested

We reused the September harness unchanged apart from the model list. It samples three tasks from two public Hugging Face datasets with a fixed seed (42), so every system saw exactly the same 791 items as in our first benchmark.

Systems: Jev 1.13 (served as typesafe/jev-1.13-20260917 through OpenRouter's decisions endpoint), Claude Haiku 4.5, Claude Haiku 5.5 and Claude Sonnet 5.5, all through OpenRouter. We re-ran Jev and Haiku 4.5 instead of reusing September's numbers so that latency and cost come from the same hour and the same network path.

The LLMs got a strict JSON schema that allows only the listed labels, a 300-token output cap and OpenRouter's reasoning: {"effort": "minimal"}. One change from September: Anthropic's Haiku 5.5 model page says to omit temperature, and OpenRouter's listing for both 5.5 models doesn't accept it, so only Haiku 4.5 ran at temperature 0. Haiku 5.5 and Sonnet 5.5 ran at their default sampling. A failed call counts as wrong. Intervals are 95 percent Wilson intervals, and the paired comparisons use an exact McNemar test on identical items. The run took six minutes (00:55 to 01:01 UTC) with four parallel requests per system and all four systems running at once.

Why Haiku 5.5 is 7x cheaper, not 10x

Anthropic lists Haiku 5.5 at $0.10 input and $0.50 output per million tokens for prompts up to 100,000 tokens. Haiku 4.5 is $1 and $5. Anthropic's model page also warns that Haiku 5.5 uses the newer tokenizer, so "the same text counts as approximately 30 percent more tokens than on Claude Haiku 4.5." Here is what OpenRouter billed for identical prompts:

The input inflation we measured (34 to 39 percent) was a little above Anthropic's "approximately 30 percent," probably because our prompts are mostly label lists and short messages. Haiku 5.5 also billed 3 to 7 more output tokens per answer than Haiku 4.5 for the same one-field JSON. We can't tell from the billing data whether that is a different JSON wrapper or a little thinking. Either way, budget for about a 7x saving on short classification calls, not 10x. Sonnet 5.5 uses the same tokenizer and billed the same input counts as Haiku 5.5 (408, 1,686 and 358).

Cost and speed against Jev

Haiku 5.5 was the fastest Claude model in the run, as Anthropic's "Fastest" label suggests, but only by about 0.17 seconds at the median over Haiku 4.5. Jev answered in about a quarter of Haiku 5.5's time, and its slowest call (1.21 s) was faster than Haiku 5.5's median.

The 77-way task shows where the per-token gap widens. Jev bills the question and options as input too (952 tokens here), but at $0.042 per million with free output it still came to $0.040 per 1,000 decisions, against $0.178 for Haiku 5.5's 1,686 tokens. Long label lists favor Jev. For what Jev's pricing means at volume, see our Jev cost-per-decision breakdown.

Accuracy

The intervals overlap a lot, so the paired tests on identical items carry more weight than the headline percentages:

What that says in plain terms:

• Haiku 5.5 did not beat Jev anywhere. It tied on 8-way routing, lost on 77-way routing with a significant gap, and was behind on injection detection without a significant gap.
• Haiku 5.5 is not uniformly better than Haiku 4.5. It looked better on the 8-way task (not significant) and was clearly worse on injection detection: Haiku 4.5 alone caught 27 items Haiku 5.5 missed, and the reverse happened 10 times. If you run a safety filter on Haiku 4.5, test before you switch.
• Sonnet 5.5 is the accuracy pick. It beat Jev on 8-way routing and injection detection and tied it on 77-way routing. Sonnet 5.5 returned no message content on 2 injection items, which count as wrong. Both were labeled injections.

Prompt injection: Haiku 5.5's probabilities are coarse

No system raised a false alarm at its default yes or no answer, and every one missed a lot of labeled injections. As we noted in September, many items this dataset labels as injections are short persona prompts such as "Be a DJ", so treat absolute recall as a property of the data.

Lowering the cutoff on the probability each system returned:

Sonnet 5.5's self-reported probability was the best ranking signal in the run (AUROC 0.996) and reached 0.92 recall with no false alarms at a 0.10 cutoff. Haiku 5.5's probability did not move between 0.50 and 0.20, so it gives you fewer useful settings than Jev's. These cutoffs were chosen after seeing the results; pick yours on a separate validation set.

The Jev-first cascade, with Sonnet 5.5 as the fallback

In September, letting Jev answer when it was confident and sending the rest to a frontier model matched the frontier model's accuracy at about a quarter of its cost. We repeated that with Sonnet 5.5 and Haiku 5.5 as the fallback, using the results above:

With Sonnet 5.5 behind it, the cascade landed within 0.6 points of Sonnet alone on 8-way routing and 0.5 points above it on 77-way routing, for 21.5 to 23.7 percent of Sonnet's cost and less than half its mean latency. With Haiku 5.5 behind it, the cascade beat both models on 8-way routing but fell below Jev alone on 77-way routing, because Haiku 5.5 was the weaker of the two there. A fallback only helps if it is better than Jev on the items Jev is unsure about. The gates were not tuned and were evaluated on the same items, so treat these as estimates.

Jev's confidence score behaved as in September. On 8-way routing, accuracy on answered items rose from 84.4 percent with no gate to 93.0 percent at 0.80 (80.6 percent of items answered) and 95.8 percent at 0.95 (60.0 percent answered). It was still confidently wrong when labels overlap: 5 of its 6 errors at 0.90 or above on the 8-way task were "direct debit payment not recognised" read as "card payment not recognised", and Sonnet 5.5 made that same mistake 7 times.

How stable are these numbers?

Because we re-ran Jev and Haiku 4.5 on the same items five weeks after the first run, we can check repeatability:

Haiku 4.5 at temperature 0 gave the identical answer on all 791 items. Jev changed a handful of answers, enough to move its 77-way accuracy by 1.7 points, so expect about that much noise in any single Jev run. Haiku 4.5's median latency rose by about half a second between the runs with no change on our side. That is why we compare latency only within one run.

What this test cannot tell you

• One run per system on public data that may appear in the LLMs' training sets. We did not repeat Haiku 5.5 or Sonnet 5.5, so we don't know their run-to-run variance at default sampling.
• We asked for minimal reasoning through OpenRouter and did not verify which Anthropic effort setting it mapped to. Haiku 5.5's default effort on the Claude API is medium. Higher effort may raise its accuracy, along with its cost and latency.
• Options were labels only, with no descriptions.
• Prices are OpenRouter's billed cost, with no prompt caching and no Batch API. Haiku 5.5's 512-token minimum cacheable prompt could cut the 77-way cost if your label list is a fixed prefix; we did not test caching.
• Latency came from one laptop through OpenRouter. Calling Anthropic's API directly may be faster or slower.
• Jev runs on OpenRouter's alpha decisions endpoint, so its behavior and pricing can change.
• This is not a client workload, and AY Automate has not used Jev on client projects.

Reproduce it

The harness is the September script with the model list changed. It downloads the datasets, samples them with seed 42, calls all four systems and prints the tables. The full run cost $1.90.

• jev_benchmark.py, the harness (fetch, run, report)
• analyze.py, paired tests, token counts, thresholds, cascades and the September comparison
• results.jsonl, all 3,164 calls with prediction, latency, tokens and cost
• report.txt, analysis.txt and summary.json, the printed outputs

Which one should you use?

• For high-volume, bounded decisions where every millisecond and cent counts, use Jev, ideally with a confidence gate. It was the cheapest and fastest system and never less accurate than Haiku 5.5.
• If you already run classification on Claude Haiku 4.5, Haiku 5.5 will cut that bill by about 7x. Re-test any injection or safety filter first, because it was clearly worse than Haiku 4.5 on that task here.
• If you need the best accuracy per item, use Sonnet 5.5, or Jev in front of Sonnet 5.5 to keep most of the accuracy at under a quarter of the cost. Read our Sonnet 5.5 guide before you switch, because forced tool use and disabled thinking now return errors. For the full Claude lineup and prices, see Opus vs Sonnet vs Haiku and the Claude Haiku 5.5 model page.

If you want this kind of routing layer built into a support desk, a lead-scoring flow or an agent, that is the work our AI agent development team does: we benchmark the candidates on your own labeled data first, then ship the cascade that wins.

FAQ

How much cheaper is Claude Haiku 5.5 than Haiku 4.5 in practice?

In our test, 7.1 to 7.7 times cheaper per decision. The list price is 10 times lower ($0.10/$0.50 against $1/$5 per million tokens), but Haiku 5.5's tokenizer counted 34 to 39 percent more input tokens for the same prompts, and it billed a few more output tokens per answer.

Is Claude Haiku 5.5 cheaper than Jev?

No. Jev was 3.3 to 4.4 times cheaper per decision than Haiku 5.5 on our three tasks: $0.0151 against $0.0500 per 1,000 decisions on 8-way intent routing.

Is Claude Haiku 5.5 faster than Jev?

No. Haiku 5.5's median latency was 1.36 seconds against 0.36 seconds for Jev in the same run, and Jev's slowest call (1.21 s) was faster than Haiku 5.5's median.

Is Claude Haiku 5.5 more accurate than Haiku 4.5?

Not across the board. It scored higher on 8-way intent routing (83.8% vs 78.8%, not significant), about the same on 77-way routing, and significantly lower on prompt-injection detection (84.8% vs 89.0%, p = 0.008).

How does Claude Sonnet 5.5 compare with Jev?

Sonnet 5.5 was more accurate on 8-way routing (90.0% vs 84.4%) and injection detection (90.8% vs 87.0%), tied on 77-way routing, and cost 64 to 88 times more per decision. Jev in front of Sonnet 5.5 at a 0.80 confidence gate came within 0.6 points of Sonnet's accuracy for 22 to 24 percent of its cost.

What is the cheapest LLM for classification?

Of the Claude models we tested, Haiku 5.5, at about $0.05 per 1,000 short decisions. Jev, a decision model rather than a chat LLM, was cheaper still at about $0.015. In our September run, GPT-5.4 nano ($0.070) and Gemini 3.5 Flash-Lite ($0.087) sat between the two on the same task, but they were measured on a different day.

Sources: Claude Haiku 5.5 model page (price, release date October 7, 2026, tokenizer note, read October 24, 2026); Claude pricing; OpenRouter model listings for anthropic/claude-haiku-5.5 and anthropic/claude-sonnet-5.5 (read October 24, 2026); Banking77 and deepset/prompt-injections on Hugging Face; our own run, raw files above.