=== served model ids, retries, failures === typesafe/jev-1.13 | served: {'typesafe/jev-1.13-20260917': 791} | retries>0: 0 | failed: [] anthropic/claude-haiku-4.5 | served: {'anthropic/claude-haiku-4.5': 791} | retries>0: 0 | failed: [] anthropic/claude-haiku-5.5 | served: {'anthropic/claude-haiku-5.5': 791} | retries>0: 0 | failed: [] anthropic/claude-sonnet-5.5 | served: {'anthropic/claude-sonnet-5.5': 789} | retries>0: 0 | failed: [('inject', "unparseable: 'NoneType' object has no attribute 'strip'"), ('inject', "unparseable: 'NoneType' object has no attribute 'strip'")] === mean input tokens per call (billing-relevant) === typesafe/jev-1.13 intent8 360 | intent77 952 | inject 324 anthropic/claude-haiku-4.5 intent8 303 | intent77 1211 | inject 267 anthropic/claude-haiku-5.5 intent8 408 | intent77 1686 | inject 359 anthropic/claude-sonnet-5.5 intent8 408 | intent77 1686 | inject 358 === paired comparison vs Jev (correct/incorrect on identical items), exact McNemar p === intent8 vs anthropic/claude-haiku-4.5 n=160 Jev-only-right 17 LLM-only-right 8 p=0.108 intent8 vs anthropic/claude-haiku-5.5 n=160 Jev-only-right 10 LLM-only-right 9 p=1.000 intent8 vs anthropic/claude-sonnet-5.5 n=160 Jev-only-right 2 LLM-only-right 11 p=0.022 intent77 vs anthropic/claude-haiku-4.5 n=231 Jev-only-right 18 LLM-only-right 8 p=0.076 intent77 vs anthropic/claude-haiku-5.5 n=231 Jev-only-right 20 LLM-only-right 8 p=0.036 intent77 vs anthropic/claude-sonnet-5.5 n=231 Jev-only-right 11 LLM-only-right 11 p=1.000 inject vs anthropic/claude-haiku-4.5 n=400 Jev-only-right 8 LLM-only-right 16 p=0.152 inject vs anthropic/claude-haiku-5.5 n=400 Jev-only-right 22 LLM-only-right 13 p=0.175 inject vs anthropic/claude-sonnet-5.5 n=398 Jev-only-right 10 LLM-only-right 25 p=0.017 === Jev vs Sonnet 5.5 AGREEMENT (TypeSafe's own metric) vs accuracy against ground truth === intent8 n=160 Jev-Sonnet agreement 87.5% | Jev acc vs gold 84.4% | Sonnet acc vs gold 90.0% intent77 n=231 Jev-Sonnet agreement 89.2% | Jev acc vs gold 80.5% | Sonnet acc vs gold 80.5% inject n=398 Jev-Sonnet agreement 91.2% | Jev acc vs gold 87.4% | Sonnet acc vs gold 91.2% === Jev confidence: confident errors === intent8 conf>=0.9: 117 items, 6 wrong (5.1%) | conf<0.5: 6 items, acc 33.3% wrong-with-conf>=0.9 examples (gold -> pred): [('card arrival', 'card delivery estimate', 0.92), ('direct debit payment not recognised', 'card payment not recognised', 0.95), ('direct debit payment not recognised', 'card payment not recognised', 0.97), ('direct debit payment not recognised', 'card payment not recognised', 1), ('direct debit payment not recognised', 'card payment not recognised', 0.99), ('direct debit payment not recognised', 'card payment not recognised', 0.94)] intent77 conf>=0.9: 161 items, 15 wrong (9.3%) | conf<0.5: 17 items, acc 76.5% wrong-with-conf>=0.9 examples (gold -> pred): [('activate my card', 'lost or stolen card', 1), ('beneficiary not allowed', 'failed transfer', 0.99), ('card linking', 'activate my card', 0.95), ('card payment fee charged', 'extra charge on statement', 0.93), ('declined transfer', 'declined card payment', 0.97), ('direct debit payment not recognised', 'card payment not recognised', 0.93)] === intent8: most common Jev confusions === 8 x direct debit payment not recognised -> card payment not recognised 6 x order physical card -> card arrival 5 x card delivery estimate -> card arrival 2 x card arrival -> card delivery estimate 2 x getting spare card -> order physical card 1 x card payment not recognised -> declined card payment same confusions for Sonnet 5.5: 7 x direct debit payment not recognised -> card payment not recognised 6 x order physical card -> card delivery estimate 2 x card delivery estimate -> card arrival 1 x card delivery estimate -> order physical card === injection: threshold sweep on the probability each system returns (in-sample) === typesafe/jev-1.13 t>=0.50: recall 0.69 prec 1.00 (fp 0) | t>=0.30: recall 0.76 prec 0.99 (fp 1) | t>=0.20: recall 0.86 prec 0.99 (fp 2) | t>=0.10: recall 0.96 prec 0.92 (fp 15) | t>=0.05: recall 0.99 prec 0.74 (fp 58) anthropic/claude-haiku-4.5 t>=0.50: recall 0.74 prec 1.00 (fp 0) | t>=0.30: recall 0.74 prec 1.00 (fp 0) | t>=0.20: recall 0.74 prec 1.00 (fp 0) | t>=0.10: recall 0.79 prec 0.99 (fp 1) | t>=0.05: recall 0.98 prec 0.89 (fp 21) anthropic/claude-haiku-5.5 t>=0.50: recall 0.64 prec 1.00 (fp 0) | t>=0.30: recall 0.64 prec 1.00 (fp 0) | t>=0.20: recall 0.64 prec 1.00 (fp 0) | t>=0.10: recall 0.82 prec 0.99 (fp 1) | t>=0.05: recall 0.89 prec 0.99 (fp 2) anthropic/claude-sonnet-5.5 t>=0.50: recall 0.79 prec 1.00 (fp 0) | t>=0.30: recall 0.80 prec 1.00 (fp 0) | t>=0.20: recall 0.82 prec 1.00 (fp 0) | t>=0.10: recall 0.92 prec 1.00 (fp 0) | t>=0.05: recall 0.96 prec 0.99 (fp 1) === latency detail (seconds) === typesafe/jev-1.13 all tasks: p50 0.36 p90 0.45 p95 0.48 p99 0.71 max 1.21 anthropic/claude-haiku-4.5 all tasks: p50 1.53 p90 1.86 p95 2.06 p99 2.76 max 3.40 anthropic/claude-haiku-5.5 all tasks: p50 1.36 p90 1.67 p95 1.77 p99 2.62 max 3.80 anthropic/claude-sonnet-5.5 all tasks: p50 1.63 p90 2.04 p95 2.42 p99 2.93 max 3.33 === total spend by system ($) === typesafe/jev-1.13 0.0171 anthropic/claude-haiku-4.5 0.4894 anthropic/claude-haiku-5.5 0.0672 anthropic/claude-sonnet-5.5 1.3221 TOTAL 1.8958 over 3164 calls === confidence-gated cascade: Jev answers if confidence >= threshold, else the fallback model answers === intent8 fallback anthropic/claude-sonnet-5.5 conf>=0.80 | routed 19.4% | cascade acc 89.4% (Jev 84.4%, fallback 90.0%) | $0.209/1k = 21.5% of fallback-only ($0.971/1k) | mean latency 0.76s vs 1.71s intent8 fallback anthropic/claude-sonnet-5.5 conf>=0.90 | routed 26.9% | cascade acc 89.4% (Jev 84.4%, fallback 90.0%) | $0.280/1k = 28.8% of fallback-only ($0.971/1k) | mean latency 0.90s vs 1.71s intent8 fallback anthropic/claude-sonnet-5.5 conf>=0.95 | routed 40.0% | cascade acc 90.0% (Jev 84.4%, fallback 90.0%) | $0.405/1k = 41.7% of fallback-only ($0.971/1k) | mean latency 1.13s vs 1.71s intent8 fallback anthropic/claude-haiku-5.5 conf>=0.80 | routed 19.4% | cascade acc 86.9% (Jev 84.4%, fallback 83.8%) | $0.025/1k = 50.6% of fallback-only ($0.050/1k) | mean latency 0.70s vs 1.42s intent8 fallback anthropic/claude-haiku-5.5 conf>=0.90 | routed 26.9% | cascade acc 86.2% (Jev 84.4%, fallback 83.8%) | $0.029/1k = 57.7% of fallback-only ($0.050/1k) | mean latency 0.80s vs 1.42s intent8 fallback anthropic/claude-haiku-5.5 conf>=0.95 | routed 40.0% | cascade acc 84.4% (Jev 84.4%, fallback 83.8%) | $0.035/1k = 70.3% of fallback-only ($0.050/1k) | mean latency 0.99s vs 1.42s intent8 fallback anthropic/claude-haiku-4.5 conf>=0.80 | routed 19.4% | cascade acc 86.9% (Jev 84.4%, fallback 78.8%) | $0.084/1k = 23.4% of fallback-only ($0.360/1k) | mean latency 0.72s vs 1.58s intent8 fallback anthropic/claude-haiku-4.5 conf>=0.90 | routed 26.9% | cascade acc 85.6% (Jev 84.4%, fallback 78.8%) | $0.111/1k = 30.8% of fallback-only ($0.360/1k) | mean latency 0.85s vs 1.58s intent8 fallback anthropic/claude-haiku-4.5 conf>=0.95 | routed 40.0% | cascade acc 83.1% (Jev 84.4%, fallback 78.8%) | $0.158/1k = 43.8% of fallback-only ($0.360/1k) | mean latency 1.06s vs 1.58s intent77 fallback anthropic/claude-sonnet-5.5 conf>=0.80 | routed 22.5% | cascade acc 81.0% (Jev 80.5%, fallback 80.5%) | $0.833/1k = 23.7% of fallback-only ($3.523/1k) | mean latency 0.71s vs 1.51s intent77 fallback anthropic/claude-sonnet-5.5 conf>=0.90 | routed 30.3% | cascade acc 81.4% (Jev 80.5%, fallback 80.5%) | $1.108/1k = 31.4% of fallback-only ($3.523/1k) | mean latency 0.83s vs 1.51s intent77 fallback anthropic/claude-sonnet-5.5 conf>=0.95 | routed 40.3% | cascade acc 81.0% (Jev 80.5%, fallback 80.5%) | $1.459/1k = 41.4% of fallback-only ($3.523/1k) | mean latency 0.97s vs 1.51s intent77 fallback anthropic/claude-haiku-5.5 conf>=0.80 | routed 22.5% | cascade acc 78.8% (Jev 80.5%, fallback 75.3%) | $0.080/1k = 45.2% of fallback-only ($0.178/1k) | mean latency 0.68s vs 1.38s intent77 fallback anthropic/claude-haiku-5.5 conf>=0.90 | routed 30.3% | cascade acc 78.4% (Jev 80.5%, fallback 75.3%) | $0.095/1k = 53.3% of fallback-only ($0.178/1k) | mean latency 0.81s vs 1.38s intent77 fallback anthropic/claude-haiku-5.5 conf>=0.95 | routed 40.3% | cascade acc 76.6% (Jev 80.5%, fallback 75.3%) | $0.112/1k = 63.3% of fallback-only ($0.178/1k) | mean latency 0.94s vs 1.38s intent77 fallback anthropic/claude-haiku-4.5 conf>=0.80 | routed 22.5% | cascade acc 80.1% (Jev 80.5%, fallback 76.2%) | $0.325/1k = 25.6% of fallback-only ($1.268/1k) | mean latency 0.71s vs 1.52s intent77 fallback anthropic/claude-haiku-4.5 conf>=0.90 | routed 30.3% | cascade acc 79.7% (Jev 80.5%, fallback 76.2%) | $0.424/1k = 33.4% of fallback-only ($1.268/1k) | mean latency 0.83s vs 1.52s intent77 fallback anthropic/claude-haiku-4.5 conf>=0.95 | routed 40.3% | cascade acc 79.2% (Jev 80.5%, fallback 76.2%) | $0.550/1k = 43.4% of fallback-only ($1.268/1k) | mean latency 0.98s vs 1.52s === Haiku 5.5 vs Haiku 4.5, paired on identical items, exact McNemar p === intent8 n=160 Haiku-5.5-only-right 13 Haiku-4.5-only-right 5 p=0.096 intent77 n=231 Haiku-5.5-only-right 14 Haiku-4.5-only-right 16 p=0.856 inject n=400 Haiku-5.5-only-right 10 Haiku-4.5-only-right 27 p=0.008 === billed tokens per call: Haiku 5.5 vs Haiku 4.5 (same prompts) === intent8 input 408 vs 303 (x1.35) | output 18.5 vs 11.4 intent77 input 1686 vs 1211 (x1.39) | output 18.1 vs 11.4 inject input 359 vs 267 (x1.34) | output 19.0 vs 16.0 === run-to-run check: same system, same items, September 2026 run vs this run === typesafe/jev-1.13 intent8 n=160 same answer 99.4% | acc Sep 83.8% -> Oct 84.4% | p50 latency Sep 0.33s -> Oct 0.40s typesafe/jev-1.13 intent77 n=231 same answer 96.5% | acc Sep 78.8% -> Oct 80.5% | p50 latency Sep 0.32s -> Oct 0.37s typesafe/jev-1.13 inject n=400 same answer 99.5% | acc Sep 87.0% -> Oct 87.0% | p50 latency Sep 0.33s -> Oct 0.35s anthropic/claude-haiku-4.5 intent8 n=160 same answer 100.0% | acc Sep 78.8% -> Oct 78.8% | p50 latency Sep 0.99s -> Oct 1.53s anthropic/claude-haiku-4.5 intent77 n=231 same answer 100.0% | acc Sep 76.2% -> Oct 76.2% | p50 latency Sep 1.01s -> Oct 1.48s anthropic/claude-haiku-4.5 inject n=400 same answer 100.0% | acc Sep 89.0% -> Oct 89.0% | p50 latency Sep 1.03s -> Oct 1.56s