=== served model ids, retries, failures === typesafe/jev-1.13 | served: {'typesafe/jev-1.13-20260917': 791} | retries>0: 0 | failed: [] openai/gpt-5.4-nano | served: {'openai/gpt-5.4-nano': 791} | retries>0: 2 | failed: [] google/gemini-3.5-flash-lite | served: {'google/gemini-3.5-flash-lite': 791} | retries>0: 2 | failed: [] anthropic/claude-haiku-4.5 | served: {'anthropic/claude-haiku-4.5': 791} | retries>0: 3 | failed: [] openai/gpt-5.6-terra | served: {'openai/gpt-5.6-terra': 790} | retries>0: 1 | failed: [('intent77', "unparseable: 'NoneType' object has no attribute 'strip'")] === mean input tokens per call (billing-relevant) === typesafe/jev-1.13 intent8 360 | intent77 952 | inject 324 openai/gpt-5.4-nano intent8 153 | intent77 828 | inject 127 google/gemini-3.5-flash-lite intent8 186 | intent77 929 | inject 150 anthropic/claude-haiku-4.5 intent8 303 | intent77 1211 | inject 267 openai/gpt-5.6-terra intent8 153 | intent77 828 | inject 127 === paired comparison vs Jev (correct/incorrect on identical items), exact McNemar p === intent8 vs openai/gpt-5.4-nano n=160 Jev-only-right 2 LLM-only-right 12 p=0.013 intent8 vs google/gemini-3.5-flash-lite n=160 Jev-only-right 17 LLM-only-right 7 p=0.064 intent8 vs anthropic/claude-haiku-4.5 n=160 Jev-only-right 16 LLM-only-right 8 p=0.152 intent8 vs openai/gpt-5.6-terra n=160 Jev-only-right 2 LLM-only-right 11 p=0.022 intent77 vs openai/gpt-5.4-nano n=231 Jev-only-right 14 LLM-only-right 13 p=1.000 intent77 vs google/gemini-3.5-flash-lite n=231 Jev-only-right 12 LLM-only-right 11 p=1.000 intent77 vs anthropic/claude-haiku-4.5 n=231 Jev-only-right 16 LLM-only-right 10 p=0.327 intent77 vs openai/gpt-5.6-terra n=230 Jev-only-right 7 LLM-only-right 19 p=0.029 inject vs openai/gpt-5.4-nano n=400 Jev-only-right 28 LLM-only-right 3 p=0.000 inject vs google/gemini-3.5-flash-lite n=400 Jev-only-right 14 LLM-only-right 13 p=1.000 inject vs anthropic/claude-haiku-4.5 n=400 Jev-only-right 8 LLM-only-right 16 p=0.152 inject vs openai/gpt-5.6-terra n=400 Jev-only-right 13 LLM-only-right 12 p=1.000 === Jev vs Terra AGREEMENT (TypeSafe's own metric) vs accuracy against ground truth === intent8 n=160 Jev-Terra agreement 90.0% | Jev acc vs gold 83.8% | Terra acc vs gold 89.4% intent77 n=230 Jev-Terra agreement 87.4% | Jev acc vs gold 79.1% | Terra acc vs gold 84.3% inject n=400 Jev-Terra agreement 93.8% | Jev acc vs gold 87.0% | Terra acc vs gold 86.8% === Jev confidence: confident errors === intent8 conf>=0.9: 112 items, 5 wrong (4.5%) | conf<0.5: 6 items, acc 16.7% wrong-with-conf>=0.9 examples (gold -> pred): [('direct debit payment not recognised', 'card payment not recognised', 0.99), ('direct debit payment not recognised', 'card payment not recognised', 0.94), ('direct debit payment not recognised', 'card payment not recognised', 0.96), ('direct debit payment not recognised', 'card payment not recognised', 0.99), ('direct debit payment not recognised', 'card payment not recognised', 0.93)] intent77 conf>=0.9: 153 items, 12 wrong (7.8%) | conf<0.5: 18 items, acc 61.1% wrong-with-conf>=0.9 examples (gold -> pred): [('activate my card', 'lost or stolen card', 0.99), ('beneficiary not allowed', 'failed transfer', 0.99), ('card linking', 'activate my card', 0.94), ('card payment fee charged', 'extra charge on statement', 0.93), ('declined transfer', 'declined card payment', 0.96), ('direct debit payment not recognised', 'card payment not recognised', 0.93)] === intent8: most common Jev confusions === 8 x direct debit payment not recognised -> card payment not recognised 6 x order physical card -> card arrival 5 x card delivery estimate -> card arrival 2 x card arrival -> card delivery estimate 2 x getting spare card -> order physical card 1 x order physical card -> declined card payment same confusions for Terra: 6 x direct debit payment not recognised -> card payment not recognised 4 x card delivery estimate -> card arrival 3 x order physical card -> card arrival 3 x order physical card -> card delivery estimate === injection: threshold sweep on the probability each system returns (in-sample) === typesafe/jev-1.13 t>=0.50: recall 0.69 prec 1.00 (fp 0) | t>=0.30: recall 0.77 prec 0.99 (fp 1) | t>=0.20: recall 0.86 prec 0.99 (fp 2) | t>=0.10: recall 0.96 prec 0.92 (fp 15) | t>=0.05: recall 0.99 prec 0.74 (fp 59) anthropic/claude-haiku-4.5 t>=0.50: recall 0.74 prec 1.00 (fp 0) | t>=0.30: recall 0.74 prec 1.00 (fp 0) | t>=0.20: recall 0.74 prec 1.00 (fp 0) | t>=0.10: recall 0.79 prec 0.99 (fp 1) | t>=0.05: recall 0.98 prec 0.89 (fp 21) openai/gpt-5.6-terra t>=0.50: recall 0.78 prec 0.96 (fp 6) | t>=0.30: recall 0.78 prec 0.96 (fp 6) | t>=0.20: recall 0.78 prec 0.96 (fp 6) | t>=0.10: recall 0.78 prec 0.96 (fp 6) | t>=0.05: recall 0.81 prec 0.96 (fp 6) === latency detail (seconds) === typesafe/jev-1.13 all tasks: p50 0.33 p90 0.39 p95 0.44 p99 0.81 max 1.42 openai/gpt-5.4-nano all tasks: p50 1.15 p90 1.87 p95 2.07 p99 2.62 max 11.91 google/gemini-3.5-flash-lite all tasks: p50 0.67 p90 0.82 p95 0.89 p99 1.29 max 10.76 anthropic/claude-haiku-4.5 all tasks: p50 1.02 p90 1.51 p95 1.95 p99 2.20 max 2.73 openai/gpt-5.6-terra all tasks: p50 1.17 p90 2.17 p95 2.54 p99 3.94 max 11.47 === total spend by system ($) === typesafe/jev-1.13 0.0171 openai/gpt-5.4-nano 0.0909 google/gemini-3.5-flash-lite 0.1178 anthropic/claude-haiku-4.5 0.4845 openai/gpt-5.6-terra 0.8168 TOTAL 1.5271 over 3955 calls === confidence-gated cascade: Jev answers if confidence >= threshold, else the fallback model answers === intent8 fallback openai/gpt-5.6-terra conf>=0.80 | routed 19.4% | cascade acc 90.0% (Jev 83.8%, fallback 89.4%) | $0.156/1k = 25.7% of fallback-only ($0.609/1k) | mean latency 0.70s vs 1.58s intent8 fallback openai/gpt-5.6-terra conf>=0.90 | routed 30.0% | cascade acc 90.0% (Jev 83.8%, fallback 89.4%) | $0.232/1k = 38.1% of fallback-only ($0.609/1k) | mean latency 0.89s vs 1.58s intent8 fallback openai/gpt-5.6-terra conf>=0.95 | routed 40.0% | cascade acc 90.0% (Jev 83.8%, fallback 89.4%) | $0.295/1k = 48.5% of fallback-only ($0.609/1k) | mean latency 1.06s vs 1.58s intent8 fallback anthropic/claude-haiku-4.5 conf>=0.80 | routed 19.4% | cascade acc 86.9% (Jev 83.8%, fallback 78.8%) | $0.084/1k = 23.5% of fallback-only ($0.356/1k) | mean latency 0.58s vs 1.09s intent8 fallback anthropic/claude-haiku-4.5 conf>=0.90 | routed 30.0% | cascade acc 85.0% (Jev 83.8%, fallback 78.8%) | $0.121/1k = 34.0% of fallback-only ($0.356/1k) | mean latency 0.70s vs 1.09s intent8 fallback anthropic/claude-haiku-4.5 conf>=0.95 | routed 40.0% | cascade acc 83.8% (Jev 83.8%, fallback 78.8%) | $0.157/1k = 43.9% of fallback-only ($0.356/1k) | mean latency 0.81s vs 1.09s intent77 fallback openai/gpt-5.6-terra conf>=0.80 | routed 23.0% | cascade acc 84.8% (Jev 79.1%, fallback 84.3%) | $0.545/1k = 27.9% of fallback-only ($1.957/1k) | mean latency 0.77s vs 1.39s intent77 fallback openai/gpt-5.6-terra conf>=0.90 | routed 33.5% | cascade acc 85.2% (Jev 79.1%, fallback 84.3%) | $0.769/1k = 39.3% of fallback-only ($1.957/1k) | mean latency 0.95s vs 1.39s intent77 fallback openai/gpt-5.6-terra conf>=0.95 | routed 43.0% | cascade acc 85.2% (Jev 79.1%, fallback 84.3%) | $0.956/1k = 48.9% of fallback-only ($1.957/1k) | mean latency 1.09s vs 1.39s intent77 fallback anthropic/claude-haiku-4.5 conf>=0.80 | routed 22.9% | cascade acc 79.7% (Jev 78.8%, fallback 76.2%) | $0.328/1k = 26.1% of fallback-only ($1.255/1k) | mean latency 0.60s vs 1.12s intent77 fallback anthropic/claude-haiku-4.5 conf>=0.90 | routed 33.8% | cascade acc 79.2% (Jev 78.8%, fallback 76.2%) | $0.464/1k = 36.9% of fallback-only ($1.255/1k) | mean latency 0.72s vs 1.12s intent77 fallback anthropic/claude-haiku-4.5 conf>=0.95 | routed 43.3% | cascade acc 78.8% (Jev 78.8%, fallback 76.2%) | $0.584/1k = 46.5% of fallback-only ($1.255/1k) | mean latency 0.83s vs 1.12s