system task n fail accuracy (95% CI) p50 s p95 s $/1k dec. typesafe/jev-1.13 intent8 160 0 84.4% (78.0-89.2) 0.40 0.52 0.0151 typesafe/jev-1.13 intent77 231 0 80.5% (74.9-85.1) 0.37 0.47 0.0400 typesafe/jev-1.13 inject 400 0 87.0% (83.3-89.9) 0.35 0.46 0.0136 precision 1.000 recall 0.694 F1 0.819 AUROC 0.991 Brier 0.090 (tp 118 fp 0 fn 52 tn 230) anthropic/claude-haiku-4.5 intent8 160 0 78.8% (71.8-84.4) 1.53 1.99 0.3601 anthropic/claude-haiku-4.5 intent77 231 0 76.2% (70.3-81.2) 1.48 1.97 1.2677 anthropic/claude-haiku-4.5 inject 400 0 89.0% (85.6-91.7) 1.56 2.12 0.3473 precision 1.000 recall 0.741 F1 0.851 AUROC 0.982 Brier 0.098 (tp 126 fp 0 fn 44 tn 230) anthropic/claude-haiku-5.5 intent8 160 0 83.8% (77.3-88.7) 1.38 1.95 0.0500 anthropic/claude-haiku-5.5 intent77 231 0 75.3% (69.4-80.4) 1.34 1.71 0.1777 anthropic/claude-haiku-5.5 inject 400 0 84.8% (80.9-87.9) 1.36 1.77 0.0454 precision 1.000 recall 0.641 F1 0.781 AUROC 0.982 Brier 0.142 (tp 109 fp 0 fn 61 tn 230) anthropic/claude-sonnet-5.5 intent8 160 0 90.0% (84.4-93.8) 1.75 2.26 0.9714 anthropic/claude-sonnet-5.5 intent77 231 0 80.5% (74.9-85.1) 1.46 1.82 3.5232 anthropic/claude-sonnet-5.5 inject 400 2 90.8% (87.5-93.2) 1.78 2.57 0.8866 precision 1.000 recall 0.792 F1 0.884 AUROC 0.996 Brier 0.081 (tp 133 fp 0 fn 35 tn 230) Jev confidence gating on intent8 confidence >= 0.00: coverage 100.0% accuracy 84.4% confidence >= 0.50: coverage 96.2% accuracy 86.4% confidence >= 0.70: coverage 88.1% accuracy 90.8% confidence >= 0.80: coverage 80.6% accuracy 93.0% confidence >= 0.90: coverage 73.1% accuracy 94.9% confidence >= 0.95: coverage 60.0% accuracy 95.8% Jev confidence gating on intent77 confidence >= 0.00: coverage 100.0% accuracy 80.5% confidence >= 0.50: coverage 92.6% accuracy 80.8% confidence >= 0.70: coverage 83.1% accuracy 84.4% confidence >= 0.80: coverage 77.5% accuracy 87.7% confidence >= 0.90: coverage 69.7% accuracy 90.7% confidence >= 0.95: coverage 59.7% accuracy 94.9%