Back to Articles ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks

ITBench‑AA, launched on May 27 2026 by Artificial Analysis and IBM, evaluates frontier models on 59 Kubernetes incident‑response tasks, split into 40 public and 19 new held‑out cases. The leaderboard shows Claude Opus 4.7 (Adaptive Reasoning, Max Effort) at 47 %, GPT‑5.5 (xhigh) at 46 %, and Qwen3.7 Max at 42 %, all scoring below the 50 % threshold and highlighting a gap in agentic IT capabilities. Turn‑count analysis reveals that GPT‑5.5 averages 31 turns per task while Gemini 3.1 Pro Preview averages 83 turns, yet longer trajectories do not translate into higher accuracy due to false‑positive penalties for extra root‑cause entities. Cost‑per‑task comparisons show Gemma 4 31B achieving 37 % at $0.14 per task, outperforming Gemini 3.1 Pro Preview ($2.23) and GLM‑5.1 ($1.23). The benchmark uses the open‑source Stirrup harness with a 100‑turn cap and 3 repeats per task, ensuring consistent evaluation across models. The dataset includes alerts, events, traces, metrics, logs, and topology snapshots, requiring agents to identify minimal sets of independent root‑cause Kubernetes entities. The results suggest that current frontier models still struggle with complex SRE scenarios, especially when penalised for over‑investigation. Future iterations of ITBench‑AA will expand to FinOps and CISO tasks, providing a broader view of enterprise IT agent performance.