In a recent evaluation of AI’s capabilities in incident response, the ARFBench benchmark, developed by Datadog and Carnegie Mellon, demonstrated that current leading AI models, including GPT-5, still lag behind human engineers in analyzing real production incidents. While GPT-5 achieved an accuracy of 62.7%, it fell short of domain experts who scored 72.7%. The benchmark, crafted from 63 genuine incidents, reflects the complexities of incident analysis, emphasizing that successful AI deployment in observability relies not on replacing engineers but on enhancing their capabilities through collaboration. This study highlights the significant potential for hybrid workflows, where AI tools assist rather than supplant human judgment.

GPT-5: GPT-5 is OpenAI’s frontier general-purpose language model, positioned as one of the most capable models for broad reasoning and analysis tasks. Here it is used as a reference point on ARFBench, where it leads among general models but still does not match human engineers on real incident questions.
Datadog: Datadog is an observability and monitoring company whose products help teams track infrastructure, application, and incident health. In this news, Datadog co-created ARFBench and used its own internal time-series system alongside an AI model to show that domain-specific tools can outperform general-purpose models on incident reasoning.
ARFBench: ARFBench, short for Anomaly Reasoning Framework Benchmark, is a benchmark for evaluating how well AI systems reason about production incidents and monitoring data. It was built from real outages and engineering discussion logs rather than synthetic examples, and in this news it is the core evaluation showing that current models still trail human engineers on incident analysis.
Carnegie Mellon: Carnegie Mellon University is a major research university known for work in computer science, AI, and data systems. In this story, Carnegie Mellon is a research partner on ARFBench, helping frame the benchmark around real-world time-series reasoning and incident response.

`json
{
“Observability”: “Modern observability benchmarks are now focusing on analyzing real production incidents, as these scenarios involve complex, time-sensitive evidence critical for accurate incident response.”,
“Time-Series Reasoning”: “Benchmarks targeting anomaly detection and cross-metric reasoning reveal weaknesses in cutting-edge models that are not apparent from general conversational or coding assessments.”,
“Human-AI Collaboration”: “Recent benchmark findings indicate that the most effective approach may involve hybrid workflows where AI supports engineers, rather than replacing them entirely.”
}
`