From Observability to Understanding: Automated Incident Triage Using Large Language Model Reasoning Over Logs, Metrics, and Traces

Abstract

Modern cloud-native and microservice-based systems generate massive volumes of heterogeneous telemetry, including logs, metrics, and distributed traces, as a direct consequence of fine-grained service decomposition, elastic scaling, and geographically distributed deployments. While these signals are indispensable for ensuring availability, performance, and reliability, their velocity, dimensionality, and predominantly unstructured formats routinely overwhelm traditional rule-based incident triage systems and threshold-driven alerting mechanisms. Existing AIOps solutions predominantly rely on statistical anomaly detection or supervised learning models trained on historical failure patterns; although effective for known issues, these approaches struggle to generalize to unseen failure modes, rapidly evolving system topologies, and context-dependent cascading faults that are characteristic of modern production environments. This paper presents an automated incident triage framework that leverages Large Language Model (LLM) reasoning over structured logs, metrics, and distributed traces to address these limitations. Building upon foundational research in distributed tracing, log parsing, and machine learning–based anomaly detection, the proposed framework enables LLMs to semantically interpret multi-modal telemetry, correlate signals across temporal and causal boundaries, and synthesize coherent causal narratives describing system failures. By ranking probable root causes and generating actionable remediation hypotheses in natural language, the framework elevates telemetry analysis from low-level signal detection to high-level operational reasoning. Integrating LLM-based reasoning atop existing observability pipelines allows organizations to preserve proven monitoring infrastructures while substantially reducing mean time to diagnosis (MTTD), improving operator situational awareness, and enhancing the reliability of large-scale distributed systems under dynamic and unpredictable workloads.