An AI Agent That Actually Investigates Your Production Incidents

It's 2 AM and something's broken in production. You've got alerts firing, dashboards lighting up, and a pile of logs you need to cross-reference across half a dozen systems before you can even figure out what went wrong. What if an agent could do that first pass for you—querying your observability stack, correlating data, and pointing you at a likely root cause? That's the pitch behind HolmesGPT, an open-source AI agent from the Robusta team that's now a CNCF sandbox project.

What It Does

HolmesGPT is an AI agent built specifically for investigating production incidents and finding root causes. It works with any stack—Kubernetes, VMs, cloud providers, databases, and SaaS platforms—so you're not locked into a particular environment. The core mechanic is an agentic loop: the agent queries live observability data from multiple sources, reasons over what it finds, and iterates until it can identify a root cause.

It connects to a wide range of data sources through what the project calls "toolsets." Built-in integrations include Prometheus, Grafana, Datadog, Kubernetes, AKS, ArgoCD, Atlassian Rovo, and many more, plus support for any REST API. It can fetch alerts from AlertManager, PagerDuty, OpsGenie, or Jira—and write findings back. On the model side, it works with any LLM provider: OpenAI, Anthropic, Azure, Bedrock, Gemini, and others. Notably, while it's a CNCF project with Kubernetes roots, it doesn't require Kubernetes to run.

The newer Operator mode extends this further. Instead of waiting for a human to notice a problem and trigger an investigation, HolmesGPT can run in the background 24/7, spot issues before your customers do, and message you in Slack with a proposed fix. With the GitHub integration connected, it can even open a pull request to fix what it finds.

Why It's Cool

• It's designed for the scale of real observability data. Anyone who's tried to feed large log or metric payloads into an LLM knows the context window becomes a wall fast. HolmesGPT handles this with server-side filtering, JSON tree traversal, and tool output transformers that keep big payloads out of the context window. There are per-tool memory limits, streaming of large results to disk, and automatic output budgeting to prevent OOM kills. That's the unglamorous plumbing that makes the difference between a demo and something you'd actually run against production.

• It meets you where your infrastructure already lives. The "no Kubernetes required" point matters more than it sounds. Plenty of incident investigation tools assume a containerized world. HolmesGPT's health checks can query any connected data source, whether that's a VM, a cloud service, a database, or a SaaS platform.

• The alert integrations are bidirectional. Reading alerts is table stakes. Writing findings back to AlertManager, PagerDuty, OpsGenie, or Jira means the investigation becomes part of your existing workflow instead of living in yet another tool nobody checks.

• Operator mode closes the detection gap. Most AI troubleshooting tools still need a human to say "go." Running a health check alongside a deployment to verify the new version, or scheduling continuous checks to catch regressions, means the agent can be the first responder rather than the second.

• You're not locked into one LLM. Provider flexibility is useful here—different teams have different constraints on cost, data residency, and model access. Being able to swap providers without rewriting your setup is a practical win.

How to Try It

The project is at github.com/robusta-dev/holmesgpt, and the docs live at holmesgpt.dev. Start with the installation guide linked from the README, then connect the toolsets that match your stack.

1. Head to the repository and follow the installation instructions.
2. Configure an LLM provider—OpenAI, Anthropic, Azure, Bedrock, Gemini, or another supported option.
3. Wire up your data sources. Start with what you already have—Prometheus, Grafana, Datadog, or Kubernetes are all built in—and add custom toolsets or REST API connections as needed.
4. Run an investigation against a real (or simulated) incident and see what the agent surfaces.
5. If you want the background monitoring, look into Operator mode: deployment verification and scheduled health checks are the two starting points.

If you want to extend it, custom toolsets let you add your own data sources, and the docs cover that path directly.

Final Thoughts

HolmesGPT is a pragmatic take on a category that's easy to overhype. It doesn't claim to replace your on-call engineers—it claims to do the tedious first pass of cross-referencing observability data so you can start from a hypothesis instead of a blank page. The engineering choices around context management and memory safety suggest the team has actually run this against large datasets, not just built a demo. If you're running a stack with a lot of moving parts and a lot of alerts, it's worth a look—especially if you've been curious about AI agents for ops but wary of the ones that fall over the moment your logs get big.