When production breaks at 3AM, 3AM is already awake.
3AM is an autonomous incident-response agent that automatically investigates production incidents before engineers start debugging. Built on TrueForge (open-source agent harness), it correlates metrics from Grafana and errors from Sentry to generate a Root Cause Analysis (RCA) then posts it to Slack and Notion.
This project started from a real shift, not a thought exercise.
I'm on production on-call at Fampay. The pattern is familiar to anyone carrying a pager. An alert fires at 3 AM, and the next 30 to 45 minutes follow the same script every time: open Sentry, read the error, pull the stack trace, open Grafana, check whether latency moved, check whether the database pool saturated, check whether traffic shifted, cross-reference timestamps, rule out the two things that aren't wrong, and finally land on a root cause. Then write it up so the next person doesn't repeat it.
That work is mechanical, repetitive, and happens at the worst possible hour. It's also the part of on-call that nobody has automated, because automating it isn't a matter of adding another dashboard.
Alerting tells you that something broke. Nothing tells you what, why, or what to do about it — because that answer requires reading infrastructure metrics and application errors side by side and reasoning across them, and no single system holds both halves.
3AM is what I'd want to receive at 3 AM instead of a raw alert: a written root-cause analysis, with cited evidence, a confidence score, and a recommended next action, already waiting.
The premise: an incident is a correlation problem, and correlation is what language models are genuinely good at. The metrics platform knows p99 latency went from 120 ms to 4.9 s and that the connection pool saturated, but not that a specific migration took a table lock. The error tracker knows a deadlock exception appeared at one line and one timestamp, but not whether it accounts for the customer impact. The answer only exists where the two overlap — and finding that overlap by hand at 3 AM is slow, repetitive, and exactly the kind of judgment work that shouldn't require a senior engineer.
https://www.loom.com/share/1892a6c703304153818e5b1354d4d438
It runs unattended. The agent has read-only tool access to metrics and errors. It cannot restart services, scale anything, or change configuration. A system that can remediate during an investigation can also remediate incorrectly during a bad inference, at the worst possible time. So its worst failure mode is a wrong paragraph in a postmortem, not an outage.
It reports uncertainty instead of manufacturing confidence. Confidence is a number derived from cross-system agreement. It drops when evidence is thin, when the two systems disagree, or when one has no data for the window. When the data cannot support a conclusion, 3AM says so and says what it would check next. A confidently wrong root cause is worse than no answer, because it sends a whole team down the wrong path for hours.
It separates observation from inference. Facts land in evidence, each attributed to its source with concrete values. Interpretation lands in hypotheses, explicitly marked unconfirmed. Every tool call is retained and inspectable, so the reasoning is auditable rather than something you have to take on faith.
It absorbs alert noise. Deduplication folds repeated firings for the same service and alert type into one incident, so a flapping alert produces one investigation rather than a swarm of them.
It gets better over time. This is the long game. Every investigation leaves behind a structured record: the alert, the queries run, the evidence found, the hypothesis, the confidence, and how it compared to the actual outcome engineers reached. Run against a real incident history, that becomes a corpus of your system's failure modes. Point it at last quarter's postmortems and the confidence calibration, the query selection, and the root-cause phrasing all improve, because it is no longer reasoning from a generic model of production but from a specific model of your production.
That's the difference between an agent that summarizes alerts and one that accumulates institutional knowledge about a codebase while the team sleeps.
Incident at 3AM
↓
Engineer wakes up, opens 5 tabs
↓
Grafana (metrics) → Sentry (errors) → Logs → Traces
↓
Manual correlation → Hypothesize → Test → Fix
↓
45 minutes later: "DB pool exhaustion from retry storm"
3AM automates the investigation — you wake up to a complete RCA with evidence.
The investigation is genuinely open-ended. The agent decides which queries to run, notices a failed datasource lookup and retries it, separates real customer-facing failures from control-plane noise in the error stream, and aligns timestamps across two systems to find where they corroborate each other. It adapts as the picture changes.
A fixed runbook would have to know in advance which queries to run, in what order, and how to interpret every result — and would have no idea what to do when a query fails, an unfamiliar error type appears, or the evidence genuinely does not support a conclusion. That last case is precisely where deterministic automation is worst and an agent is most useful.
1. Application emits signals
Your app exposes /metrics (Prometheus format) and sends errors to Sentry on every request. Latency histograms, error counters, DB pool gauges — all automatic.
2. Grafana evaluates, alerts fire
Grafana Cloud runs PromQL alert rules continuously. When p95 latency crosses 2s, or error rate exceeds 5%, or DB pool hits 80% — an alert fires and POSTs to 3AM's webhook.
3. 3AM receives, deduplicates, creates incident
The webhook accepts Grafana's payload. 3AM deduplicates by alertname + service within a 5-minute window — so a flapping alert doesn't create multiple incidents. One incident = INC-0001, INC-0002, etc.
4. TrueForge session starts, agent investigates
3AM calls TrueForge SDK: sessions.create({ agent: "three-am-investigator" }) → createTurnStream(). The agent now has a dedicated session for this incident.
5. Agent queries both systems, correlates
The agent uses Grafana MCP to query: "What's the p95 latency trend? DB pool waiting? Error rate?"
It uses Sentry MCP to query: "What exceptions spiked? Stack traces? Affected endpoints?"
The LLM correlates: latency spike + pool exhaustion + pg_advisory_lock timeouts = DB lock contention
6. Structured RCA generated
Output is forced JSON with: summary, timeline, root cause (facts separated from hypotheses), evidence table with queries, confidence score, mitigation steps.
7. Delivered to humans
Slack gets a 3-line summary with a link. Notion gets the full investigation with timeline and evidence. Engineer opens Notion, understands the issue in 30 seconds.
# INC-0001 — Database Pool Exhaustion
## Summary
Blog API at 100% error rate due to database connection pool exhaustion.
## Timeline
**03:12:00** — Alert fired: DbPoolExhausted
**03:12:05** — INC-0001 created, TrueForge session started
**03:12:30** — RCA complete
## Affected Service
`blog` (Express API)
## Root Cause
**Observed Fact:** `db_pool_waiting` spiked to 18/20 connections waiting
**Observed Fact:** Sentry shows 247 `pg_advisory_lock` timeout errors in 2 minutes
**Hypothesis:** Chaos endpoint `/chaos/db-lock` holds advisory lock + exhausts pool
## Evidence
| Source | Query | Result |
|--------|-------|--------|
| Grafana | `max(db_pool_waiting) by (instance)` | 18 waiting |
| Sentry | `issue.type:exception pg_advisory_lock` | 247 events |
| Grafana | `rate(http_requests_total{status=~"5.."}[1m])` | 100% error rate |
## Confidence: 95%
## Recommended Mitigation
1. Immediate: POST `/chaos/stop` to release locks
2. Short-term: Add query timeout + pool monitoring alert
3. Long-term: Implement circuit breaker for DB calls| Layer | What It Does | Technology |
|---|---|---|
| Application | Demo production service with full instrumentation | Express, Postgres, prom-client, Sentry SDK |
| Metrics | Collects /metrics, stores in Grafana Cloud, evaluates alerts |
Grafana Alloy → Grafana Cloud (managed Prometheus) |
| Errors | Captures exceptions, stack traces, traces automatically | Sentry Cloud |
| Incident Ingestion | Receives Grafana webhooks, deduplicates, creates incidents | 3AM Express API (:3001) |
| Agent Runtime | Runs agent sessions, manages MCP tools, executes LLM | TrueForge (@truefoundry/trueforge) |
| Investigation | Queries Grafana + Sentry via MCP, correlates, outputs RCA | 3AM Agent (Grafana MCP + Sentry MCP + LLM) |
| Notifications | Delivers RCA to Slack (summary) + Notion (full) | Slack API + Notion API |
TrueForge owns the agent, handling sessions, MCP tools, model execution, context, and lifecycle. 3AM owns the incident workflow around it: webhook intake, deduplication, incident lifecycle, and notifications. That split is the whole architecture, and it means 3AM never implements an agent runtime of its own.
One TrueForge session per incident, giving a clean mapping from INC-xxxx to session.id, so every investigation has an isolated context and an inspectable transcript.
MCP servers are attached on TrueForge, with Grafana MCP for metrics and Sentry MCP for errors, configured as connectors so credentials never live in the agent definition.
Cloud-first observability, using Grafana Cloud and Sentry Cloud. A local collector scrapes and remote-writes, but no state is stored locally.
You'll need Docker with Docker Compose, a .env file populated with your credentials, and an OpenAI API key configured through the TrueForge UI.
git clone <repo> && cd 3AM
cp .env.example .env
# Edit .env with your credentials
docker compose up --build -dOpen these for the demo:
| What | Where |
|---|---|
| Admin panel (start here) | http://localhost:3000/admin |
| TrueForge UI (watch the session) | http://localhost:8791 |
| 3AM health | http://localhost:3001/healthz |
| Grafana dashboard | your Grafana Cloud URL |
- Start traffic — Admin panel → "Start Normal Traffic"
- Verify healthy — Grafana shows flat latency, 0% errors
- Break it — Admin panel → "DB Lock" (or Latency, Error Rate, etc.)
- Watch — Grafana spikes → Alert fires → 3AM webhook hit → TrueForge session opens
- Get RCA — Check Slack (summary) + Notion (full details)
- Recover — Admin panel → "Stop All" → Metrics normalize → Incident auto-closes
curl -X POST http://localhost:3001/trigger \
-H 'Content-Type: application/json' \
-d '{"alertname":"HighErrorRate","service":"blog"}'
curl http://localhost:3001/incidents/INC-0001 | python3 -m json.toolCreate .env from .env.example:
# Application
DATABASE_URL=postgres://user:pass@host:5432/db # Neon Postgres (unpooled)
SENTRY_DSN=https://xxx@o1.ingest.sentry.io/xxx
# 3AM Incident Response
TRUEFORGE_BASE_URL=http://host.docker.internal:8791 # TrueForge on host
TRUEFORGE_AGENT_NAME=three-am-investigator
TRUEFORGE_MODEL=openai/gpt-4o # Model FQN from TrueForge UI
THREE_AM_PORT=3001
# Notifications (optional for demo)
SLACK_BOT_TOKEN=xoxb-xxx
SLACK_CHANNEL_ID=Cxxx
NOTION_TOKEN=ntn_xxx
NOTION_DATABASE_ID=xxx
# Metrics → Grafana Cloud (required)
GCLOUD_HOSTED_METRICS_URL=https://prometheus-xxx.grafana.net/api/prom/push
GCLOUD_HOSTED_METRICS_ID=xxx
GCLOUD_RW_API_KEY=glc_xxxTrueForge model setup: Open http://localhost:8791 → Settings → Models → Add your OpenAI/Anthropic key → Copy model FQN → Set as TRUEFORGE_MODEL.
| Scenario | What It Does | Grafana Signal | Sentry Signal |
|---|---|---|---|
| DB Lock | Holds PG advisory lock + exhausts connection pool | db_pool_waiting ↑, latency ↑ |
pg_advisory_lock timeouts |
| Latency | Injects 800ms–2s delay on /posts |
p95 latency ↑ | Timeout errors |
| Traffic Spike | 10x RPS background load generator | RPS ↑, latency ↑ | Rate limit errors |
| Retry Storm | Downstream retry loop with backoff failure | Latency ↑, error rate ↑ | Retry exhaustion errors |
| Dependency Fail | Fake external service returns 500 | Error rate ↑ | 502/504 errors |
| Error Rate | Random 500s on blog routes | Error rate ↑ | 500 exceptions |
| Stop All | Resets all flags, kills background loops | Metrics recover | Errors stop |
Each scenario moves both Grafana metrics and emits Sentry errors — so the agent always has correlated evidence.
3AM/
├── AGENTS.md # Agent spec & workflow (source of truth)
├── ARCHITECTURE.md # Detailed architecture diagrams
├── RUN.md # All run commands
├── docker-compose.yml # Blog + Alloy + 3AM (TrueForge runs on host)
├── package.json # pnpm workspace root
├── .env.example
│
├── apps/
│ ├── blog/ # Demo production app
│ │ ├── src/
│ │ │ ├── index.js # Express + routes + /metrics + /healthz
│ │ │ ├── db.js # pg pool + gauges
│ │ │ ├── metrics.js # Prometheus histograms/counters
│ │ │ ├── sentry.js # Sentry initialization
│ │ │ ├── chaos/ # 6 chaos scenarios + stop
│ │ │ ├── routes/
│ │ │ └── public/admin.html # Admin panel (zero-dep vanilla JS)
│ │ └── Dockerfile
│ │
│ └── 3am/ # Incident-response workflow
│ ├── src/
│ │ ├── index.js # Webhook + incident API
│ │ ├── incidents.js # Create + in-memory dedupe
│ │ ├── trueforge.js # TrueForge SDK client
│ │ ├── investigate.js # Build prompt, parse RCA
│ │ └── notify/ # Slack + Notion + RCA template
│ └── Dockerfile
│
├── infra/
│ └── alloy/config.alloy # Scrapes blog → Grafana Cloud
│
└── trueforge/
├── agent.json # 3AM agent spec
├── mcp.json # Grafana + Sentry MCP configs
└── README.md # TrueForge setup guide
# Terminal 1: TrueForge (Neon + Upstash)
DATABASE_URL=... REDIS_URL=... PUBLIC_BASE_URL=http://localhost:8791 \
npx @truefoundry/trueforge --port 8791
# Terminal 2: Blog (against Neon)
cd apps/blog && pnpm dev
# Terminal 3: Alloy (scrapes local blog)
cd infra/alloy && docker run -p 12345:12345 -v $(pwd)/config.alloy:/etc/alloy/config.alloy:ro \
-e GCLOUD_HOSTED_METRICS_URL -e GCLOUD_HOSTED_METRICS_ID -e GCLOUD_RW_API_KEY \
grafana/alloy:v1.7.4 run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
# Terminal 4: 3AM (against TrueForge :8791)
cd apps/3am && pnpm devBuilt on TrueForge as the agent harness, with Grafana for metrics and alerting, Sentry for error tracking, and OpenAI for model reasoning.
MIT — Hackathon project, use freely.
Thank you TrueForge Team for the harness really cool!
This is going to be actual on call agents at fampay - which will handling thousands of alert everyday : )
