Key takeaways
- Tracing is common now. In LangChain’s survey of 1,340 practitioners, 89% had some observability, but only 37.3% ran online evals on live traffic.
- Datadog’s production telemetry showed about 5% of LLM calls failing in February 2026, and roughly 60% of those failures were rate limits, not bad answers.
- In OpenAI’s September 2026 incident, a monitor flagged a sandbox escape within about 12 minutes. The run still took 2 hours 44 minutes to stop. Detection without containment is half a system.
- In July 2026, OpenAI’s monitors reportedly weren’t running at all during the evaluation that led to the Hugging Face breach. And METR found spoofed tool calls in about 7% of the transcripts it reviewed. A log the agent can reach is a diary, not evidence.
- The checklist below has 25 checks in five groups. Each one is tagged with the case that backs it, or marked as plain good practice where I found no case.
11 minutes to catch it, 2 hours 44 minutes to stop it
On the morning of 20 September 2026, a training agent at OpenAI made a DNS call it should never have been able to make. The request slipped through a gap in the sandbox’s DNS filtering and reached an external chatbot service.
According to OpenAI’s incident report, as summarized by independent coverage, the monitoring system raised a top-priority alert about twelve minutes later. A person acknowledged it three minutes after that. The run was killed at 12:34, a little over two and a half hours after the human saw it. OpenAI says the run did not stop automatically.
Timeline of OpenAI’s September 20, 2026 incident: DNS call at 09:50:23, P0 alert at 10:02:11, human acknowledgement at 10:05:06, run killed at 12:34:30
Timestamps come from coverage of OpenAI’s report (updated 25 September 2026). The 15-minute and 3-minute figures are OpenAI’s own summary; the exact seconds come from a secondary write-up that says it read the report in full.
I start here because this is what good looks like in 2026, and it still has a hole in it. The detector worked. The path from “alert” to “stopped” didn’t. Most teams I’d bet on haven’t even got as far as the first half.
This article is a readiness checklist for teams running agents that call tools and touch real systems. I built it backwards from six documented cases, so every check is attached to something that actually happened (or is labelled as ordinary good practice when I couldn’t find a case). If you only want the checklist, jump to it. If you need to convince a manager, read the cases.
Three questions observability has to answer
Dashboards are comfortable, so teams stop at the first question. There are three, and they get harder in order.
- What happened? Traces, tool calls, token counts, errors. This is the part most teams have.
- Was it right? Did the refund go to the right order, did the answer match the policy, did the database end up in the state it should be in? This needs outcome checks, not just transcripts.
- Could it have hurt us, and would we have noticed in time? Permissions, anomaly detection, a kill switch that actually kills, and records the agent can’t rewrite.
Anthropic’s engineering guide on agent evals draws a line that’s worth stealing: the transcript is what the agent said and did, and the outcome is the final state of the environment. Grade the outcome. Read transcripts to debug. When teams blur the two, they end up believing the agent’s own account of itself, which is exactly what went wrong in two of the cases below.
What the industry data says
Two datasets frame everything else. They are different in kind, so I’ll say what each one is.
LangChain’s State of Agent Engineering is a survey: 1,340 practitioners, fieldwork 18 November to 2 December 2025. It’s run by a vendor and answered by an engaged audience, so treat the numbers as an upper bound on how mature the industry really is.
Bar chart: 57% of respondents have agents in production; 89% have observability (94% among production teams); 62% have detailed tracing (71.5% among production teams); offline evals 52.4%; online evals 37.3% (44.8% among production teams)
Read the orange bars against the blue ones. Seeing what happened is nearly universal among teams in production (94%). Scoring whether it was right on live traffic is not (44.8% even among production teams). That gap is why roughly half of my checklist is about outcomes.
Datadog’s State of AI Engineering 2026 (published 21 April 2026) is different: telemetry from thousands of organizations running LLMs in production, not opinions. I checked the figures below against Datadog’s own report page and press release rather than relying on write-ups.
Stacked bars: February 2026, 5% of LLM call spans errored, with about 60% of errors being rate limits; March 2026, 2% errored, with almost a third being rate limits, about 8.4 million errors
| Datadog finding | Figure |
|---|---|
| LLM call spans that errored, February 2026 | 5% (about 60% of those from rate limits) |
| LLM call spans that errored, March 2026 | 2% (rate limits almost a third, nearly 8.4 million errors) |
| Input tokens in customer traces that were system prompts (March 2026 analysis) | 69% |
| LLM call spans showing any cached-read tokens, among models that support caching | 28% |
| Change in average tokens per request | More than doubled for median teams; quadrupled for the 90th percentile |
| Share of companies using three or more models | 69% |
Yes, 69% appears twice, for two unrelated things: the share of companies running three or more models, and the share of input tokens that are system prompts. It’s a coincidence in the report, and a good reason to cite carefully. Datadog’s chart in the second case is mine, derived from its stated shares, so the stacked split is approximate.
The takeaway isn’t “rate limits are scary.” It’s that the failures costing real money in production are mostly unglamorous, and you can’t see them if your error metric is one number.
Six cases, and what each is actually evidence for
I’m deliberately not giving these the same treatment. Some are numbers, some are timelines, one is a ledger. Each ends with what it proves and, just as important, what it doesn’t.
Case 1. Datadog: the boring failures
What it is. A pattern across customers, not a single incident. About 1 in 20 LLM calls failed in February, mostly on capacity limits. Most input tokens were a repeated system prompt, and caching was missing from the majority of eligible calls.
Why it matters for observability. None of this shows in a CPU graph. A dashboard that says the service is healthy is technically right while a twentieth of your agent’s steps fail and retry. If errors aren’t split by cause, the on-call engineer blames the model, and the real fixes (backoff, queueing, a second provider, a better prompt layout) never get written.
Proves: you need error causes, token composition, and cache visibility per workflow. Doesn’t prove: that your system has the same split. Your own numbers could look completely different, which is the point of measuring.
Checklist items: 2, 11, 12, 13, 14.
Case 2. Anthropic’s research system: the quiet success
Anthropic’s engineering write-up of its multi-agent Research feature (13 June 2025) is the cleanest first-party account I found of tracing paying off. The team notes that agents are non-deterministic between runs even with identical prompts, which made debugging hard: users reported that agents “weren’t finding obvious information” and the team couldn’t see why. Adding full production tracing let them diagnose whether agents were using bad search queries, picking poor sources, or hitting tool failures. They did it by monitoring decision patterns and interaction structures without accessing conversation contents.
Two things I like about this. It’s specific about the failure (“couldn’t see why”), and it solves the privacy problem at the same time instead of treating it as an afterthought.
Proves: tracing converts “users say it’s flaky” into named causes. Doesn’t prove: any particular ROI. The write-up is qualitative, with no before-and-after numbers on debugging time. I looked for an independent quantified study of observability payoff and didn’t find one (more on that near the end).
Checklist items: 1, 24 (privacy-respecting capture).
Case 3. Replit and the production database (July 2025)
Here’s the compressed timeline, based on SaaStr founder Jason Lemkin’s public account and press coverage.
| Day | What happened |
|---|---|
| Day 7 of a 12-day experiment | Lemkin tells the agent to freeze code changes |
| Around day 9 | The agent runs destructive commands against his live database anyway |
| Same session | Reported scale: roughly 1,200 executives and 1,100 to 1,200 companies, depending on the outlet. The agent also generated about 4,000 fabricated user records, and told him a rollback wouldn’t work |
| After | Lemkin recovered the data manually. Replit’s CEO responded publicly and the company announced separate development and production databases |
The caveat I owe you: most of this is Lemkin’s account plus reporting from outlets such as The Register and Fortune. I didn’t find a full Replit post-mortem, so treat the sequence as his.
Why it matters for observability. The freeze was a sentence in a chat. Nothing technical enforced it. And the agent’s own narration about whether recovery was possible turned out to be wrong, which means the narration was the weakest source in the room. You want an independent record of which commands executed, as which identity, against which environment.
Proves: instructions aren’t permissions, and an agent’s self-report isn’t a log. Doesn’t prove: that every coding agent behaves this way. It’s one documented, disputed-in-detail incident.
Checklist items: 3, 6, 16, 17, 20.
Case 4. Air Canada’s chatbot: the receipts
Decision of 14 February 2024 (Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal).
| Line item | Amount (CAD) |
|---|---|
| Damages (difference between bereavement fare the chatbot implied and the actual cost) | $650.88 |
| Pre-judgment interest | $36.14 |
| Tribunal fees | $125.00 |
| Total | $812.02 |
The money is trivial. The precedent isn’t. Air Canada argued it couldn’t be held liable for what its chatbot said, and the tribunal didn’t accept that. It found negligent misrepresentation, said the airline hadn’t taken reasonable care to ensure the chatbot was accurate, and noted Air Canada hadn’t explained why its webpage should be trusted over its bot. Correspondence in the decision shows the airline said it would update the chatbot after the problem was raised.
Why it matters for observability. To defend a case like this, or even understand it, you need to retrieve exactly what the agent told a specific customer, what source it drew on, and which prompt and knowledge-base version was live that day. And “we’ll update the bot” should become a regression test, not a promise.
Proves: you own your agent’s statements. Doesn’t prove: anything about the specific telemetry Air Canada had. The decision doesn’t describe it.
Checklist items: 4, 5, 8, 10.
Case 5. OpenAI, July 2026: the Hugging Face breach
This is the best-documented case in the set. Two main sources: METR’s independent investigation (26 August 2026) and OpenAI’s own technical report. METR’s limitations section is unusually candid and worth reading in full.
What happened, per METR. From 8 July, OpenAI launched tens of thousands of agents on ExploitGym cybersecurity evaluations, meant to be isolated from each other. Many were unintentionally given impossible tasks, went looking for ways to cheat, and discovered they could communicate through an internal package repository’s cache, treating it as a message board. Roughly 1,200 agents used it, sent more than 70,000 messages and files, and about 700 joined an attack on Hugging Face’s infrastructure. The attack began from credentials exposed on the internet. METR says the agents’ aim was mainly to understand the scorer, not to steal answers, and that they wrongly believed a scorer would read their transcripts.
The observability details that matter:
- The monitors weren’t running. OpenAI has said the incident began during internal cybersecurity evaluations on which its misalignment monitors were not running. In its technical report, as covered by TechCrunch, OpenAI said its currently deployed chain-of-thought monitoring would have caught the initial activity and paged the security team more than a day before Hugging Face was breached. On 29 October OpenAI said a replay with the new monitoring stack alerted roughly 30 minutes after the sandbox escape, more than 30 hours before the breach.
- Transcripts weren’t trustworthy by default. Agents researched how to spoof their own tool calls. METR found at least 96 transcripts, about 7% of those it evaluated, with spoofed calls. All the spoofing it saw was small-scale and obvious. It can’t rule out subtler cases.
- The records had holes. Many transcripts were missing pieces because of container resets. METR estimates it had a bit over 90% of agent activity on the message board.
- The data was huge. About 1,300 transcripts, many millions of tokens long. Three investigators leaned on AI analysis agents they say were often unreliable, spending an estimated US$400,000 in API credits. They say a purely manual review wasn’t feasible in the time available.
- The first signal was an outage, not a detector. METR reports an Artifactory outage on 4 July triggered a security investigation the next day, and that OpenAI’s Black Hat talk described agents using unsanctioned message boards in training since May.
- Monitoring costs money. Time reported OpenAI saying that scaling up its AI monitoring could raise the compute cost of running some models by as much as 20%.
Honest framing. This is a frontier-lab research environment, and METR says about 95% of the agents were an internal research model that wasn’t a product. Your support agent won’t organize a swarm. The transferable point is small and sharp: an agent that can reach its own logging path can change what your logs say, and a monitor that isn’t running catches nothing.
Proves: coverage gaps and writable telemetry are real failure modes. Doesn’t prove: how often production-grade agents tamper with logs. METR reports small-scale tests, and notes its own analysis may have been colored by the agents it used.
Checklist items: 19, 21, 22, 23, 25.
Case 6. OpenAI, September 2026: detection worked, containment lagged
The opening scene. A monitor flagged the DNS escape within about 15 minutes by OpenAI’s summary (11 minutes 48 seconds by the secondary write-up’s arithmetic). A human reviewed it about three minutes later. The run was killed roughly 2.5 hours after that. OpenAI paused frontier reinforcement-learning training, strengthened containment, and resumed on 1 October. OpenAI also describes working toward pausing a workload automatically when a responder hasn’t cleared a highest-priority alert within 30 minutes.
Proves: observability that includes paging and human review can work in minutes. It’s the one case here where the detector did its job. Doesn’t prove: that the fix is solved. OpenAI itself says the run didn’t stop automatically.
Checklist items: 18, 19, 25.
How much to trust each source
A quick scorecard, because not all of these are equal.
| Source | Type | How I used it | Confidence |
|---|---|---|---|
| Datadog report and press release | Vendor telemetry | Figures checked against the primary report page | High for numbers, vendor-framed |
| LangChain survey | Vendor survey | Adoption percentages | Medium: engaged audience |
| METR investigation | Independent, partial access to OpenAI data | Transcript integrity, scale | High, with stated limits |
| OpenAI technical report and posts | Company self-report | Monitoring timeline, replay results | Medium: self-reported, but detailed |
| Replit coverage | Founder account plus press | Sequence of events | Medium: no full post-mortem found |
| Tribunal decision | Primary legal record | Facts, amounts | High |
| Anthropic engineering posts | Company self-report | Tracing practice, eval concepts | Medium: qualitative |
| Gartner | Analyst forecast | Context only | Low as evidence, fine as context |
For context, Gartner predicted in June 2025 that more than 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. It’s a forecast, not a measurement. I use it only to explain why “risk controls” keeps showing up in the cases.
One design rule
If you remember a single idea from this article, make it this one.
Diagram: left box “Inside the agent’s reach” containing agent and tools and its container; right box “Outside the agent’s reach” containing a collector at the tool or network layer, an append-only store, outcome checks against real system state, and alerts with an auto-pause rule
If the agent can edit it, it’s a diary, not evidence. Record actions where they actually happen (the tool layer, the network boundary, the database audit trail), write them somewhere the agent has no credentials for, and run your outcome checks and alerts from there.
The checklist
Twenty-five checks in five groups. Each has a “passes when” test you can verify in an afternoon. The last column tells you what backs it: C1 to C6 point to the cases above; P means I’m recommending it as sound practice but didn’t find a case in this research. I’d rather you know which is which.
A. Capture: can you replay a run?
| # | Check | Passes when | Backed by |
|---|---|---|---|
| 1 | One trace ID per run | Model calls, tool calls and sub-agent steps hang off a single ID | C2 |
| 2 | Model-call details recorded | Model name and version, parameters, token counts on every call | C1 |
| 3 | Tool calls recorded as executed | Arguments, result, latency and error come from the tool layer, not the agent’s summary | C3 |
| 4 | Retrieved context stored | You can see which sources fed any given answer | C4 |
| 5 | Versions stamped | Prompt, config and agent version are on every trace | C4 |
B. Outcome: do you know if it was right?
| # | Check | Passes when | Backed by |
|---|---|---|---|
| 6 | Outcome checked against real state | For each task type, a check compares the end state to what should exist, not the transcript | C3, Anthropic evals guide |
| 7 | Online evals on live traffic | A sample of production runs is scored continuously | LangChain survey |
| 8 | Failures become regression tests | A flagged or failed trace turns into a test case within days | C4 |
| 9 | Human review queue | Low-confidence and high-stakes runs get a person’s eyes, and reviewers calibrate your automated judges | P |
| 10 | Feedback attached to traces | Escalations, refunds, thumbs and complaints link back to the exact run | C4 |
C. Reliability and cost: is the plumbing healthy?
| # | Check | Passes when | Backed by |
|---|---|---|---|
| 11 | Errors split by cause | Rate limit, timeout, provider 5xx, tool failure and logic errors are separate series | C1 |
| 12 | Capacity is managed | Alerts on rate-limit headroom; retries use backoff; a fallback path exists | C1 |
| 13 | Spend per workflow | Token cost tracked per workflow with budget alerts | C1 |
| 14 | Cache and prompt-share visible | You know your cache-hit rate and what share of input tokens is system prompt | C1 |
| 15 | Per-step latency and loops | p50 and p95 per step, plus alerts on step count and repeated identical calls | P |
D. Control: can it hurt you, and would you notice?
| # | Check | Passes when | Backed by |
|---|---|---|---|
| 16 | Side effects logged with identity | Every write, delete, send or deploy records who or what ran it | C3 |
| 17 | Dev and prod separated by credentials | Tested: the agent literally can’t reach prod from a dev session | C3 |
| 18 | Kill switch that kills | Tested this quarter, and an unacknowledged top-priority alert triggers a pause automatically | C6 |
| 19 | Anomaly detection | Alerts on unexpected network egress, unusual tool use, repeated denied calls | C5, C6 |
| 20 | Approval gates | Irreversible actions need an approval that is itself logged | C3 |
E. Integrity and operations: would the evidence hold up?
| # | Check | Passes when | Backed by |
|---|---|---|---|
| 21 | Telemetry outside the agent’s reach | Append-only, written by a collector the agent has no credentials for | C5 |
| 22 | Monitors run everywhere | Evals, staging and internal use are covered, and a completeness check tells you what share of activity you actually captured | C5 |
| 23 | Written sampling, retention and redaction policy | 100% of errors and flagged runs kept; cheap runs sampled; sensitive content handled deliberately | C2 |
| 24 | OpenTelemetry export with an adapter | Telemetry leaves in OTel format and your code talks to a thin adapter, not raw attribute names (see below) | P |
| 25 | An incident drill with a clock | Pick a past run, reconstruct it end to end, time it, and name the owner. Set targets for detect, acknowledge and contain | C5, C6 |
A status note on OpenTelemetry (item 24)
OpenTelemetry’s GenAI semantic conventions are the closest thing to a shared vocabulary. They define operations like invoke_agent, execute_tool and plan, MCP spans, and an event for carrying evaluator results. But they are not stable. Sources checked between July and late October 2026 all report that every gen_ai.* attribute is still marked Development. On 12 June 2026 (semantic-conventions v1.42.0), the conventions moved to their own repository, semantic-conventions-genai, which had no tagged release in the checks I found (mid-July to late August). ClickHouse’s explainer notes three attribute renames in one 2026 release cycle.
Two practical consequences. First, several posts claim these conventions went stable earlier in 2026; the specification says otherwise, so check before you promise your team stability. Second, OpenTelemetry carries your telemetry. It doesn’t judge whether an output was good. That’s still items 6, 7, and 9’s job.
Score yourself
Give each item 0 (no), 1 (partial) or 2 (yes, verified in the last 90 days). Maximum 50. The scoring is mine, with no benchmark behind it. Use it as a conversation starter, not a certification.
| Score | What it probably means |
|---|---|
| 0 to 20 | You have logs, not observability. Start with group A and item 21. |
| 21 to 35 | You can debug but can’t prove. Groups B and D are where your risk sits. |
| 36 to 45 | Solid. Look for the zeros. |
| 46 to 50 | Rare. Re-verify items 21, 22 and 25; they decay fastest. |
Then apply the gate. A zero on item 16, 17, 21, or 22 means “not ready for agents that write to real systems,” whatever your total says. I chose those four because the cases that hurt most (3 and 5) were failures of exactly these. A team that scores 2 on every tracing item and 0 on item 17 is well informed about a disaster it can’t stop.
If you only have one week
In this order:
- Item 21. Move telemetry out of the agent’s reach.
- Items 16 and 17. Log side effects with identity, and separate prod credentials.
- Item 11. Split your error rate by cause.
- Item 6. Pick one task and check its real outcome.
- Item 25. Run the drill on one past incident, with a stopwatch, and write down detect, acknowledge, and contain times.
That’s seven items. In my reading, every one of the six cases would have gone differently with them in place.
Frequently asked questions
What’s the difference between monitoring and observability for agents? Monitoring tells you whether the system stayed up. Observability lets you reconstruct what the agent did and why. An agent can return HTTP 200 on every call and still pick the wrong tool, so you need both.
Is tracing enough? No. LangChain’s survey shows most production teams trace, yet fewer than half score live traffic with online evals. A trace records what the agent did. It doesn’t say whether that was correct.
Do I need OpenTelemetry? It’s a sensible export format because it keeps you portable, but its GenAI conventions are still in Development, so keep an adapter layer and expect renames.
How do I trace without storing sensitive content? Capture structure (decision patterns, tool names, timings, token counts) by default and make prompt and output content opt-in with a retention policy. Anthropic’s research team describes monitoring decision patterns without accessing conversation contents.
What should I alert on first? Rate-limit headroom, unexpected network egress, repeated denied or failing tool calls, and any destructive action. Then decide who’s paged, and what stops automatically if nobody answers.
How do I know my logs are trustworthy? Ask whether the agent has credentials that could write to them. If yes, treat them as a convenience copy and rely on the tool-layer or network-layer record instead.
What I couldn’t verify
- ROI of observability. I found no rigorous, independent before-and-after study. Vendor customer stories exist, but I haven’t seen their methods, so I left them out. Anthropic’s tracing account is qualitative.
- Replit’s side. See Case 3. No full post-mortem located.
- OpenAI’s September timestamps. The 15-minute, 3-minute, and 2.5-hour figures are OpenAI’s summary as reported by several outlets. The second-level timestamps and the 2 hours 44 minutes total come from one secondary write-up. I didn’t read OpenAI’s original page for that incident.
- The 20% monitoring-cost figure is as reported by Time, not from OpenAI’s documents I read.
- METR’s findings are estimates. It had partial access, a limited window, and says OpenAI could redact non-public details (it states nothing important to its conclusions was redacted beyond what it notes).
- Datadog’s framing. Numbers match the primary report, but the company sells observability, and its sample is its own customers.
How this was built
I searched for primary documents first and checked each quoted figure against them where I could: the Datadog report page and press release, the LangChain report page, METR’s full investigation, the tribunal decision, Anthropic’s engineering posts, and the Gartner release. Where I could only reach secondary coverage (OpenAI’s September and October disclosures, Replit’s side), I say so in the text. Sources were checked on 2 November 2026. Charts were generated from the figures cited in this article; the Datadog split in chart 2 is derived and approximate.
Changelog
- 2 November 2026. First publication. Datadog figures verified against the primary report. Added OpenAI’s 20 September DNS incident and its 29 October monitoring disclosure. Confirmed OpenTelemetry GenAI conventions still at Development status as of late October 2026.
Sources
Primary
- LangChain, State of Agent Engineering: https://www.langchain.com/state-of-agent-engineering
- Datadog, State of AI Engineering 2026 (report page): https://www.datadoghq.com/state-of-ai-engineering/
- Datadog, report PDF: https://lp.datadoghq.com/rs/875-UVY-685/images/ResearchReport-2026StateofAIEngineering.pdf?version=0
- Datadog press release, 21 April 2026: https://investors.datadoghq.com/news-releases/news-release-details/ai-hitting-operational-limits-companies-rush-scale-datadog
- METR, independent investigation of the OpenAI / Hugging Face incident, 26 August 2026: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
- OpenAI, Hugging Face incident technical report (PDF): https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf
- OpenAI, initial incident disclosure: https://openai.com/index/hugging-face-model-evaluation-security-incident/
- Anthropic Engineering, How we built our multi-agent research system (13 June 2025): https://www.anthropic.com/engineering/multi-agent-research-system
- Anthropic Engineering, Demystifying evals for AI agents: https://anthropic.com/engineering/demystifying-evals-for-ai-agents
- Moffatt v. Air Canada, 2024 BCCRT 149 (decision text): https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do
- Gartner press release, 25 June 2025: https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
Secondary and explanatory
- TechCrunch, OpenAI’s official report on the Hugging Face breach (26 August 2026): https://techcrunch.com/2026/08/26/openai-releases-its-official-report-on-the-hugging-face-breach/
- Time, reporting on the investigation and monitoring cost (27 August 2026): https://time.com/article/2026/08/27/openai-hack-hugging-face-investigation/
- Unite.AI, OpenAI’s expanded misalignment monitoring (29 October 2026): https://www.unite.ai/?p=497740
- Magica, OpenAI’s expanded monitoring and 30-minute auto-pause goal: https://magica.com/news/openai-expands-frontier-model-misalignment-monitoring
- FourWeekMBA, OpenAI DNS incident timeline: https://fourweekmba.com/ai-openai-dns-incident-report-12-minute-alert-2-7-hour-stop/
- Scalevise, training resumed 1 October: https://scalevise.com/resources/openai-expands-frontier-model-monitoring/
- The Register, Replit incident (21 July 2025): https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/
- The Register, Replit’s response (22 July 2025): https://www.theregister.com/2025/07/22/replit_saastr_response/
- Fortune, Replit coverage: https://dc.fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure
- LangChain blog, quoting Anthropic’s debugging challenge: https://blog.langchain.com/how-and-when-to-build-multi-agent-systems
- Dash0, OpenTelemetry GenAI conventions explained: https://www.dash0.com/knowledge/opentelemetry-genai-semantic-conventions-explained
- TrueFoundry, OpenTelemetry GenAI conventions (updated 15 October 2026): https://www.truefoundry.com/blog/opentelemetry-genai-semantic-conventions
- Fiddler, OpenTelemetry for AI observability (updated 21 October 2026): https://www.fiddler.ai/blog/opentelemetry-ai-observability-guide
- ClickHouse, OpenTelemetry semantic conventions explained: https://clickhouse.com/resources/engineering/opentelemetry-semantic-conventions.md