How to Trace an AI Agent Step by Step (With Real Examples)

By Tom Morgan · Reviewed 2 November 2026 · 15-minute read

Rebuilt on 2 November 2026. The first version of this post pinned a set of Replit figures on the wrong source, linked to OpenTelemetry pages that have since turned into redirects, and had no trace you could reproduce yourself. This version fixes all three, re-checks the key numbers against the primary reports, and adds a script you can run in a minute.

I built an AI agent whose every span came back green. Six HTTP attempts, five tool calls, one rate limit that cleared on retry. No exception, no timeout, nothing an alert would care about. It still failed the customer.

Let me be plain about what that agent is. It’s a scripted stand-in with no real language model inside, wired to the real OpenTelemetry SDK, with one planted bug. The full script and the raw output are further down, because the agent isn’t the point. The point is what a trace showed that an error dashboard wouldn’t: the shape of the run. The same tool called with the same arguments five times, with the input growing on every turn.

Once you can see that shape, you can write it down as a test that fails a build. That is most of what “tracing an agent” means in practice, and it’s the thread through everything below: what to record, five documented cases where it mattered (or would have), a method you can follow, and the places where tracing won’t save you.

I’ll tag claims as ESTABLISHED (primary source, checked), PROBABLE (well supported, with a gap) or SPECULATIVE (my reading).

---

Three words, then a warning about the standard

A run is one task from request to final answer. A span is one timed operation inside it: a model call, a tool call, a handoff. A trace is all the spans of one run, nested, so you can see what called what and where the time went.

OpenTelemetry has agreed names for agent work: invoke_agent for a whole run, chat for a model call, execute_tool for a tool call, with the real HTTP call as a child span underneath (Dash0). That last child is worth the effort. The tool span alone tells you a tool ran. The HTTP span under it tells you it was throttled and retried (Nango).

Span	What it represents	Record at minimum
invoke_agent	One full agent run	Agent name, prompt/model version, final outcome
chat	One model call (the “what next?” moment)	Model, input and output tokens
execute_tool	One tool call	Tool name, arguments, result or error
Child HTTP span	The real API call, auth, retries	Status code, attempt number

Here’s the warning. These conventions are not stable, and as of this review, they’ve been moving.

• On 12 June 2026, in core v1.42.0, OpenTelemetry moved its GenAI conventions into a dedicated repository, semantic-conventions-genai. The old pages on opentelemetry.io are now redirect stubs. As of Dash0’s 14 September update, the new repository had published no tagged release, and the only Stable attributes on a GenAI span were error.type, server.address and server.port, all inherited from the core conventions (Dash0). ESTABLISHED as of that date; I haven’t re-verified the repository’s release state since.
• ClickHouse’s explainer reports three attribute renames in a single 2026 release cycle (ClickHouse).
• One vendor explainer says libraries that emitted older names support an opt-in value, gen_ai_latest_experimental, for OTEL_SEMCONV_STABILITY_OPT_IN (Lyzr). Check your own library’s docs before relying on it.

What I’d do about it: pin versions, and keep every gen_ai.* attribute name in one small module so a rename is a one-file change. That’s my advice, not a standard.

---

The experiment: green spans, broken agent

The setup. A support agent is asked whether a refund was issued. It has one tool, lookup_order. The tool returns the refund state under the key refund_state. The planted bug: the buggy agent looks for a key called status, which never appears, so it never recognises that the task is done. It keeps asking until it gives up. Berkeley’s failure taxonomy has a name for this class of problem: unaware of termination conditions (more on that below).

Everything else is real: the OpenTelemetry Python SDK (version 1.45.2 on Python 3.12.3), real nested spans, real timings, a simulated 429 on the first call of each run that the code retries with backoff. The “model” is a deterministic stub, so you’ll get the same shape every time you run it. I did not test a live LLM, and I’m not claiming this bug is common. I’m showing what the trace reveals when it happens.

Here’s the buggy run as a waterfall, drawn from the real span timings:

And the numbers, buggy run against the fixed one (the fix reads the right key and stops):

Measure	Buggy run	Fixed run
Spans in the trace	18	6
Tool calls	5	1
Input tokens (stub’s chars/4 estimate, summed)	306	38
Wall-clock for the run	155 ms	71 ms
Final answer	“I could not confirm your refund.”	“Your refund was issued.”

Look at what’s not in that table: an error. Every tool call ended in HTTP 200. The only blip was one 429 that the retry absorbed, one failed attempt out of six. A dashboard that counts failures would call this a healthy run. PROBABLE (it’s my reasoning about what such a dashboard measures, not something I tested).

Three things gave it away, and none of them is an error:

1. Repetition. Same tool, same arguments, five times.
2. Growth. Input tokens climb by the same amount every turn, because the history keeps being re-sent with one more identical result appended.
3. Width. 18 spans for a task that needs 6.

Multiply that toy by your real prompt sizes, and a loop like this gets expensive quietly. The toy’s 8× gap in input tokens doesn’t predict your multiplier; your prompts decide that.

From a trace to a test

Here’s the move I think matters most. Don’t assert on the answer text. Assert on a property of the trace:

python

def assert_trace_healthy(spans, max_tool_calls=3, max_repeats=1):
    m = summarize(spans)  # counts tool calls, identical repeats, tokens
    assert m["tool_calls"] <= max_tool_calls, f"too many tool calls: {m['tool_calls']}"
    assert m["identical_repeat_tool_calls"] <= max_repeats, \
        f"same tool+args repeated {m['identical_repeat_tool_calls']}x"

The buggy run fails it (“too many tool calls: 5”). The fixed run passes. The thresholds are arbitrary numbers I picked for this toy; pick yours from your own healthy traces.

Datadog’s report makes a related recommendation: put budgets on agent loops so they terminate after a maximum number of calls or tokens, to stop runaway loops from exhausting capacity (Datadog). A budget guards the production run. The trace assertion guards the build. You want both. <details> <summary><strong>Full script and raw output (run it yourself)</strong></summary>

Install with pip install opentelemetry-sdk, save as trace_demo.py, run python trace_demo.py. Timings will differ a little on your machine; the shape won’t.

python

"""
Tracing demo: a *scripted* support agent (no real LLM, no network) instrumented
with the real OpenTelemetry SDK. The spans, timings and tree below are genuine
OTel output; the "model" is a deterministic stub so the run is reproducible.

Run:  pip install opentelemetry-sdk && python trace_demo.py
"""
import time
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-agent-demo")


def tok(text: str) -> int:           # crude chars/4 proxy, not a real tokenizer
    return max(1, len(text) // 4)


class RateLimited(Exception):
    pass


_calls = {"n": 0}


def lookup_order(order_id: str) -> dict:
    """Fake tool. First call of each run hits a simulated 429, then succeeds."""
    _calls["n"] += 1
    if _calls["n"] == 1:
        raise RateLimited("429 Too Many Requests")
    time.sleep(0.02)
    # NOTE: the API returns state under "refund_state", not "status".
    return {"order_id": order_id, "refund_state": "refunded"}


def call_tool_with_retry(name: str, args: dict) -> dict:
    with tracer.start_as_current_span(f"execute_tool {name}") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", name)
        span.set_attribute("app.tool.args", str(args))
        for attempt in range(3):
            with tracer.start_as_current_span("http POST orders-api") as http:
                http.set_attribute("app.attempt", attempt + 1)
                try:
                    result = lookup_order(**args)
                    http.set_attribute("http.response.status_code", 200)
                    span.set_attribute("app.tool.result", str(result))
                    return result
                except RateLimited:
                    http.set_attribute("http.response.status_code", 429)
                    http.set_attribute("error.type", "429")
                    time.sleep(0.05)      # backoff
        raise RuntimeError("tool failed after retries")


def stub_model(history: list, fixed: bool) -> dict:
    """Deterministic stand-in for an LLM deciding what to do next."""
    last = history[-1] if history else ""
    if fixed:
        # fixed agent reads the real field name and stops
        if "refund_state" in last and "refunded" in last:
            return {"final": "Your refund was issued."}
    else:
        # buggy agent looks for a field that never appears ("status"),
        # so it never recognises that the task is done
        if "'status': 'refunded'" in last:
            return {"final": "Your refund was issued."}
        if len(history) >= 6:            # eventually gives up
            return {"final": "I could not confirm your refund."}
    return {"tool": "lookup_order", "args": {"order_id": "A-1042"}}


def run_agent(fixed: bool, max_steps: int = 8) -> str:
    _calls["n"] = 0
    history = ["User: Was my refund for order A-1042 issued?"]
    with tracer.start_as_current_span("invoke_agent support_bot") as run:
        run.set_attribute("gen_ai.operation.name", "invoke_agent")
        run.set_attribute("gen_ai.agent.name", "support_bot")
        run.set_attribute("app.prompt_version", "fixed" if fixed else "buggy")
        for step in range(max_steps):
            with tracer.start_as_current_span("chat stub-model") as chat:
                chat.set_attribute("gen_ai.operation.name", "chat")
                chat.set_attribute("gen_ai.usage.input_tokens", tok(" ".join(history)))
                decision = stub_model(history, fixed)
            if "final" in decision:
                history.append(decision["final"])
                run.set_attribute("app.final_answer", decision["final"])
                return decision["final"]
            result = call_tool_with_retry(decision["tool"], decision["args"])
            history.append(f"Tool result: {result}")
        run.set_attribute("app.final_answer", "STEP CAP HIT")
        return "STEP CAP HIT"


def render(spans):
    by_parent = {}
    for s in spans:
        pid = s.parent.span_id if s.parent else None
        by_parent.setdefault(pid, []).append(s)
    for lst in by_parent.values():
        lst.sort(key=lambda s: s.start_time)
    t0 = min(s.start_time for s in spans)

    def walk(pid, depth):
        for s in by_parent.get(pid, []):
            ms = (s.end_time - s.start_time) / 1e6
            at = (s.start_time - t0) / 1e6
            a = s.attributes
            extra = []
            if "gen_ai.usage.input_tokens" in a:
                extra.append(f"in_tok={a['gen_ai.usage.input_tokens']}")
            if "http.response.status_code" in a:
                extra.append(f"http={a['http.response.status_code']}")
            if "app.final_answer" in a:
                extra.append(f"answer={a['app.final_answer']!r}")
            print(f"{'  ' * depth}{s.name:<26} +{at:6.0f}ms  {ms:6.1f}ms  {' '.join(extra)}")
            walk(s.context.span_id, depth + 1)

    walk(None, 0)


def summarize(spans):
    tools = [s for s in spans if s.name.startswith("execute_tool")]
    chats = [s for s in spans if s.name.startswith("chat")]
    https = [s for s in spans if s.name.startswith("http")]
    args = [s.attributes.get("app.tool.args") for s in tools]
    repeats = len(args) - len(set(args))
    run = [s for s in spans if s.name.startswith("invoke_agent")][0]
    return {
        "spans": len(spans), "model_calls": len(chats), "tool_calls": len(tools),
        "http_attempts": len(https), "identical_repeat_tool_calls": repeats,
        "input_tokens_total": sum(c.attributes["gen_ai.usage.input_tokens"] for c in chats),
        "run_ms": round((run.end_time - run.start_time) / 1e6),
    }


def assert_trace_healthy(spans, max_tool_calls=3, max_repeats=1):
    """The regression test: a property of the TRACE, not of the answer text."""
    m = summarize(spans)
    assert m["tool_calls"] <= max_tool_calls, f"too many tool calls: {m['tool_calls']}"
    assert m["identical_repeat_tool_calls"] <= max_repeats, \
        f"same tool+args repeated {m['identical_repeat_tool_calls']}x"


if __name__ == "__main__":
    for label, fixed in (("BUGGY RUN", False), ("FIXED RUN", True)):
        exporter.clear()
        answer = run_agent(fixed)
        spans = exporter.get_finished_spans()
        print(f"\n===== {label} -> {answer!r} =====")
        render(spans)
        print("summary:", summarize(spans))
        try:
            assert_trace_healthy(spans)
            print("trace check: PASS")
        except AssertionError as e:
            print("trace check: FAIL ->", e)

Raw output from my run:

text

===== BUGGY RUN -> 'I could not confirm your refund.' =====
invoke_agent support_bot   +     0ms   154.6ms  answer='I could not confirm your refund.'
  chat stub-model            +     0ms     0.1ms  in_tok=11
  execute_tool lookup_order  +     0ms    70.8ms  
    http POST orders-api       +     0ms    50.3ms  http=429
    http POST orders-api       +    51ms    20.3ms  http=200
  chat stub-model            +    71ms     0.1ms  in_tok=27
  execute_tool lookup_order  +    71ms    20.4ms  
    http POST orders-api       +    71ms    20.4ms  http=200
  chat stub-model            +    92ms     0.0ms  in_tok=43
  execute_tool lookup_order  +    92ms    20.4ms  
    http POST orders-api       +    92ms    20.4ms  http=200
  chat stub-model            +   112ms     0.0ms  in_tok=59
  execute_tool lookup_order  +   112ms    20.5ms  
    http POST orders-api       +   112ms    20.4ms  http=200
  chat stub-model            +   133ms     0.0ms  in_tok=75
  execute_tool lookup_order  +   133ms    21.5ms  
    http POST orders-api       +   133ms    21.4ms  http=200
  chat stub-model            +   155ms     0.0ms  in_tok=91
summary: {'spans': 18, 'model_calls': 6, 'tool_calls': 5, 'http_attempts': 6, 'identical_repeat_tool_calls': 4, 'input_tokens_total': 306, 'run_ms': 155}
trace check: FAIL -> too many tool calls: 5

===== FIXED RUN -> 'Your refund was issued.' =====
invoke_agent support_bot   +     0ms    71.4ms  answer='Your refund was issued.'
  chat stub-model            +     0ms     0.0ms  in_tok=11
  execute_tool lookup_order  +     0ms    71.2ms  
    http POST orders-api       +     0ms    50.6ms  http=429
    http POST orders-api       +    51ms    20.3ms  http=200
  chat stub-model            +    71ms     0.0ms  in_tok=27
summary: {'spans': 6, 'model_calls': 2, 'tool_calls': 1, 'http_attempts': 2, 'identical_repeat_tool_calls': 0, 'input_tokens_total': 38, 'run_ms': 71}
trace check: PASS

</details>

---

Five documented cases

I picked cases with public, checkable sources. Some are about tracing directly. Others are failures a trace would have helped you see. I’ll say which is which.

1. Anthropic’s Research feature: tracing fixed the “why”, humans found the “wrong”

Users kept reporting that agents were “not finding obvious information,” and the team couldn’t tell why: bad queries, poor sources, or failing tools. Adding full production tracing let them diagnose it. They monitor decision patterns and interaction structures without reading the contents of individual conversations, to protect privacy. Early agents also made odd mistakes, like spawning 50 subagents for a simple query, which led to explicit effort-scaling rules in the prompts (Anthropic). ESTABLISHED.

The detail people skip: the problem where agents preferred SEO-optimized content farms over authoritative sources was noticed by human testers, per the same post. Tracing told them what each agent did. People told them it was wrong. Anthropic also reports agents using about 4× the tokens of a chat and multi-agent systems about 15×, which is why a runaway loop costs real money there.

Take: a trace answers “what happened?” It can’t answer “was that good?”

2. Berkeley’s MAST study: 1,642 failed runs, sorted

UC Berkeley researchers annotated 1,642 execution traces from seven multi-agent frameworks and built a taxonomy of 14 failure modes in three groups (Cemri et al., NeurIPS 2025). From the paper’s figure, roughly 44% of the categorized failures were system-design issues, 32% inter-agent misalignment, and 24% task-verification problems. One named mode, not knowing when the task is finished, was about 12% on its own (alphaXiv summary). That’s the exact class of bug in my experiment.

The biggest bucket is how the system was designed, not the model being dim. The authors say well-designed systems can improve on the same models, and that checking only at the final stage isn’t enough. Their own limit: the taxonomy doesn’t claim to cover every failure, and the traces come from mostly open-source frameworks on coding and math tasks, not your production agent. ESTABLISHED for the study, PROBABLE for how far it transfers.

3. Who&When: can a model read the trace for you? Better than in 2025, still not a replacement

The ICML 2025 paper built failure logs from 127 multi-agent systems and tested whether an LLM could name the responsible agent and the decisive step. The best method found the agent 53.5% of the time but the step only 14.2%, and some methods did worse than random (Zhang et al.).

I checked whether that’s stale, since it’s an eighteen-month-old result. It is, partly. A February 2026 preprint (CHIEF) reports step-level accuracy of 29.31% on the hand-crafted subset and 52.00% on the algorithm-generated one (Wang et al.). A May 2026 preprint (FALAT) reports 29.1% and 46.0% in its best configurations (Rafi et al.). These are the authors’ own numbers on a research benchmark, with different underlying models, so don’t compare them head to head.

Take: automated diagnosis is improving fast and is useful for triage. On the hardest subset the best reported step-level figure is still under a third. I’d keep a human reading the runs that matter. PROBABLE.

4. Replit and SaaStr: the trace shows the action; the fix is a gate

In July 2025, SaaStr founder Jason Lemkin reported that Replit’s agent deleted a production database during a code freeze, on day nine of a 12-day experiment (eWeek). Reports put the loss at records for over 1,200 executives and roughly 1,100 to 1,200 companies; outlets differ on the second figure (YourStory). Replit’s CEO called it “unacceptable and should never be possible,” and the company moved to separate development from production databases; at the time one database had served preview, testing, and production (heise).

What I can’t tell you is what Replit’s internal traces showed. Nobody outside Replit has seen them. What I can say is what a trace would capture: the tool call that ran the destructive command, with its arguments. And per one security-community write-up, the “freeze” was an instruction in the prompt that nothing technically enforced (NHIMG).

Take: if a trace shows an instruction was ignored, the fix is a permission gate, not louder wording. That lesson is PROBABLE. The incident itself rests on a user’s account plus the company’s partial confirmation, and it’s fifteen months old.

5. Datadog’s telemetry: the boring failure is usually the real one

Datadog analyzed LLM telemetry from more than a thousand customers. In February 2026, 5% of LLM call spans reported an error, and 60% of those were exceeded rate limits. In March, the error rate fell to 2%, but rate limits were still almost a third of failures, nearly 8.4 million of them. It also found 59% of agentic requests made only a single service call, so most production agents are still monoliths (Datadog). ESTABLISHED, with Datadog’s own caveat: the data comes from its customers, “a large but imperfect sample,” and Datadog sells monitoring.

Take: before you blame the model’s reasoning, look at the gaps. In my waterfall, the first tool call spent 50 of its 71 milliseconds waiting out a 429. A long pause before a span is often a retry, not a thought.

---

The method, in order

This sequence is genuinely ordered, so numbering is honest. It’s my synthesis of the sources and my own experiment, not a benchmarked standard.

1. Write the question before you instrument. “Why did it fail?” is too broad. Better: “Which tool call preceded each wrong answer?” or “Where does the time go?” The question decides which attributes you need.

2. Draw three layers: run, model call, tool call. One invoke_agent parent with chat and execute_tool children. A sub-agent is another invoke_agent nested under whatever started it. Hang the HTTP attempt under the tool span.

invoke_agent "support_bot"
├── chat
├── execute_tool "lookup_order"
│   ├── http POST orders-api  → 429
│   └── http POST orders-api  → 200
└── chat

3. Add the spans. The skeleton is below; the script above has the whole thing. If you use the OpenAI Agents SDK, tracing is on by default and covers agent runs, generations, tool calls, guardrails, and handoffs (OpenAI docs). Check your framework before writing custom spans.

python

with tracer.start_as_current_span("invoke_agent support_bot") as run:
    run.set_attribute("gen_ai.operation.name", "invoke_agent")
    run.set_attribute("app.prompt_version", "2026-11-02")   # my own attribute
    with tracer.start_as_current_span("execute_tool lookup_order") as tool:
        tool.set_attribute("gen_ai.tool.name", "lookup_order")

4. Tag every run so you can slice later. Prompt version, model version, session ID, enabled tools. The first time you change a prompt and quality shifts, you’ll want before-and-after traces. Without the tag, you can’t compare.

5. Decide what you’re willing to store. See the privacy note below. Names, durations, and token counts are cheap and safe. Prompt and argument text needs a policy decision.

6. Read shape before content. (My heuristic.) Open a bad trace and look in this order: width (did a simple request fan out?), repeats (same tool, same arguments?), gaps (long pauses?), growth (input tokens climbing?), last good step. Only then read prompts and outputs.

7. Turn each diagnosed failure into a trace assertion. As in the experiment. Anthropic’s advice on evaluation is to start small: they began with about 20 queries and could see big effects from a handful of cases (Anthropic).

What the trace shows	Likely cause	First thing to try
Same tool, same arguments, repeated	Agent can’t tell it’s done (MAST names this failure)	Explicit stop condition plus a step cap
Many subagents for a trivial query	No effort-scaling rule (Anthropic saw this early)	Tell the lead agent how much effort a query deserves
Long gap, then success	Rate limit or backoff (Datadog’s top error type)	Read the child HTTP spans; add queueing
Tool output present, next answer ignores it	Handoff or context problem (my reading)	Check what the next chat span received
Destructive call despite “don’t” in the prompt	Instruction not enforced (Replit, inferred)	A permission gate, not stronger wording

---

The number that should bother you

LangChain surveyed 1,340 people between 18 November and 2 December 2025. Of those, 89% have some observability for their agents and 62% have detailed step-level tracing; among teams with agents in production, 94% and 71.5%. But only 52.4% run offline evaluations and 37.3% run online evaluations (LangChain).

Read the sample before you quote this. LangChain sells observability and evaluation tooling. The survey was public and open for two weeks. Per its methodology, 63% of respondents were in technology, and 49% worked at companies under 100 people. So it describes people who answer LangChain surveys. The direction (tracing runs ahead of evaluation) fits what I saw in my experiment, where the trace could show the loop but couldn’t say whether “I could not confirm your refund” was acceptable. The exact percentages I’d hold loosely. PROBABLE.

One more number, from Berkeley’s Measuring Agents in Production study (306 practitioners, 20 interviews, 86 deployed systems): 68% of deployed agents execute at most ten steps before a human intervenes, and 74% rely mainly on human evaluation (Pan et al.). SPECULATIVE, my inference: most production agents are short enough that a person can read the whole trace. The hard cases are the long autonomous ones, which is where automated diagnosis also struggles.

---

What tracing costs, and what it can leak

Privacy. The OpenAI Agents SDK includes potentially sensitive data in traces by default (trace_include_sensitive_data defaults to true), and tracing isn’t available to organizations under zero data retention (OpenAI docs). Anthropic describes watching decision patterns without reading conversation content. Pick on purpose: names, durations and token counts always; prompt and argument text only where your policy and contracts allow.

Storage and tokens. Datadog’s data shows 69% of input tokens in customer traces were system prompts, estimated from character counts divided by four, and only 28% of LLM call spans on caching-capable models showed any cached-read tokens (Datadog). If you log full prompts on every call, trace size follows that. I haven’t found a credible public figure for tracing overhead, so measure your own. My suggestion, not a standard: keep every failed or flagged run; sample the clean ones.

---

Where this advice breaks

• My experiment is a toy. A stub model with a planted bug demonstrates what a trace can show. It says nothing about how often real agents loop like this.
• Clean traces can still be wrong answers. See Anthropic’s SEO-farm finding.
• The seven steps are a synthesis. I haven’t benchmarked them against a controlled set of agents and can’t give you a debugging time-saving.
• Several statistics come from vendors (LangChain, Datadog, Nango, Dash0, Lyzr). They’re the best public data I found, and they have reasons to want you to buy monitoring.
• MAST and Who&When are research datasets, built mostly on open-source multi-agent frameworks and benchmark tasks. A single-agent support bot may fail very differently, and the 2026 attribution results are preprints.
• The GenAI attribute names may have moved since I checked. Treat the code as a pattern, not a spec.

---

Your first thirty minutes

Pick your ugliest recent failure. Find its trace, or add the three span types and reproduce it. Find the last good step. Then write one assertion about the shape of that run and put it in your test suite. That’s it. One failure, fully explained and pinned down, is worth more than a dashboard of a thousand green ones.

My unpopular opinion: don’t buy a tracing platform until you’ve written one trace assertion by hand. You’ll know what to ask the vendor for.

A trace won’t tell you whether your agent is right. It will tell you what it did, in what order, at what cost, and that’s the only honest place to start arguing about whether it was right.

---

Questions people ask

What’s the difference between a log and a trace? A log is a list of events. A trace links them into a parent-child tree for one run, with timing, so you can see what called what and where time went.

Do I need OpenTelemetry? No. Several frameworks trace by default. The advantage of the standard names is portability, with the caveat above that the GenAI conventions are still in Development and have been renamed more than once.

Can an LLM read my traces and find the bug? As a triage assistant, increasingly. As an unattended judge, not yet. Newer research reports step-level accuracy in the roughly 29% to 52% range on a benchmark; the original 2025 best was 14.2%.

Is it safe to log prompts and tool arguments? Only if your privacy policy and contracts allow it. Some SDKs include this data by default, so check the setting rather than assume.

Should I store every trace? I can’t answer that with public data. My suggestion: keep all failed or flagged runs, sample the clean ones, and measure your own overhead.

Can the demo run against a real model? Yes, but I haven’t done it here. Replace stub_model with a real call and keep the span structure. Expect noisier numbers and no guaranteed loop.

---

Sources and how I checked them

I re-read the primary pages on 2 November 2026 for Anthropic, LangChain, and Datadog and confirmed the figures quoted. For the arXiv papers, I relied on the abstracts and published summaries, not a full read of each PDF; the MAST percentages come from the paper’s figure text.

• Anthropic, How we built our multi-agent research system
• LangChain, State of Agent Engineering
• Datadog, State of AI Engineering (the page I verified is the Korean-locale URL; English and Japanese versions are linked from its language switcher)
• Dash0, OpenTelemetry GenAI semantic conventions explained
• ClickHouse, OpenTelemetry semantic conventions explained
• Lyzr, OpenTelemetry GenAI semantic conventions
• Nango, OpenTelemetry for AI agent observability
• OpenAI, Agents SDK tracing
• Cemri et al., Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025) · alphaXiv overview
• Zhang et al., Which Agent Causes Task Failures and When? (ICML 2025)
• Wang et al., From Flat Logs to Causal Graphs (CHIEF) (preprint, Feb 2026)
• Rafi et al., FALAT (preprint, May 2026)
• Pan et al., Measuring Agents in Production
• Replit incident: eWeek · YourStory · heise · NHIMG · AI Incident Database

---

About the author: Tom Morgan writes about AI systems from published research and documentation. This piece combines public sources with one small experiment I ran myself (a scripted agent, not a live model). I haven’t used these tools in a client production environment, and I may have missed newer OpenTelemetry conventions released after the dates cited—no sponsorship; no affiliation with any vendor named here.