Conceptual Guide Agent observability powers agent evaluation
Implement agent observability by capturing runs, traces, and threads in LangSmith to enable systematic evaluation of agent reasoning.
Instrument your agent to emit runs, traces, and threads into LangSmith, then use those traces to write evaluation assertions at run, trace, and thread levels.
Summary
Agent observability is essential for evaluating and debugging AI agents, which differ from traditional software due to non‑deterministic reasoning and long execution loops. The article defines three core primitives: runs (single LLM call with prompt and context), traces (complete agent execution linking runs), and threads (multi‑turn conversations grouping traces). These primitives allow evaluation at run, trace, and thread levels, enabling systematic assertions on tool calls, final responses, and state. Production traces become real‑world test cases, driving offline evaluation and continuous improvement. The article contrasts agent observability with software observability, noting that traditional stack traces are insufficient for agents. LangSmith provides tools for capturing runs, traces, and threads, and for writing evaluation assertions.
Key takeaways include the importance of capturing detailed reasoning context, the need for evaluation at multiple granularities, and the role of LangSmith in providing observability and evaluation infrastructure.
Key changes
- Defines runs, traces, and threads as core primitives
- Runs capture single LLM call with prompt and context
- Traces link runs to show full agent execution
- Threads group multiple traces for multi‑turn conversations
- Observability enables evaluation at run, trace, and thread levels
- Production traces become test cases for offline evaluation
- Agent evaluation differs from software testing due to non‑determinism
- LangSmith provides tools for tracing and evaluation