Deep Agents Agent Architecture Tutorials & How‑Tos Evaluating Deep Agents: Our Learnings
Document the evaluation patterns for future reference.
Document the evaluation patterns for future reference.
Summary
In December 2025 LangChain shipped four new Deep Agents applications—DeepAgents CLI, LangSmith Assist, Personal Email Assistant, and Agent Builder—each built on the Deep Agents harness. To support these, the team added bespoke evaluation logic for every datapoint, because Deep Agents require custom success criteria beyond final output. The article explains four key evaluation patterns: single‑step evals that validate a single tool call, full‑turn evals that assess the entire agent trajectory, multi‑turn evals that simulate realistic user interactions, and environment setup that ensures a clean, reproducible test sandbox. It also details how LangSmith’s Pytest and Vitest integrations allow developers to log inputs, outputs, tool calls, and LLM‑as‑judge feedback directly into experiments. The author provides code snippets for asserting that a calendar‑scheduling agent correctly edits a memories.md file and confirms the update in the final message. Additionally, the article discusses using LangGraph’s streaming to interrupt after a tool node for single‑step tests, and how to conditionally run subsequent turns based on earlier outputs. The piece emphasizes that Deep Agents are stateful, so each test must reset the environment, for example by creating a temporary directory or using Docker containers. Finally, it recommends mocking external API requests with tools like vcr or a Hono proxy to keep tests fast and deterministic.
Key changes
- Added bespoke evaluation logic for each Deep Agents datapoint, requiring custom success criteria.
- Introduced single‑step evals that validate a single tool call and save tokens.
- Added full‑turn evals that assess the entire agent trajectory, final response, and state changes.
- Implemented multi‑turn evals that simulate realistic user interactions with conditional logic.
- Integrated LangSmith’s Pytest and Vitest to log inputs, outputs, tool calls, and LLM‑as‑judge feedback.
- Emphasized environment isolation by resetting a temporary directory or Docker container for each test.