Agent Architecture Iterating Towards LLM Reliability with Evaluation Driven Development
Implement evaluation driven development using LangSmith for Dosu.
Implement evaluation driven development using LangSmith for Dosu.
Summary
Dosu, an AI teammate that assists with software maintenance, adopts evaluation‑driven development (EDD) to ensure reliability as it scales to thousands of repositories. The EDD workflow starts with creating a new behavior and a handful of initial evals, then launches it to users, monitors production results, and iterates on failure modes. Human feedback, such as thumbs‑up/down and sentiment, is collected to identify errors, latency issues, and incorrect responses.
Dosu instruments its LLM calls with LangSmith’s @traceable decorator, which streams raw function inputs, rendered prompts, and LLM outputs into the LangSmith UI for auditability. The platform automatically aggregates traces, allowing teams to search for explicit feedback, error incidents, or response time delays, and to attach additional metadata for deeper analysis. By storing human corrections as few‑shot examples, LangSmith can refine its evaluators over time, reducing the need for manual prompt engineering.
The combination of EDD and LangSmith enables Dosu to continuously improve its performance while maintaining a high quality bar across diverse use cases. This approach demonstrates how production‑grade LLM products can be built with confidence through systematic evaluation.
Key changes
- Implements evaluation‑driven development (EDD) workflow: create evals, launch, monitor, iterate.
- Instruments LLM calls with LangSmith @traceable decorator for raw inputs, prompts, and outputs.
- Aggregates traces to search for explicit feedback, errors, latency, and sentiment.
- Stores human corrections as few‑shot examples, automatically feeding them back into evaluators.
- Enables online and offline LLM‑as‑judge evaluations with minimal prompt engineering.
- Provides a UI for attaching metadata and visualizing failure modes.
- Supports continuous improvement by adding new examples to evaluation datasets.
- Demonstrates scalability to thousands of repositories with automated dataset collection.