Observability & Evals Improving Agents is a Data Mining Problem

The talk emphasizes that traces are the currency of continual learning for agents, and that mining them at scale unlocks the most leverage. LangSmith Engine is introduced as an automated system that reads every trace, looks for signals, generates code fixes, creates evals, commits important information to memory, and drives an improvement loop. Open models are highlighted as cost‑efficient options for processing traces, with a Trace Judge model that outperforms closed models on narrow tasks. Harness engineering is presented as a data‑driven loop: evaluate with a suite, read traces, cluster failures, engineer a targeted change, and re‑evaluate, repeating until the win is stable. The talk shows that harness tuning can bring open models close to frontier performance without touching weights, and that specialized agents are needed to handle the volume of trace data. The key takeaway is that trace mining, coupled with harness engineering and cost‑efficient open models, is the path to scalable, continual agent improvement.

This content is a synthesis of a talk and a blog post, aimed at teams looking to build robust continual learning pipelines for their agents.