Harrison's In the Loop Observability & Evals Aligning LLM-as-a-Judge with Human Preferences
Enable self‑improving LLM‑as‑judge evaluators in LangSmith.
Enable self‑improving LLM‑as‑judge evaluators in LangSmith.
Summary
The article discusses the challenges of aligning LLM‑as‑judge evaluators with human preferences, noting that prompt engineering is often required to get accurate scores. LangSmith introduces a self‑improving evaluator that stores human corrections as few‑shot examples, automatically feeding them back into the prompt for future runs. This approach eliminates the need for extensive manual prompt adjustments.
The self‑improving cycle works online or offline, collecting feedback on correctness, relevance, or other criteria, and storing corrections with optional explanations. Subsequent evaluator runs incorporate stored examples, leveraging the LLM’s few‑shot learning to better match team standards. The system also supports aligning with human preferences over time, reducing prompt engineering effort and improving evaluation reliability.
By integrating human corrections directly into the evaluation pipeline, teams can focus on reviewing and correcting outputs rather than crafting new prompts. The solution demonstrates how automated evaluation can be continuously refined with real‑world feedback.
Key changes
- Self‑improving evaluators store human corrections as few‑shot examples automatically.
- LLM‑as‑judge evaluator uses prompt with examples to score outputs for correctness, relevance, etc.
- Feedback corrections are stored with optional explanations for future runs.
- Subsequent evaluator runs incorporate stored examples, reducing prompt engineering effort.
- Works both online and offline, collecting feedback on various criteria.
- Aligns evaluator outputs with human preferences over time via few‑shot learning.
- Supports continuous refinement of evaluation metrics without manual prompt adjustments.
- Improves reliability of LLM‑as‑judge evaluations by integrating real‑world feedback.