LangSmith Tutorials & How‑Tos Promptim: an experimental library for prompt optimization
Use Promptim to automatically optimize a single prompt against a LangSmith dataset, providing evaluators and optional human feedback.
Install Promptim via pip, create a LangSmith dataset, define a prompt and evaluator, then run Promptim to optimize the prompt automatically.
Summary
Promptim, released on 13 November 2024, is an experimental open‑source library that automates prompt optimization for specific tasks. It integrates with LangSmith for dataset management, prompt storage, and optional human feedback via Annotation Queues. The core algorithm runs a baseline prompt over a dev split, then iterates over training examples, scores them, and passes the results to a metaprompt that suggests prompt changes. If the updated prompt improves metrics on the dev split, it is retained; otherwise the original is kept. The process repeats up to a configurable number of iterations. Promptim currently focuses on optimizing a single prompt, unlike DSPy which targets entire agent systems. Future plans include dynamic few‑shot prompting, UI integration, additional optimization methods, and optimizing LangGraph graphs.
The library is installed via pip and requires a LangSmith project. It supports custom evaluators and optional human feedback, making it a flexible tool for systematic prompt improvement. The article compares Promptim to DSPy, noting differences in scope, human involvement, and focus on prompt rewriting.
Key takeaways include the optimization loop, human feedback integration, and the focus on single‑prompt optimization.
Key changes
- Promptim library released 13 November 2024
- Integrates with LangSmith for dataset and prompt management
- Runs optimization loop: baseline, train, metaprompt, evaluate, retain if improved
- Supports human feedback via Annotation Queues
- Currently optimizes single prompt, not entire agent
- Compared to DSPy, focuses on prompt rewriting
- Future plans: dynamic few‑shot, LangSmith UI integration, more optimization methods
- Uses metaprompt to suggest prompt changes