Case Studies LangSmith Improving Memory Retrieval: How New Computer achieved 50% higher recall

New Computer’s Dot agent uses an agentic memory system that dynamically creates documents enriched with optional meta‑fields such as "status" and "datetime". By leveraging LangSmith’s experiments UI, the team labeled synthetic user queries and measured recall, precision, and F1, achieving a 50 % higher recall and 40 % higher precision over a baseline dynamic memory retrieval implementation. Synthetic users were generated with LLM‑created backstories to preserve privacy while providing a realistic evaluation set. The LangSmith dataset stores each query alongside the full memory set, enabling rapid iteration on retrieval methods like semantic search, keyword BM25, and meta‑field filtering. The experiments UI visualizes metrics and allows real‑time prompt adjustments via an integrated playground. After the improvements, 45 % of new users converted to the paid tier once they hit the free message limit. The system’s dynamic conversational prompts incorporate tool usage and contextual instructions to produce accurate responses. The article was published on July 2, 2024 and highlights how iterative experimentation can boost memory retrieval performance.