LangSmith Benchmarking Query Analysis in High Cardinality Situations
Implement the post‑LLM embedding selection approach for high‑cardinality categorical data in your query analysis pipeline.
Implement the post‑LLM embedding selection approach for high‑cardinality categorical data in your query analysis pipeline.
Summary
High‑cardinality categorical values, such as author names, pose a challenge for LLM query analysis because the model must map user aliases to valid names. The benchmark dataset contains 10,000 author names and 10,000 random aliases, and the system must parse queries like "what are books by Harry Chase about aliens?" into a structured format. A baseline that asks the LLM to analyze the query without any knowledge of valid names returns 0 % correct. Context stuffing of all 10,000 names yields only 26 % correct with GPT‑4, 5 s latency, and $8.44 cost. Pre‑LLM filtering using embedding similarity achieves 57 % correct with GPT‑3.5, 0.76 s, $0.002, while n‑gram similarity reaches 65 % correct, 0.57 s, $0.002. Post‑LLM selection with embedding similarity tops the chart at 83 % correct, 0.66 s, $0.001, and n‑gram similarity achieves 74 % correct, 0.48 s, $0.001. Only a single LLM call is allowed to mimic real‑world latency constraints. The article was published on March 15, 2024 and demonstrates that post‑LLM embedding selection is the most effective approach for high‑cardinality categorical data.
Key changes
- Baseline LLM query analysis without valid name knowledge returns 0 % correct
- Context stuffing of all 10,000 names yields 26 % correct with GPT‑4, 5 s latency, $8.44 cost
- Pre‑LLM filtering via embedding similarity achieves 57 % correct with GPT‑3.5, 0.76 s, $0.002
- Pre‑LLM filtering via n‑gram similarity achieves 65 % correct, 0.57 s, $0.002
- Post‑LLM selection with embedding similarity achieves 83 % correct, 0.66 s, $0.001
- Post‑LLM selection with n‑gram similarity achieves 74 % correct, 0.48 s, $0.001
- Only a single LLM call is allowed to mimic real‑world latency constraints
- Dataset comprised 10,000 author names and 10,000 random aliases