EnterpriseRAG‑Bench: Open Benchmark for Testing RAG on Realistic Company‑Internal Data
Download the EnterpriseRAG‑Bench repo, run the evaluation harness against your RAG system, and compare results to the baseline leaderboard.
Download the EnterpriseRAG‑Bench repo, run the evaluation harness against your RAG system, and compare results to the baseline leaderboard.
Summary
The authors have released EnterpriseRAG‑Bench, a synthetic corpus of 500 k documents that mimics a real company called Redwood Inference, covering Slack, Gmail, Linear, Google Drive, HubSpot, Fireflies, GitHub, Jira, and Confluence. The benchmark includes 500 questions across ten categories—ranging from single‑doc lookups to multi‑doc project queries, semantic overlap, and unanswerable cases—designed to probe retrieval failure modes.
An evaluation harness with correction‑aware scoring updates the gold set when evidence supports it, and baseline results show BM25 outperforming vector search, while agentic/bash‑style retrieval achieves best completeness at the cost of speed. The repository contains the dataset, generation framework, evaluation harness, and leaderboard, and invites community feedback on retrieval strategies such as hybrid search, rerankers, agents, and graph traversal.
Key changes
- Introduces 500 k synthetic documents simulating a real company across Slack, Gmail, Linear, Drive, HubSpot, Fireflies, GitHub, Jira, and Confluence
- Provides 500 benchmark questions in ten categories to test retrieval and reasoning
- Includes a correction‑aware evaluation harness that can update the gold set
- Baseline shows BM25 beats vector search; agentic/bash retrieval offers best completeness but slower
- Repository contains dataset, generation framework, evaluation harness, and leaderboard
- Encourages community feedback on hybrid search, rerankers, agents, and graph traversal
- Highlights realistic noise such as misplaced docs, near‑duplicates, and conflicting information
- Offers a realistic testbed for enterprise‑scale internal knowledge retrieval