Systems at LangChain Full Text Search in SmithDB: Constructing and Querying our Inverted Index (Pt. 2)

SmithDB implements an object‑storage backed inverted index for full‑text search. Index construction occurs inline during ingestion, making new runs searchable within seconds. A JSON tape adapted from Apache Arrow’s arrow‑json crate tokenizes payloads, and string interning maps unique terms to compact integer IDs, cutting construction time by ~2.2×. Radix sort groups postings by term in O(n), and flush thresholds—32 MB row groups, 2 MB aligned chunks, 8 MB mid‑term spills—bound memory usage. The compaction service merges smaller files using a min‑heap, keeping only one decoded chunk per input in memory. Query time uses DataFusion and Vortex LayoutReader, routing predicates to indexed or scanned columns before any object‑storage request. The index layout is virtualized so that a segment with multiple indexed columns appears as a single file to the planner. Query latency is bounded by the number of row groups surviving pruning, each GET bounded by the row‑group byte budget.

This architecture enables efficient full‑text search with minimal latency, leveraging in‑memory FSTs and careful memory management to keep queries fast even with large datasets.