RAG Pipeline Drift Detection and Monitoring

The article explains how a RAG pipeline can fail when the corpus changes while the embedding model, chunker, and reranker remain unchanged. Three common causes are identified: re‑indexed corpus, model swap, and fine‑tuning, all of which alter vector space geometry and bias retrieval toward newer data. Global metrics such as top‑1 hit rate may stay high, but per‑class recall can drop dramatically, as illustrated by a legacy‑product recall falling from 0.81 to 0.54. To detect this drift, the author proposes computing similarity moments per query class daily, measuring mean, standard deviation, and a noise‑floor maximum, and using a z‑score threshold of three sigma to trigger alerts. The monitoring script uses SentenceTransformer all‑MiniLM‑L6‑v2 to embed probe queries and known‑good chunks, storing results in a JSONL file. The approach is generic and can be applied to any RAG system that relies on fixed embeddings.

By focusing on per‑class metrics, teams can catch subtle degradation before it impacts users, even when overall dashboards look healthy. The method requires only a few lines of Python and a scheduled job, making it easy to integrate into existing pipelines.

The key takeaway is that embedding stability alone does not guarantee retrieval quality; continuous per‑class monitoring is essential to maintain service reliability.