Your Playwright Tests Are Lying to You (And How I Built a System to Catch It)
Set up a requirement‑aware test drift analyzer that pulls Jira requirements, Playwright test code, and Git history, then uses RAG and Claude to flag semantic drift.
Deploy the test drift analyzer in your CI pipeline to catch semantic drift before tests pass.
Summary
Your Playwright Tests Are Lying to You (And How I Built a System to Catch It) tackles the subtle phenomenon of test drift, where passing tests no longer validate the intended requirements.
The author explains that assertions can be removed, requirements evolve, and coverage silently drops, yet the suite remains green. To detect this, a requirement‑aware test drift analyzer was built that pulls Jira tickets for requirement intent, Playwright test code for test intent, and Git history for change analysis. The system then uses RAG with embeddings from sentence‑transformers or Ollama, stores them in ChromaDB, and queries Claude via the Claude API to compare intent versus implementation. The analyzer outputs a drift report highlighting missing assertions, historical changes, coverage percentages, and even suggests existing tests that already implement the missing coverage. For example, the report flagged a missing error‑message validation for an invalid login and noted that an assertion for dashboard visibility was removed two commits ago, with a coverage of 65%. The tool also recommends referencing auth/error.spec.ts for correct implementation. By integrating this system into the CI pipeline, teams can catch semantic drift before tests pass and maintain alignment with product requirements.
Key changes
- Extracts requirement intent from Jira tickets (expected behaviors, validation scenarios)
- Extracts test intent from Playwright tests (assertions and flows)
- Analyzes Git history to detect removed assertions and coverage degradation
- Uses RAG with embeddings (sentence‑transformers/Ollama) stored in ChromaDB for repository context
- Compares requirement vs test intent to detect semantic drift
- Outputs drift reports with missing assertions, coverage %, and suggestions for existing tests
- Built with Python, LangChain, ChromaDB, Claude API, Jira API, and Git history analysis