Client‑Side Caching as a Decision System Under Uncertainty
Patch: design a layered decision system with rule, ML, and LLM fallback for caching.
Implement rule engine first, then add ML scoring, and configure LLM fallback with confidence threshold.
Summary
Standard caching strategies like SWR, TTL, and heuristic scoring plateau when data volatility varies within the same dataset. The article introduces a layered decision system that replaces static caching with a rule engine, an ML scoring layer, and an LLM fallback for ambiguous cases. It details how to compute a volatility score using EWMA, a priority score from user interaction weight, and derive TTLs adaptively.
Machine‑learning models such as logistic regression, XGBoost, and embedding classifiers can predict cache actions, but they require labeled data and retraining. LLMs are only invoked when ML confidence falls below a threshold, and decision caching is used to amortize LLM inference costs. The architecture diagram shows UI → Context Builder → Policy Engine (Rule, ML, LLM) → Cache Layer → Network.
Key changes
- Caching is a decision system with actions HIT, REVALIDATE, BYPASS, SWR
- Rule engine handles deterministic cases, ML scoring predicts actions, LLM fallback for low confidence
- Decision caching stores strategy per context to reduce LLM calls
- Architecture layers: UI, Context Builder, Policy Engine (Rule/ML/LLM), Cache, Network
- ML models require labeled data and retraining, LLMs are costly and unpredictable
- LLM invoked only when ML confidence < threshold and signals conflict