Spring Boot LLM Gateway: Multi‑Tenant Quotas and Cost Control
Patch your LLM integration to use the Spring Boot gateway for rate limiting, budgeting, and audit logging.
Patch your LLM integration to use the Spring Boot gateway for rate limiting, budgeting, and audit logging.
Summary
The article describes a Spring Boot LLM Gateway that sits between clients and OpenAI to enforce API keys, rate limits, token budgets, caching, and audit logging, addressing the cost blowouts experienced when tenants run unlimited requests. The gateway processes requests through eight stages—authentication, input normalization, policy decision, quota enforcement, cache lookup, provider call, response filtering, and audit—using PostgreSQL for durable state, Redis for hot counters, and stateless instances behind a load balancer for horizontal scaling. It supports three enforcement modes—HARD, SOFT, and OBSERVE—allowing tenants to reject, degrade, or observe traffic before tightening limits, and it includes detailed data models for tenants, api_keys, policies, and usage rollups.
Key changes
- Gateway architecture includes eight stages from authentication to audit logging.
- PostgreSQL stores tenants, api_keys, policies, and daily usage rollups.
- Redis holds hot counters for rate limits, in‑flight semaphores, and optional cache.
- Three enforcement modes: HARD (reject), SOFT (degrade), OBSERVE (log only).
- Policy decision stage allows ALLOW/DENY/DEGRADE based on model, token limits, and budget.
- Cache lookup only for deterministic calls with temperature=0.
- Provider call includes bounded timeout and circuit breaker.
- Audit logs written to PostgreSQL for traceability and billing.