Lyft’s Self‑Serve AI Agent Platform: LangGraph, LangSmith, and DynamoDB State Persistence
Deploy a LangGraph router‑based multi‑agent system with DynamoDB state persistence, LangSmith tracing, and LLM‑as‑a‑judge evaluation, and configure PagerDuty alerts for error rate >5% or p95 latency >10 s.
Implement a LangGraph router‑based multi‑agent architecture, configure rider and driver intent subagents, enable DynamoDB state persistence, set up LangSmith tracing with custom metadata, add an LLM‑as‑a‑judge evaluation pipeline, and configure PagerDuty alerts for error rate >5% or p95 latency >10 s.
Summary
Lyft has built a self‑serve AI agent platform that lets ops teams, VoC leads, and product managers create and refine agents without MLE involvement. The architecture uses LangGraph's router‑multi‑agent pattern, where a meta agent classifies incoming requests and dispatches them to rider‑ or driver‑specific subagents, each implemented as a LangGraph `StateGraph`. Safety checks run in parallel at every turn via fan‑out commands, and subagents are modular, allowing new intents to be added by registering a new subgraph.
Configurable agents are loaded at runtime from JSON stored in an internal config service, pulling prompts from LangSmith's Prompt Hub; this lets a product manager define a new agent with a prompt and JSON config without code changes. Conversation state is persisted in DynamoDB through a custom `DynamoDBSaver`, enabling multi‑turn replay and debugging. LangSmith provides end‑to‑end tracing, dashboards, and an LLM‑as‑a‑judge evaluation pipeline that automatically scores production traces and triggers PagerDuty alerts when error rates exceed 5% or p95 latency surpasses 10 s. The result is a 6‑month development cycle reduced to weeks, with high‑quality, safe agents handling millions of rider and driver interactions.
Key changes
- Meta agent routes requests to rider or driver intent subagents via Command(goto)
- Safety checks run in parallel before LLM reasoning using fan‑out commands
- Configurable agents are defined by JSON config and prompts from LangSmith Prompt Hub, enabling non‑technical creation
- Conversation state persisted in DynamoDB via custom DynamoDBSaver for multi‑turn replay
- LangSmith tracing captures node execution, token usage, latency, and custom metadata
- LLM‑as‑a‑judge evaluation automatically scores production traces and triggers PagerDuty alerts
- PagerDuty alerts fire when error rate >5% or p95 latency >10 s over 15‑minute window
- Development cycle shortened from 6 months to weeks