Lambda Locking Storm: Orphaned Transactions Cause Outages
Patch: set statement_timeout to 5000 ms; enable connection pool idle timeout; wrap payment calls in separate transaction; monitor for orphaned locks.
Patch: configure statement_timeout to 5000 ms in Postgres; add idle_in_transaction_timeout; wrap payment calls in a separate transaction; monitor pg_stat_activity for idle_in_transaction.
Summary
The article examines a Lambda‑based booking platform that suffered from a connection storm caused by orphaned database locks. When a Lambda function is killed mid‑transaction, Postgres remains in an idle‑in‑transaction state holding exclusive row locks, which blocks subsequent booking attempts. The author demonstrates that a single long‑running payment call can leave a transaction open for minutes, draining the connection pool and triggering timeouts. The resulting outage manifests as a spike in Lambda concurrency, high CPU usage, and 100 % error rates on bookings. The root cause is a lack of statement timeout and orphaned transaction handling. The solution involves four knobs: setting statement_timeout, using connection pooling with idle timeout, wrapping payment calls in a separate transaction, and cleaning up orphaned locks. By applying these settings, the platform can prevent a single slow upstream call from cascading into a full outage. The article emphasizes that performance work is high‑leverage and can be done in a few hours.
Key changes
- Orphaned transactions cause row locks to persist
- Lambda timeout leads to container kill
- Connection pool drains
- Statement_timeout can abort long queries
- Payment call duration can be 25 s
- 100 % error rate on bookings
- Solution: set statement_timeout, use separate transaction, monitor locks
- Idle_in_transaction_timeout helps