Engineering How OpenAI Delivers Low‑Latency Voice AI at Scale

OpenAI’s real‑time voice AI team re‑architected the WebRTC stack to meet the demands of 900 million weekly active users, focusing on global reach, instant connection setup, and stable media round‑trip time. The new split relay + transceiver architecture replaces the traditional one‑port‑per‑session model with a single UDP port per server, demultiplexing packets at the relay while the transceiver owns the full WebRTC session state, including ICE, DTLS, and SRTP. By moving media termination to an edge service built on Pion in Go, the team eliminated the need for large public UDP port ranges, simplifying load‑balancer configuration and improving security posture. The transceiver handles both signaling (SDP negotiation, codec selection, ICE credentials) and media termination, allowing backend services to scale as ordinary micro‑services within Kubernetes. This design preserves standard WebRTC behavior for clients while enabling OpenAI to route packets deterministically to the correct transceiver across a load‑balanced fleet, addressing state stickiness and packet loss. The architecture also supports low jitter and packet loss, ensuring turn‑taking feels crisp for conversational agents. The team’s work demonstrates how to combine WebRTC’s mature ecosystem with custom routing to deliver low‑latency voice AI at scale. The solution is now powering ChatGPT Voice, the Realtime API, and several research projects.