pg-cdc: One‑Way CDC Pipeline for AI‑Safe Data Access
Patch: Deploy pg‑cdc as a WAL consumer, configure it to write Parquet files to S3, and grant IAM roles to AI consumers for read‑only access, ensuring no direct database credentials are exposed.
Patch: Install pg‑cdc, configure WAL consumer, set up S3 Parquet storage, and grant IAM roles to AI services for read‑only access.
Summary
The article introduces pg‑cdc, a PostgreSQL logical replication consumer that streams WAL changes into immutable Parquet files while enforcing a strict one‑way air‑gap between production and downstream consumers. By consuming the WAL, pg‑cdc writes data to S3 (or similar) without exposing any database credentials, and all reads are mediated through IAM roles, eliminating the need for connection strings or shared execution environments. The architecture guarantees no return path, unidirectional WAL flow, append‑only storage, and governance tags that control which tables and columns are exposed to AI or analytics workloads. Key properties include no database credentials, no queries executed on PostgreSQL, and auditable read tracking for every consumer request. pg‑cdc’s design addresses the risk of AI agents inadvertently accessing production databases by providing a physical and logical air‑gap that isolates the OLTP system from AI pipelines. The tool is open‑source, written in Go, and can be deployed alongside existing PostgreSQL instances; it exposes a simple REST API for consumers to request data slices, which are served from the Parquet store. The article outlines a layered approach: isolated OLTP, CDC boundary, governed storage, controlled query layer, and AI/analytics consumers. The author invites feedback and links to the GitHub repo, encouraging teams to adopt pg‑cdc for secure, governed data ingestion into AI pipelines.
Key changes
- pg‑cdc consumes PostgreSQL WAL and writes immutable Parquet files
- No database credentials are used; reads are mediated via IAM roles
- Guarantees a one‑way air‑gap: no return path, unidirectional flow, append‑only storage
- Governance tags control exposed tables/columns; no queries run on PostgreSQL
- Exposes a REST API for consumers to request data slices from Parquet store
- Open‑source Go implementation can be deployed alongside existing PostgreSQL