SQLite is All You Need for Durable Workflows
Use a local SQLite database with Litestream for durable workflow state, avoiding extra orchestration layers.
Implement workflow state persistence using SQLite and Litestream, and evaluate if asynchronous replication meets your durability requirements.
Summary
The article argues that for many durable workflow systems, a local SQLite database coupled with Litestream backup is sufficient, eliminating the need for a separate orchestration tier. It explains that durable execution requires only persistent workflow state; compute can remain cheap and disposable, which fits the Obelisk framework that stores progress in an execution log and replays from history. SQLite offers transactional durability without a network hop, control plane, or additional operational surface, making it ideal for small, self‑contained AI agents or experimental pipelines. Litestream streams SQLite changes asynchronously to S3‑compatible object storage, allowing easy backup, migration, and inspection, though its asynchronous nature means a restore may miss the latest writes if the local volume disappears. The article notes that while SQLite is perfect for bursty, low‑scale workloads, Postgres remains the choice when higher availability, shared scalability, or synchronous replication is required. It highlights that a fleet of micro‑VMs or containers each hosting a single SQLite database can provide better fault isolation and cost efficiency than a single large shared database. The piece also cautions that asynchronous replication is not a substitute for a highly available shared database and that the decision should be driven by the specific durability and scalability needs of the workflow. Finally, it positions SQLite + Litestream as a sensible default for AI agents and experimentation workflows, offering simplicity and resilience without unnecessary infrastructure overhead.
Key changes
- Durable workflow state can be stored in a local SQLite file, eliminating the need for a separate orchestration tier.
- Compute can remain cheap and disposable; only state persistence matters.
- Litestream streams SQLite changes asynchronously to S3‑compatible storage for backup and migration.
- Asynchronous replication may miss the latest writes if the volume disappears before sync; acceptable for AI experiments but not HA.
- SQLite is suitable for AI agents with small self‑contained state; micro‑VMs or containers can each host one SQLite DB.
- For higher availability or shared scalability, Postgres remains appropriate.
- Obelisk supports both SQLite and Postgres; choose SQLite when state demands are minimal.
- Using SQLite + Litestream reduces infrastructure complexity and cost.