SysMoBench Reveals LLMs Fail at Conformance in TLA+ Specifications
Run SysMoBench to evaluate LLM‑generated TLA+ specs for syntax, runtime, conformance, and invariants before deployment.
Integrate SysMoBench into your LLM evaluation pipeline to catch conformance gaps before code generation.
Summary
The Specula team introduced SysMoBench, a benchmark that automatically evaluates TLA+ specifications generated by large language models against real system traces. The tool feeds eleven concurrent and distributed systems, including Etcd, ZooKeeper, and RedisRaft, into a four‑phase pipeline: syntax, runtime, conformance, and invariant checks. While leading LLMs such as Claude, GPT‑4, Gemini, and Qwen compile cleanly and run through TLC, their conformance scores hover around 46 % and invariant scores around 41 %. The benchmark exposes two systematic failure modes: (1) the model allows transitions that real systems can never reach, and (2) it omits transitions that real systems always perform. These gaps arise because LLMs default to textbook formalizations rather than the actual implementation details of the target system.
SysMoBench’s per‑action diagnostics reveal that models often merge multi‑step operations into a single guard, erasing intermediate states. For example, Claude Sonnet’s ZooKeeper spec incorrectly accumulates all votes, leading to impossible states, and it also disables the HandleNotification action that should be enabled when an incoming epoch is higher. The benchmark demonstrates that syntax and runtime success are insufficient; conformance and invariant phases are essential to validate real‑world behavior. The authors suggest that future LLM training should include trace‑based evaluation to close the gap between textbook modeling and system‑level correctness. The study highlights the need for more rigorous formal‑methods pipelines when relying on LLM‑generated specifications.
Key changes
- SysMoBench provides a four‑phase evaluation (syntax, runtime, conformance, invariant) for LLM‑generated TLA+ specs.
- Eleven concurrent and distributed systems are benchmarked, including Etcd, ZooKeeper, and RedisRaft.
- Leading LLMs score near‑perfect on syntax but only ~46 % on conformance and ~41 % on invariants.
- Failure modes include allowing impossible states and omitting necessary transitions due to textbook modeling.
- Per‑action diagnostics show that models merge multi‑step operations into a single guard, erasing intermediate states.
- Claude Sonnet’s ZooKeeper spec accumulates all votes, creating states that real code never reaches.
- The HandleNotification action is incorrectly disabled when an incoming epoch is higher, missing real transitions.
- The study underscores that syntax and runtime checks are insufficient; conformance and invariant checks are essential.