PyCon US 2026 Lightning Talk: LLMs in Six Months
Benchmark your own models using the pelican test to assess coding agent quality.
Benchmark your own models using the pelican test to assess coding agent quality.
Summary
Simon Willison’s five‑minute lightning talk at PyCon US 2026 reviews the last six months of LLM progress, highlighting a November 2025 inflection point where the leading model changed hands five times among Anthropic, OpenAI, and Google. He notes that coding agents have improved dramatically thanks to reinforcement learning from verifiable rewards, enabling them to produce reliable code without extensive post‑editing.
The talk uses a whimsical pelican‑on‑a‑bicycle test to compare model output quality, showing that Gemini 3 produced the best illustration among the contenders. Willison also showcases a micro‑JavaScript project written in Python that runs in the browser via Pyodide, illustrating the experimental nature of some LLM‑driven tools. The presentation underscores the rapid pace of model iteration and the growing maturity of coding agents. It serves as a concise snapshot for developers to gauge which LLMs are most suitable for their use cases. The slides are available online for those who missed the live session. The talk also hints at future directions for LLM research and tooling.
Key changes
- November 2025 inflection point with model shifts
- Best model changed hands five times among Anthropic, OpenAI, Google
- Coding agents improved via reinforcement learning from verifiable rewards
- Coding agents now produce reliable code with minimal post‑editing
- Gemini 3 produced best pelican illustration
- Opus 4.5 dominated next months
- Micro‑JavaScript project written in Python runs in browser via Pyodide
- Slides provide concise snapshot for developers