LLMs Corrupt Your Documents When You Delegate – DELEGATE‑52 Study
Benchmark your LLMs against DELEGATE‑52 to quantify document corruption before deployment.
Run DELEGATE‑52 on your own models to assess reliability; adjust workflows accordingly.
Summary
The DELEGATE‑52 benchmark simulates long delegated workflows across 52 professional domains, including coding, crystallography, and music notation. Researchers tested 19 large language models and found that even frontier models—Gemini 3.1 Pro, Claude 4.6 Opus, and GPT 5.4—corrupt an average of 25 % of document content by the end of extended interactions. The study shows that agentic tool use does not improve performance, and that corruption severity increases with document size, interaction length, and the presence of distractor files. Sparse but severe errors accumulate over time, silently degrading documents.
The paper highlights the unreliability of current LLMs as delegates, emphasizing the need for rigorous testing before deploying them in knowledge‑work workflows. The authors provide the DELEGATE‑52 dataset, methodology, and baseline results to encourage further research into mitigating document corruption.
These findings suggest that organizations relying on LLMs for document editing should benchmark their models against DELEGATE‑52 and implement safeguards to detect and correct corruption before finalization.
Key changes
- Introduces DELEGATE‑52 benchmark with 52 domains
- Tests 19 LLMs, finding 25 % average corruption
- Frontier models still corrupt documents
- Agentic tool use does not improve performance
- Corruption worsens with document size, interaction length, and distractors
- Provides dataset and methodology for future research