Backtrack Sampler with Verifier Dramatically Improves Tiny LLMs
Implement a backtrack sampler with a verifier to reduce hallucinations, noting doubled VRAM and compute.
Implement a backtrack sampler with a verifier to reduce hallucinations, noting doubled VRAM and compute.
Summary
A recent paper demonstrates that a backtrack sampler combined with a verifier can dramatically improve the performance of tiny LLMs. The authors claim a 0.5 b model can match the coding quality of a 2–4 b model without any weight changes, and that hallucinations can be reduced by 30–50 %. The sampler forces the model to backtrack and regenerate when it makes a mistake, which introduces a 5–30 % decode‑speed hit. To compensate, a verifier model of roughly the same size is trained, doubling VRAM usage and increasing compute by 1.5–3×. The verifier generalises across weight classes lower or equal to the original model, provided it has seen the same domain distribution during training. Training the verifier requires only about 0.01 % of the token size used for full pre‑training. The paper suggests that future VGB models could achieve fast generation while maintaining quality.
Key changes
- Backtrack sampler with verifier reduces hallucinations by 30–50 %
- Verifier model size equals original model, doubling VRAM usage
- Decode speed hit 5–30 % due to backtracking
- Verifier generalises across lower or equal weight classes
- Training verifier requires ~0.01 % of pre‑training token size
- Future VGB models could achieve fast generation while maintaining quality