Heretic 1.3 Released: Reproducible Runs, Benchmarking, and VRAM Optimizations
Check Heretic 1.3's reproducibility and benchmarking features.
Check Heretic 1.3's reproducibility and benchmarking features.
Summary
Heretic 1.3 is now available on GitHub and brings a suite of new features aimed at making language model decensoring more transparent and reproducible. The headline addition is a reproducible runs system that captures the exact PyTorch version, GPU, driver, and accelerator library used, and can optionally generate a `reproduce` directory that can be pushed to Hugging Face for byte‑for‑byte replication.
In addition, Heretic now ships a lightweight benchmarking engine that runs standard tests such as MMLU, EQ‑Bench, GSM8K, and HellaSwag directly from the CLI, leveraging lm‑evaluation‑harness for direct comparison to published scores. The release also includes optimizations that cut peak VRAM usage, enabling larger models to be processed on the same hardware. Layer and module handling logic has been refactored to support the newest generation of models, including Qwen3.5 and Gemma 4. All new features are optional and the tool prompts before uploading any reproducibility data. The full list of improvements is documented in the release notes, and future releases promise even more ambitious capabilities.
Key changes
- Reproducible runs system generates a `reproduce` directory with environment details
- Optional upload of reproducibility data to Hugging Face
- Integrated benchmarking engine runs MMLU, EQ‑Bench, GSM8K, HellaSwag via lm‑evaluation‑harness
- Optimizations reduce peak VRAM usage for larger models
- Layer and module handling refactored to support Qwen3.5 and Gemma 4
- All new features are optional and prompt before publishing
- Release notes document full list of improvements
- Future releases will build on reproducibility foundation