World's Smallest Llama2 Inference Engine Fits in 1369 Bytes

A new Llama2 inference engine has been built that occupies only 1,369 bytes of x86 real‑mode assembly, allowing it to boot directly from disk and generate text before any operating system loads. The engine loads a quantized stories260K model—260,000 parameters across five layers, eight attention heads, and a 512‑token vocabulary—directly into high memory, then performs a full transformer forward pass for each token. Quantization uses int8 weights with a global absmax scale, and pre‑computed exp and silu lookup tables are embedded to reduce decoding overhead. The Q/K/V and gate/up weight matrices are fused so the assembly can issue a single matmul call, while the KV cache is also quantized to int8 at runtime with a per‑token scale stored in a separate buffer, keeping the cache small enough for the full 512‑token context. Sampling is currently limited to greedy argmax, but the design leaves room for more sophisticated techniques. The code is heavily golfed, so performance and precision are not optimal, and the architecture and prompt are hard‑coded. Extending the engine to larger models like stories15M would require switching to protected or unreal mode.

The project includes a Python script (quantize.py) that packs the model into a custom binary format, a makefile that builds the assembly, and a download.sh script to fetch the model data. Contributors are invited to shrink the binary further, and the repository encourages adding a name to the code as a sign of participation.