Testing MiMo-V2.5-IQ3_S with 1,048,576 Context on LLaMA Server
Run MiMo‑V2.5‑IQ3_S with 1M context on your GPU and measure token throughput.
Run the MiMo‑V2.5‑IQ3_S command on your GPU, adjust repetition penalty to 1.1 and temperature to 0.2, and test for loops with a fixed seed.
Summary
MiMo‑V2.5‑IQ3_S was tested with a 1,048,576‑token context on llama‑server.exe. The command used 16 threads, no‑mmap, flash‑attn on, and fit‑on with -sm layer.
Offloading 49/49 layers to the GPU produced Vulkan buffer sizes of 72,842.29 MiB, 34,524.53 MiB, and 488.91 MiB. The RTX 6000 96 GB and W7800 48 GB GPUs handled the load, and the model outperformed minimax at 50k context in both prefill and response throughput.
Looping was observed, but applying a repetition penalty of 1.1 and temperature 0.2 reduced it, and using a fixed seed further stabilized generation. The setup demonstrated stable, fast inference with a 1M context, suggesting viability for large‑context workloads.
Key changes
- Context size set to 1,048,576 tokens
- Offloaded 49/49 layers to GPU
- Vulkan buffer sizes: 72,842.29 MiB, 34,524.53 MiB, 488.91 MiB
- Processing speed outperformed minimax at 50k context
- Looping mitigated with repetition penalty 1.1 and temperature 0.2
- Fixed seed suggested to avoid loops
- GPU usage: RTX 6000 96 GB + W7800 48 GB
- Command used 16 threads, no‑mmap, flash‑attn on, fit‑on, -sm layer