How to Run Bonsai 27B 1‑bit Model Locally with llama.cpp
Compile PrismML’s llama.cpp with GGML_METAL or GGML_CUDA and run the Bonsai 27B Q2_0.gguf model via llama‑server on port 8080.
Compile PrismML’s llama.cpp with GGML_METAL or GGML_CUDA and run the Bonsai 27B Q2_0.gguf model via llama‑server on port 8080.
Summary
To run the Bonsai 27B 1‑bit model locally, users must download the Q2_0.gguf checkpoint from the PrismML Hugging Face collection and compile PrismML’s fork of llama.cpp. On macOS, the build requires the GGML_METAL flag, while on Linux or Windows CUDA support is enabled with GGML_CUDA. After compiling, the llama‑server can be launched with the command specifying the model path, port 8080, host 0.0.0.0, and default inference parameters (temp 0.7, top‑p 0.95, top‑k 20). The server exposes a REST API that can be accessed via localhost:8080, allowing browser interaction with the model. The guide also notes that the --host flag binds all interfaces, which can be removed if external access is not needed. This setup enables privacy‑preserving, low‑latency inference without relying on external cloud services.
Key changes
- Download Q2_0.gguf checkpoint from PrismML Hugging Face collection
- Compile PrismML’s fork of llama.cpp (GGML_METAL on macOS, GGML_CUDA on Linux/Windows)
- Launch llama‑server with --port 8080 --host 0.0.0.0 and default params
- Server exposes REST API at localhost:8080
- --host flag binds all interfaces; remove if not needed
- Enables privacy‑preserving, low‑latency inference locally