Running Qwen 3.6 Models Locally on an RTX 3090 – Performance & Reliability Issues
Test smaller Qwen models or different quantizations to improve speed; tweak llama‑server flags like –c and –b for optimal GPU memory usage; consider Qwen 3.6 27B with q8_0 quant for better reliability.
Test smaller Qwen models or different quantizations to improve speed; tweak llama‑server flags like –c and –b; consider Qwen 3.6 27B with q8_0 quant; evaluate auto routing.
Summary
Running Qwen 3.6 models locally on a Windows 11 machine with an RTX 3090 (24 GB VRAM), 64 GB RAM, and a Ryzen 5700X CPU, the user experiments with the 35 B UD Q4_K_M and 27 B UD Q4_K_XL variants. The 35 B model is invoked via llama‑server.exe with flags –ngl 99 –c 131072 –np 2 –fa on –ctk f16 –ctv f16 –b 2048 –ub 512 –t 8 –mlock –rea on –reasoning‑budget 2048 –reasoning‑format deepseek –jinja –metrics –slots –port 8081 –host 0.0.0.0, while the 27 B model uses –c 196608, –np 1, –ctk q8_0, –ctv q8_0, –no‑mmap, –reasoning‑budget -1, and the same remaining flags. The user’s use cases include a Hermes agent on a Raspberry Pi 5 for Reddit and job scraping, local coding with OpenCode/QwenCode for small scripts and debugging, and occasional infrastructure setup via prompts. However, the 35 B model is too slow for iterative tasks, and the 27 B model, while faster, is unreliable, often taking 20–30 minutes for simple prompts and producing broken code. The user seeks better model and quantization recommendations that balance speed and coding reliability on a 3090, wants to improve throughput per second, and desires an automated model loading or routing solution that can switch models or keep multiple models warm without manual restarts. They also ask whether auto‑switching based on request type is possible and how to reduce context size or flag usage to boost performance.
Key changes
- Qwen 3.6 35B UD Q4_K_M runs with llama‑server flags –ngl 99 –c 131072 –np 2 –fa on –ctk f16 –ctv f16 –b 2048 –ub 512 –t 8 –mlock –rea on –reasoning‑budget 2048 –reasoning‑format deepseek –jinja –metrics –slots –port 8081 –host 0.0.0.0
- Qwen 3.6 27B UD Q4_K_XL runs with llama‑server flags –ngl 99 –c 196608 –np 1 –fa on –ctk q8_0 –ctv q8_0 –b 2048 –ub 512 –t 8 –no‑mmap –rea on –reasoning‑budget -1 –reasoning‑format deepseek –jinja –metrics –slots –port 8081 –host 0.0.0.0
- 35 B model is too slow for iterative tasks; 27 B model is faster but unreliable, often taking 20–30 minutes for simple prompts
- Use cases include Hermes agent on Raspberry Pi 5, local coding with OpenCode/QwenCode, and infra setup via prompts
- User seeks better model and quantization recommendations for speed and reliability on a 3090
- User wants throughput improvements and an automated model loading/routing solution