Gradual Memory Increase Observed with Step‑3.5‑Flash on 128GB System
Investigate and patch llama.cpp 2.13.0 to fix the gradual memory increase observed when running large models.
Check for llama.cpp 2.13.0 memory‑leak bug, report to the repo, and consider downgrading or patching.
Summary
A user with a 128GB Strix Halo box ran the Step‑3.5‑Flash model (105 GB) with 150K context, leaving roughly 17 GB of free memory. During successive queries, memory usage climbed from 108 GB to about 120 GB and never returned to the baseline, indicating a possible memory leak. The user attempted to free memory with the /compact command, but it had no effect. The model was unloaded before the system ran out of memory. The test was performed using llama.cpp 2.13.0 with the Vulkan backend through LM Studio. No other memory‑management options were explored, and the issue persisted across multiple runs. The author suspects a leak in the llama.cpp 2.13.0 Vulkan implementation but has not found a fix. The post serves as a warning to developers and users of large models on high‑capacity systems.
Key changes
- Step‑3.5‑Flash model 105 GB on 128 GB system
- Memory usage climbs to 120 GB, not returning to baseline
- /compact command does not free memory
- Model unloaded before OOM
- Using llama.cpp 2.13.0 vulkan backend via LM Studio
- Memory leak suspected
- No resolution reported
- Suggests reporting to repo