Microsoft VibeVoice Port Enables Edge Voice Cloning and Long‑Form ASR

Microsoft’s VibeVoice, a cutting‑edge speech model, has been re‑implemented in a pure‑C++ library called vibevoice.cpp. The new port uses the lightweight ggml inference engine and is fully cross‑platform, supporting CPU‑only, CUDA, Metal, Vulkan and hipBLAS back‑ends. It can be deployed as a single binary or linked via a flat C ABI (libvibevoice.so), making it ideal for edge devices where Python, Torch or vLLM are undesirable.

The library ships with a 0.5 B GGUF model for text‑to‑speech (TTS) voice cloning. The model is pre‑converted on HuggingFace and can produce 24 kHz speech from a 30‑second reference clip in just 28 seconds when run on a CUDA Q4_K GPU. For long‑form automatic speech recognition (ASR), a 7 B model returns JSON segments that include start and end timestamps, speaker labels and the transcribed content for audio streams of up to 17 minutes. Performance benchmarks show that the ASR model processes 17 minutes of audio in 150 seconds on a CPU Q4_K and 1,929 seconds on a CPU Q8_0, while using only 6 GB of peak resident set size on CUDA.

The diffusion head of the model builds 20 small graphs per latent frame; future optimisations aim to reuse these graphs to further reduce latency. By eliminating the need for heavyweight dependencies, vibevoice.cpp offers a production‑ready, lightweight solution for speech‑to‑speech and ASR tasks across a wide range of hardware, from high‑end GPUs to low‑power edge processors.