LipDub Beta: Open‑Source Lipsync IC‑LoRA Adapter for 1080p Video
Test LipDub on 1080p 8‑second clips, ensuring single‑speaker videos and supported languages; verify that the output preserves speaker identity and tone.
Test LipDub on a sample 8‑second clip in English, verify lip sync, and report any artifacts to the GitHub repo.
Summary
LipDub is a beta release of an open‑source lipsync capability built on the LTX framework.
It uses an IC‑LoRA adapter to replace the dialogue of an existing video while preserving the speaker’s appearance, vocal identity, tone, and delivery. The model generates speech and lip motion together in a single pass, accepting a source video and a text prompt with the new dialogue.
Output is 1080p Full HD and supports clips up to 8 seconds long, limited to single‑speaker videos. Validated languages include English, French, Spanish, German, and Russian, allowing dubbing into another language or re‑phrasing in the same language.
The beta is available on HuggingFace, with a ComfyUI workflow, a Python pipeline, and full documentation. It is grounded in the research paper “Video Dubbing via Joint Audio‑Visual Diffusion” from Lightricks and Tel Aviv University. The release invites the community to experiment, break, and build with it before the official API ships.
Key changes
- 1080p Full HD output
- Up to 8‑second clip support
- Single‑speaker only
- Supported languages: English, French, Spanish, German, Russian
- Preserves speaker appearance, vocal identity, tone, and delivery
- Replaces dialogue via text prompt in a single pass
- Open‑source beta on HuggingFace
- ComfyUI workflow and Python pipeline available