I built an open-source, end-to-end Speech-to-Speech translation pipeline with voice preservation (RVC) and lip-syncing (Wav2Lip).

Hey everyone,

I wanted to share a project I've been working on: a complete S2ST pipeline that translates a source video (English) to a target language (Telugu) while preserving the speaker's voice and syncing the lips.

english video

telugu output with voice presrvation and lipsync

Full Article/Write-up: medium
GitHub Repo: GitHub

The Tech Stack:

• ASR: Whisper for transcription.
• NMT: NLLB for English-to-Telugu translation.
• TTS: Meta's MMS for speech synthesis.
• Voice Preservation: This was the tricky part. After hitting dead ends with voice cloning models for Indian languages, I landed on Retrieval-based Voice Conversion (RVC). It works surprisingly well for converting the synthetic TTS voice to match the original speaker's timbre, regardless of language.
• Lip Sync: Wav2Lip for syncing the video frames to the new audio.

In my write-up, I go deep into the journey, including my failed attempt at a direct speech-to-speech model inspired by Translatotron and the limitations I found with traditional voice cloning.

I'm a final-year student actively seeking research or ML engineering roles. I'd appreciate any technical feedback on my approach, suggestions for improvement, or connections to opportunities in the field. Open to collaborations as well!

Thanks for checking it out.