Although I have published several reviews —and frequently use— many of other Descript features, this is the first time I am testing and reviewing Descript’s a/v translation + dubbing. Descript not only translates (with the full capability of human correction by a native speaker, as long as you are on the Business or Enterprise plan, I’m currently on the Creator plan.) and dubs, but also maintains astonishingly accurate lip sync with the person speaking. My first inclination was to imagine that Descript was rotoscoping the video to achieve that. However, my later research indicates that instead of rotoscoping, Descript says that it uses generative AI to achieve lip sync. So when you translate and dub a video with Desript, it not only creates a new voiceover. Descript then automatically regenerates the lower half of the speaker’s face to match the new language’s sounds, unique pacing, and mouth movements. As a certified translator myself, I disagree with doing translations that are not curated by a human native speaker. However, my goal with this test is to evaluate the lip sync quality, knowing that if there is a real project which requires a perfect translation, I can upgrade to the Business plan for that month. Ahead you’ll see an original without translation or dubbing, followed by results in two types of French, German and Greek.

How the Descript lip sync process works

Instead of tracing over frames, Descript‘s AI’s video model works by:

1. Compressing the video: It encodes the original video and reference frames of the speaker’s face into a learned latent space.
2. Generating mouth movements: Using the translated audio, the AI regenerates the lower face so the words follow the new speech while keeping the original lighting, teeth, and identity of the speaker.
3. Merging smoothly: The AI blends the newly generated mouth movements with the untouched, original parts of your video (like the background and upper face) to create a seamless, authentic look.

Original video in US English (without any translation or dubbing)

The above video is one I borrowed from my recent Review: OBSBOT Meet Flip 4K UHD/1080p/720p UVC camera for broadcast TV + online meetings too.

Descript’s translation from English to Canadian French, with dubbing and lip sync

Descript’s translation from US English to European French, with dubbing and lip sync

Descript’s translation from US English to German

Descript’s translation from US English to Greek, with dubbing and lip sync

Original video in Castilian (without any translation or dubbing)

This is the one I borrowed from my recent Reseña: Cámara Meet Flip 4K UHD/1080p/720p de OBSBOT para televisión y reuniones internéticas (in Castilian), which I had published on Escuchalibros.com. The purpose of including it here is to see if Descript does a better job translating to certain target languages from a Castilian original, as opposed to a US English one.

Descript’s translation from Castilian to Canadian French, with dubbing and lip sync

Descript’s translation from Castilian to European French, with dubbing and lip sync

Descript’s translation from Castilian to German, with dubbing and lip sync

Descript’s translation from Castilian to Greek, with dubbing and lip sync

Conclusions

Whether we want to call it «rotoscoping» or «generative AI», the end-product lip sync que get from these translations/dubbing from Descript is amazing. As a certified translator myself, I disagree with doing automatic translations that are not curated by a human native speaker. I am a bit insulted that they don’t allow us to correct the auto-translation before execution with the Creator plan. I hope that «option» comes to the Creator plan soon. I’m not asking Descript to give us a professional translator. I am only asking the option to hire one ourselves to polish the auto-translation. In the meantime, the solution is to upgrade to the Business plan. Congratulations to Descrpt on the lip sync result.

Lee este artículo en buen castellano

Traducción y doblaje de Descrpt con sincronización labial, más allá del rotoscopio