Hot new tool makes your localised videos look and sound authentic with AI-powered dubbing and lip-syncing.
Machine translation or even dubbing is fast becoming the new normal for content creators. However, lip-syncing has been uncharted territory. For a long time, there was a lack of synchronisation between lip movements and voices in translated and dubbed video content. Experts believe this is one of the reasons why dubbing is relatively unpopular in English-speaking countries.
Today, Rask AI, a technology company that provides video creators with all sorts of AI-powered features, unveiled its first-ever multi-speaker lip sync.
The complexity of lip-syncing lies in matching lip movements to sound. And that’s easier said than done. In fact, the AI re-creates the lower part of the face every second, based on gazillions of facial impressions and micro-movements. The technology is based on generative adversarial network (GAN) learning, which consists of a generator and a discriminator. Both the generator and the discriminator compete to stay one step ahead of the other. The generator clearly generates content (lip movements), while the discriminator is responsible for quality control.
For a user, however, the whole process only takes a few minutes. According to Maria Chmir, founder and CEO of Rask AI, the new feature will help content creators expand their audience. The AI visually adjusts lip movements to make a character appear to speak the language as fluently as a native speaker.