Viral AI dub tool Heygen: Is this the final step for globalizing the creator economy?

We may not be that far off from your favorite YouTube vloggers and celebs speaking any language under the sun. Whether it’s Elon Musk speaking French, MKBHD speaking Spanish, Casey Neistat speaking Hindi or Emma Chamberlin parla’ing the Italiano, Heygen seems on the cusp of harnessing the immense power in AI-based translation tools in a very tangible way. While the tool is providing a taste of what’s to come in the creator economy’s globalization moving forward, YouTube has not only taken note, but also taken action, testing a similar tool with hundreds of creators. Today, OMR is taking a closer look at the status quo, history and current use cases.

"Alright, supposedly this is a vocal translator. That’s drag and drop. So I can just drop this in and theoretically it should just make it the new language. And then it should translate my lip movements as well so that it matches the new language." The rendered German coming out of Jon Finger’s “mouth” is not perfect. But his X tweet a few days ago is impressive never the less as it shows that in just a matter of minutes, his above-quoted 30-second clip can be translated instantly into French and German, mirrored lip movements and all.

Traffic spike to Heygen website

Not only did Heygen translate Finger's original video from English, but it cloned his voice (using another piece of software from Eleven Labs), and generated a new video in which Finger speaks lip-synched German and French. All of this was done automatically. Finger only had to upload the original to Heygen's server and make sure it was at least 30 seconds long. The resulting video has been viewed about 6.8 million times so far on X alone.

In principle, anyone can do what Finger did and do so for free (at the moment). However, the hype is real and so are the queues: almost 180,000 videos are waiting to be translated. As is often the case, paying users (USD 48 or 59 US per month, depending on the payment method) are supposed given preferential treatment and their videos are said to be processed more quickly.

Another X-Tweet, this time from Heygen’s official account, provides a glimpse of the potential impact on future creator content. It shows A-list YouTubers Marques Brownlee ("MKBHD"), Emma Chamberlain and Casey Neistat, as well as a clip of Elon Musk (in the Joe Rogan Experience podcast) all “speaking” foreign languages.

Youtube testing proprietary tool

Founded in China and initially named Surreal, then Movio, Heygen is now based in the U.S. According to a Tech Crunch report from 2022, Heygen has raised USD 9m so far from IDG, Sequoia Capital China and Baidu, among others. " Video translation can play a pivotal role for YouTubers and the education sector," founder Joshua Xu posted to Linkedin last week. "Breaking down language barriers makes content accessible to the entire globe, not just the 10% who speak English.”

This is a trend that Youtube wants to further foster. In June, Amjad Hanif, a Youtube exec responsible for product development, announced on stage at Vidcon that it was developing a tool similar to Heygen. Hundreds of channel operators are already testing the software, according to The Verge, including the animated cartoon channel The Amoeba Sisters, as well as (according to Rest of World) history channel Kings and Generals and (according to Advertising Age) Lia Shelesh aka Sssniperwolf.

15 percent spike in watch time with multi-language videos

According to The Verge, Youtube's tool is based on software from Aloud, a startup from Google's internal startup incubator Area 120. Until recently, the Aloud website allowed interested vloggers to apply for access to an "early access program." At the time of publishing, it was closed off to new testers but promised be “working on bringing Aloud to YouTube Studio, and our goal is to make it available to as many YouTube creators as possible in the near future, including the creators who previously signed up for our waitlist.” A year-old demo video of the startup on Youtube shows that the Aloud software (at least at the time of creation) cannot yet create lip-synced videos. As Youtube spokesperson Jessica Gibby told The Verge, this feature is planned for 2024.

In February of this year, Youtube introduced a feature for making videos available in multiple languages. According to Youtube, channel operators with multilingual videos have seen a 15 percent increase in watch time. Since then, videos by YouTube superstar Jimmy “Mr. Beast” Donaldson, have been available in 14 languages. However, they have so far been dubbed by professional dubbing voices, some of which are well-known in the respective countries.

Opportunities for creators and marketers?

According to Advertising Age, Marc Hustvedt, head of the "MrBeast Company," emphasized the opportunities created by multilingual content: On the one hand, the reach potential for content creators, because after all, of the 1.2 billion people born after 2010, only two percent are from the USA. The majority of YouTube's audience comes from India.

On the other hand, brands and marketers will find it much easier to run global campaigns with creators in the future. "I believe that creators are currently more efficient and better able to reach a larger audience than traditional media. For example, eCommerce software outfitter Shopify reached 145 million viewers worldwide through a placement in one of MrBeast's recent videos.

Startups aplenty around the world

In addition to YouTube and Heygen, various other startups are currently working solutions of their own for AI dubbing. A cursory Google search yields a large number of providers, while Techcrunch reports that Deepdub from Israel and Papercup from the UK each raised USD 20m in funding last year. Flawless AI, another British startup, attracted attention in early 2023 with videos showing how AI can be used to easily rework dialog in movies.

As far as Meta and Tiktok go, there is nothing public at the moment indicating that they are working on their own solutions. Nevertheless, Tiktok presented an AI-based tool in June that is supposed to make video production easier. However, the script generator seems to be primarily aimed at potential advertising customers.