YouTube Trials 'Dream Track' AI Tool that Clones Singers' Voices

Google is testing a new generative AI feature for YouTube that will let you clone singers’ vocals using AI versions of their singing voices as soundtracks for videos.

The new feature, dubbed ‘Dream Track’ uses a music generation model created by the tech giant’s AI division to create the unique 30-second clips for a small group of US creators using YouTube Shorts. 

These users can choose from a total of nine artists including Charli XCX, John Legend and Sia, and generate lyrics, a backing track and AI-generated voices through a single text prompt.

The idea is for these tracks to be used with YouTube’s TikTok-style Shorts service. That’s also where YouTube recently announced a new AI feature called Dream Screen – which generates videos and photos to use as backgrounds.

Today with @YouTube, we’re announcing Lyria: our most advanced music generation model to date. 🎶

We’re also releasing 2️⃣ AI experiments in close collaboration with participating artists and creators to bring their ideas to life responsibly. →https://t.co/i9ve66A5rv pic.twitter.com/h3mdp9R3bq

— Google DeepMind (@GoogleDeepMind) November 16, 2023

“By simply typing an idea into the creation prompt and selecting a participating artist that appears in the carousel, an original Shorts soundtrack featuring the AI-generated voice of that artist will be produced for the creator to use in their Short,” said YouTube executives Lyor Cohen and Toni Reid.

“At this initial phase, the experiment is designed to help explore how the technology could be used to create deeper connections between artists and creators, and ultimately, their fans."

How does YouTube Dream Track AI Work?

YouTube's Dream Track AI is an experimental feature powered by Google DeepMind's advanced music generation model, Lyria. Currently in a closed beta test, it allows creators to generate 30-second musical tracks in the style of specific artists for use in their YouTube Shorts videos. 

At the heart of Dream Track is Lyria, an AI model trained on a massive dataset of music and musical styles. This allows it to understand and mimic the melodic, rhythmic, and vocal characteristics of various artists.

Read: Deepmind LLM Just Solved an Unsolveable math Problem

Creators start by describing their desired music using a text prompt. This could be anything from a general mood ("upbeat and danceable") to a specific theme ("love song in the style of John Legend").

Dream Track currently offers music in the style of nine participating artists, including Demi Lovato, Alec Benjamin, and Charli XCX. Creators choose the artist whose style they want to emulate.

Once the prompt and artist are selected, Lyria goes to work. It analyzes the text prompt and the artist's musical style, then generates a unique 30-second audio track featuring an AI-generated vocal performance that mimics the chosen artist.

While creators cannot directly edit the lyrics or melody, they can choose from different variations of the generated track based on tempo, energy, and overall feel.

Dream Track AI in the Music Industry

In the blog post, YouTube shared samples of what AI-generated music experiments would look like, along with statements from the singers who gave the video-sharing app permission to clone their voices.

"The development of AI technology is rapidly changing the way we navigate the landscape and I believe as artists we need to be a part of shaping what that future looks like,” said Demi Lovato.

“I am open-minded and hopeful that this experiment with Google and YouTube will be a positive and enlightening experience,” the pop singer added.

Many other artists taking part in the experiment shared similar views to Lovato, sharing their excitement for being involved in the project while also remaining cautious of AI’s role in the Music industry. 

“When I was first approached by YouTube I was cautious and still am, AI is going to transform the world and the music industry in ways we do not yet fully understand,” said Boom Clap singer Charlie XCX. 

“This experiment will offer a small insight into the creative opportunities that could be possible and I’m interested to see what comes out of it.”

The nine artists who have agreed to have their voices cloned as part of the experiment are not the only singers to embrace the potential of AI in the music industry. 

In April, Genesis and Oblivion singer Grimes made headlines after saying she would allow anyone to her voice to generate AI music as long as she was given royalties on AI-generated songs that make it big. 

“I’ll split 50% royalties on any successful AI-generated song that uses my voice,” Grimes said in a Tweet posted on Twitter. “Same deal as I would with any artist I collaborate with.”

“I'm just curious about what even happens and interested in being a Guinea pig. Feel free to use my voice without penalty,” she said, claiming she has no label and “no legal bindings.”

YouTube's crackdown on AI

AI-generated songs have taken the internet by storm in recent months, with deep fake songs of popular artists becoming incredibly popular on video-sharing apps like YouTube and TikTok. 

But YouTube has previously consistently removed AI-generated songs that are uploaded without the permission of the artist or their label. 

When 'Heart On My Sleeve' went viral earlier this year for mimicking the voices of f multi-platinum artists Drake and the Weekend, for instance, YouTube was quick to remove it after Universal Studios called for the song to be removed from streaming services. 

“We removed the video after receiving a valid copyright notification for a sample included in the video,” YouTube spokesperson Jack Malon said in a statement at the time.

“Whether or not the video was generated using artificial intelligence does not impact our legal responsibility to provide a pathway for rightsholders to remove content that allegedly infringes their copyrighted expression.”

YouTube is also in the process of introducing a tool that will allow music labels and distributors to flag content that mimics an artist’s “unique singing or rapping voice."

It will also soon require creators to disclose when they have made realistic-looking “manipulated or synthetic” content, including AI-generated material.

“We’ll make it possible to request the removal of AI-generated or other synthetic or manipulated content that simulates an identifiable individual, including their face or voice,” the platform said in the blog post by Jennifer Flannery O’Connor and Emily Moxley, product management VPs at the company. 

“Not all content will be removed from YouTube, and we’ll consider a variety of factors when evaluating these requests,” they added.