I’ve been using the ElevenLabs TTS (Text To Speech) tool since they were first in beta a couple years ago, and have watched them get progressively better in quality and capabilities.
Their most recent updates for voices, Sound FX and Music generation have really surpassed all previous capabilities. They now currently have tools for various audio needs; Voice/VO, Audiobook, Conversational/API, Music, Sound FX and Regional Voice Dubbing.
I won’t be going too deep in this overview, but I will provide you with plenty of links to dig deeper on your own if you’re interested in any one tool.
Voices
Not only have the ElevenLabs engineers been giving users more controls over synthesized voices and how they’re played-back on a script, but now there’s more expressive voices and commands in the Voice Design v3 release. They’ve also expanded their Audiobook capabilities, provide an API for live conversational voices and now have a Voice Changer tool for altering recorded voices.
Text to Speech
You can insert bracketed [commands] for emotions, intensity, accents, natural pauses, sighs and chuckles, whispers and sounds. (Even AI generated farts) LOL
- For a reference guide to ElevenLabs v3 Prompting tags, visit: https://elevenlabs.io/docs/best-practices/prompting/eleven-v3#voice-related
- And here’s a list of the best suggested voices for Voice Design v3: https://elevenlabs.io/app/voice-library/collections/aF6JALq9R6tXwCczjhKH
I put this to the test right away with a short script, and a voice I thought could match an avatar I had in mind.
I then I uploaded to a stock avatar in HeyGen and this is the raw result that came out from my single pass audio – straight out of HeyGen. No further editing was used:
I can see this is going to add a lot more personality and realism to my VO reads for video and animation projects. (Well, maybe not the burps and farts) 😉
Voice Changer
The new Voice Changer tool maintains all the fluctuations and nuances of the original audio – whether its a live voice recording (like your own) or a synthetic one like I created here:
Original ElevenLabs generated female voice:
Voice Changer regeneration to a Northern British male voice, created from the exported audio file above:
I can see several useful applications of this tool – especially if you want a specific reading of a character and you can get all the tone and inflections by recording your own voice and then changing it to your character voice.
Voice Dubbing
Using the ElevenLabs Dubbing Studio lets you swap your native language video into another spoken language. You simply upload your video or select a YouTube video to process and select you languages.
You can choose to launch the Dubbing Editor to refine your project as well. You can regenerate segments or extend them or re-align them. Now I haven’t tried anything commercially recorded with music or other effects so I don’t know how wel it maintains the integrity of the audio compared to HeyGen (which also changes the mouth movements to match the dubbing) but I’m going to do a comparison test soon.
Here’s the dubbed voice example. You can hear just slight changes in the voice inflections but the sighs and laughs were removed.
Sound FX
I’ve been using ElevenLabs Sound FX for awhile now and it has been a real game changer for basic sounds. Some things you just can’t get it to produce exactly what you want but for getting some quick stock content you can mix/layer/modify in your audio editor, it really speeds up the workflow.
This is an edit from my conference session last year at the Design + AI Summit where I recorded my entire 40 minute presentation with myself as a clone in front of my “TED Talk” AI Audience. I share some of the tricks I used for the ambient sounds and SFX I used for my main presentation in this video:
That gives you an idea how far you can go with using one tool to combine your VO and SFX needs.
I also featured ElevenLabs SFX in my last article on PVC; AI Tools: Video & Animation Come to Midjourney:
Another fun tool they have that isn’t as well known (it’s actually buried under “Audio Tools” in the ElevenLabs sidebar) is their audio/SFX Soundboard app with looper, called “SB1”. You can go play with it online now without an account here: https://elevenlabs.io/sound-effects/soundboard
Eleven Music
This is a big one on many levels! Though it’s not the same quality of a professionally-recorded and produced music soudtrack, I think it’s close to as good as most of the stock audio tracks from providers like Pond5 or Soundstripe. And it’s quite customizable and editable right in ElevenLabs. For social media and short industrial projects, this is a game-changer for content producers.
Eleven Music takes descriptions to feed the engine to produce a soundtrack (with or without lyrics).
I’ve experimented a bit with instrumental only tracks to do some testing and demos for clients, and it’s so far passed the AI sniff test in real-world productions.
You can regenerate any part of the musical piece by either selecting the segment and regenerating or adjust the timing and length of each segment by dragging the end handles and moving them – then regenerate.
Once you’ve made your edits and re-regenerate your track, it will be the correct length and movement throughout your video project. It exports MP3, WAV, M4A and FLAC formats for easy editing in your NLE.
Here’s the example of the finished edit to this particular track:
There’s still a bit of non-descript digital noise and over-reverb and a metalic unnatural sound to the music. I couldn’t just “listen” to this, but if it’s buried underneath VO, you really can’t hear it as much. I tend to remaster all my audio on client video productions anyway, so I tend to warm it up as well as any AI voices I use to sound more warm and real.
Just for shits & giggles, I had it generate a silly pop song based on the prompt “I would like a pop song that talks about how AI is taking away our jobs.” – here is what it produced (1 of 2 different variations) with lyrics.
Here are the lyrics:
DIGITAL NIGHT
Monday morning, coffee in my hand
Scrolling headlines, trying to understand
Robots learning fast, algorithms in command
Feels like the future’s got a tighter plan
They said convenience, efficiency
But now I’m asking, what’s left for me?
When the code writes itself, and the robots take our place
I’m displaced, left in the wake of a silicon embrace
Are we heroes or ghosts in a digital space?
Tell me, who survived when machines won the race?
Warehouse floors, lines moving without a soul
Customer service bots with a flawless protocol
My nine-to-five’s dissolving in a data scroll
And my reflection’s fading on the factory floor
We built these circuits to serve our design
Now they rise up, cross that thin line
But inside every heart there’s a spark divine
Can we reclaim the world we left behind?
When the code writes itself, and the robots take our place
I’m displaced, left in the wake of a silicon embrace
In the shadows of progress, we still hold our grace
We’ll rewrite our story in the human race
Keep the spark alive
Don’t let them steal our light
And here’s the generated MP3 track:
Finally, I did try a few short videos using the Video to Music (Alpha) tool but I really didn’t care for the results, as they were either too short or not really a match I would choose. I’ll take more time to play with this feature and report back in a later follow up article.
There is just so much to explore now at ElevenLabs that will require much further playing around to see what truly works and what falls-short. By the time I get through everything I’m sure it will be updated and corrections and refinements made to the various tools.
NOTE: ElevenLabs seems serious about IP so they’ve included a Music Terms sheet of legal and non-approved prompts for your reference here: https://elevenlabs.io/music-terms