Tavus, a four-year-old generative AI startup that helps companies create digital “replicas” of individuals for automated, personalized video campaigns, confirms $18 million in new funding, It can be built into its own software, which has opened up its platform to third parties to integrate their technology.
Reports date back to August that Tavus had raised “approximately $18 million,” but details have not been released. The company confirmed to TechCrunch that it has indeed raised $18 million in a Series A round led by Scale Venture Partners, an early-stage VC firm that has previously backed companies like Box, HubSpot, and DocuSign. Other notable investors include Sequoia, which led Tavus' $6.1 million seed round last year and participated along with Y Combinator (YC) and HubSpot.
Video takes center stage
The generative AI movement is best exemplified by text-based search engines such as ChatGPT and text-to-image models such as DALL-E, and OpenAI is integrating these into a single all-singing platform. I'm in the middle of doing that. But if the past few months are anything to go by, generative AI could be on the cusp of another mini-revolution, with video taking center stage.
OpenAI recently debuted Sora, a text-to-video model that has the potential to transform the creative industry as we know it. But it's far from the only player, with tech giants like Google having been working on similar tools for several years, not to mention spending a significant amount of money over the past year on various realizations of generative AI. There are also a number of startups that have raised VC change. May intersect with video.
Tavus works with clients to create replicas of individuals through voice and facial cloning. The idea is for sales and marketing teams to use Tavus to send personalized videos to prospects at scale, or for product teams to create personalized walkthrough videos for new customer onboarding. It means that you can do it. It's all done through simple text-based prompts that leverage previously created digital replicas. Companies can also automate much of this by integrating Tavus with third-party systems like Salesforce and Mailchimp. For example, a customer who fills out an online form requesting more information about a product is immediately emailed a video with a sales representative's address. Give the prospect's name and explain next steps.
So far, Tavus has managed to secure some pretty big-name clients in its short time in business, including Salesforce and Facebook parent company Meta, and is co-founder and chief executive of Meta. CEO Hassan Raza said the company uses the platform to upsell each B2B customer through personalized demo videos.
Tavus as a platform
So far, Tavus is offered through a SaaS app through which customers create their own AI video templates. The onboarding process requires an individual, such as a CEO or sales executive, to record his 15-minute video based on a script provided by Tavus.
Cloning of Thevas in progress Image credit: Thevas
This is used to train the AI. Users then access the web editor and choose which parts of the video they want to personalize by defining variables such as location, executive name, company, and product. By tying Tavus to their CRM system, companies can tailor each of these variables to specific customer segments, such as customers who have expressed interest in a particular product.
Edit variables Image credit: Tavus
A company can create hundreds of these replicas, with different personnel involved and different backgrounds for different target markets.
The in-app editor allows you to generate any number of different scripts to attach to each use case without having to re-record the original video.
Tavus' various avatars Image credit: Tavus
While this core SaaS product is not being retired, Tavus today announced new updates to its technology, alongside the first in a suite of developer APIs that will allow third parties to integrate Tavus into their own applications. Remove the lid on the turbocharged version.
duplicate
The first aspect of Tavus' new developer platform is a “replica API” that creates “photorealistic” digital replicas with text-to-video generation. This allows companies to clone people (such as marketing directors or CEOs) using a new proprietary model created by Tavus called Phoenix. This model is based on a deep learning technique called Neural Radiance Field (NeRF). This allows him to generate a 3D structure of a person from a 2D image in just a few minutes.
“Essentially, you can create an entire video with just two minutes of training data, which is a big step forward from how we used to do personalization at scale,” Raza told TechCrunch. Told. “And all you have to do is record two minutes of training data, and we'll create a perfect replica of you. And once you have a replica, you can create as many videos as you like from one, two, or 1,000 scripts. You can create one.”
Simulation showing how Tavus maps a user's face to create a realistic replica Image credit: Tavus
Output: Tavus' Phoenix model builds a 3D model using 2D video input via Neural Radiation Field (NeRF) Image credit: Tavus
The first Replica API relies on the full power of the Phoenix model to capture an individual's facial movements, including cheeks, nose, eyebrows, and lips.
“Moving the entire face increases realism, naturalness, and quality. When speaking, the face expresses emotion beyond lip movements,” Raza explained. “If you want to generate an entire video from a script, one that looks natural and of incredibly high quality, we recommend using the Replica API.”
However, Tavus has also developed a number of additional APIs, including one specifically for lip-syncing. One is for dubbing. The other is for running large scale personalized video campaigns.
According to Raza, the LipSync API has a “low implementation cost” and is suitable for situations where “high quality and realism are not required.”
Meanwhile, the Dubbing API also uses a lip-sync model, but also includes multilingual voice clones, so monolingual users can create video campaigns in any number of languages using their own voices. can be sent. In this example, considering that the majority of the video remains the same, the API allows you to easily replace the lip movements to match different sounds coming from the user's mouth. This could be useful, for example, to the creators of his suite of video editing software who want to allow users to add lip-syncing, editing, and dubbing to their videos.
And the Video Campaign API essentially bundles the Replica API with a set of additional tools such as hosting, variable mapping, thumbnails, and analytics for users looking to start large-scale video campaigns.
“We provide the ability for any developer to instantly deliver an end-to-end video campaign experience within their own solution,” said Raza. “Whereas the Replica API and the Lip Sync API are more of an 'as-a-service model,' the Campaign API provides the tools to easily build an AI video campaign platform.”
Raza declined to say that some of its early users are using the Tavus platform, but said it is “working with one of the largest video platforms” when it comes to customer engagement. “They're trying to offer this to the millions of customers who already use their platform to create videos every day,” Raza said.
The deepfake dilemma
Intuitively, a platform like Tavus is ripe for exploitation. After all, what's stopping you from uploading an existing video and creating a digital replica? Deepfakes are certainly a growing concern in the burgeoning AI movement, but avoiding deception Raza said that checks are being carried out. For example, if a user submits a 2-minute training video of her, she must also provide a specific verbal consent statement. Make sure this consent statement matches the audio in your training video.
“We run these checks automatically and every replica that passes the automated checks undergoes human checks to ensure safety,” Raza said.
It's easy to see how Tavus works as a standalone SaaS app, but now that it's a platform that can be accessed by any number of companies via an API, who's in control of the validation? As it turns out, that's what Tavus is. The company wants to stay involved in validation work, even if it's just providing the engine to third-party developers.
“We carry out the same checks and are responsible for verification. [the] The same goes for APIs,” Raza continued.
augmented reality
OpenAI has become pretty much the public face of generative AI, but there's plenty of room for different players to bring something different to the mix. Indeed, while the main purpose of DALL-E and his OpenAI's recently released Sora model is to enable people to create visuals from text prompts, Raza says the raison d'être of Tavus is more about how individuals create visuals. It states that the goal is to “expand” the reality of
“We see a future where everyone wants to have a digital replica of themselves, and they control it and have full authority over it,” Raza said. “And it's going to be important that it actually captures more of your personality, your gestures and your characteristics. That's how we're going to look at things going forward – things that don't exist.” There will be models to create, and then there will be models to augment reality.”
Raza, who has $18 million in the bank, said the latest cash injection would be used to “add fuel to the fire that is already burning” at Thevas Tower.
“We are an AI research company, so we want to be able to continue developing new models like Phoenix,” Raza said. “But just to maintain our growth, we continue to have a lot of demand. And we continue to hire machine learning and engineering teams to support the developer and his SaaS customers.” I want to be able to do that.”