Modern generative models make great demos, but will they really change the way movies and TV shows are made? Filmmaking and VFX experts say in the short term they won't, but in the long term they could literally change things beyond our imagination.
At a panel at SIGGRAPH in Denver, Nikola Todorovic (Wonder Dynamics), Freddy Chavez Olmos (Boxel Studio), and Michael Black (Meshcapade, Max Planck Institute) discussed how generative AI and other systems can change (but not necessarily improve) the way media is created today. Their consensus was that while it's natural to question the short-term usefulness of these tools, the rate of innovation means we need to be prepared for rapid change beyond that.
One of the first topics we tackled was the impracticality of today's video generators.
Todorovic pointed out, “There's a misconception that AI is a one-click solution and can create final VFX shots, but that's simply not possible. You can probably get there, but without editing capabilities, there's not much to gain from the black box. Right now, UX is still being discovered, and these research companies are just starting to learn 3D methods and filmmaking terminology.”
Black pointed out that language fundamentally lacks the ability to explain some of the most important aspects of visual creativity.
Final shots, motion capture data, masks, and 3D environments produced by Wonder Studio. Images courtesy of Wonder Dynamics
“There are names for the classic things that we can define, like yoga poses or ballet poses, but most things we do don't have names,” he said. “And there's a reason for that, and it's because humans actually have generative models of behavior in their minds. But I don't have a generative model of an image in my head. If I want to describe to you what I see, I can't project it from my eyeball, and I'm not a good enough artist to draw it for you. So I have to use words. And we have a lot of words to describe the visual world. But if I want to describe a certain movement to you, I don't have to describe it in words, I can just do it for you, and your motor system will look at me and try to understand it. So I think there are biological reasons, neuroscientific reasons, why we can't put words to all our movements.”
This may seem a bit philosophical, but the result is that a text-based image prompt system offers fundamentally limited control – even the hundreds of technical and artistic terms used every day on set and in post-production are inadequate.
Image credit: Devin Coldewey
Chavez Olmos noted that as a Mexican native, he had few opportunities to participate in the filmmaking world because all the funding and expertise was concentrated in Los Angeles. But AI expertise (and the demand for it) is more widespread, he said. “I had to leave Mexico because there were no opportunities here. Now I think there are the same opportunities for people who don't have to go abroad.”
But Black worries that sudden access to these processes could have unintended consequences in the short term.
“You give someone a powerful car, but you're not going to turn them into an F1 driver. It's a bit like the situation now. People are saying they're going to make movies. They're going to be like, frankly, shit,” he said. “Democratization is really [Chavez Olmos] “The power of saying that is that maybe new voices will get opportunities they wouldn't have otherwise. But the number of people making really good films, in my opinion, will still be small.”
Example assets for a virtual character shot – a girl model walking between waypoints that correspond to real-world spaces. Image credit: Fuzzy Door
“The real revolution,” he continued, “the real power that we see with AI is that we're going to see a whole new genre of entertainment, and I don't know exactly what it'll be. I predict it'll be something between video games and movies and real life. The film industry is passive storytelling. I sit there and observe. It's like theater or podcasts. I'm a passive recipient of entertainment. But in our daily lives, we tell each other stories, we chat about what we did over the weekend, and so on. And it's a very active kind of interactive storytelling.”
But before that happens, Chavez Olmos said he expects a more traditional acceptance curve for AI-generated images and actors.
“I think it'll be the same reaction as when people saw the first Final Fantasy or Polar Express. There might still be something missing, but people will start to accept these movies. And instead of being a fully CGI movie, it'll be a fully AI movie. I think we'll see this as soon as the end of the year. I think people will get over it and accept, 'OK, this is AI.'”
“The key is, and Pixar has taught us so clearly, it's all about story. It's all about connecting with the characters. It's about heart. If the movie has heart, it doesn't matter if the characters are AI. I think people will enjoy the movie,” Black said. “That doesn't mean we don't want human actors. It's exciting to know that they're real human beings, just like us, but so much better than us. Seeing humans performing at their best is inspiring for all of us, and I don't think that's ever going to go away.”