World Labs, a startup founded by AI pioneer Fei-Fei Li, has announced its first project, an AI system that can generate video game-like 3D scenes from a single image.
Many AI systems can convert photos into 3D models and environments. But World Labs scenes are unique in that they are interactive and changeable.
“[Our tech] You can step into any image and explore it in 3D,” World Labs wrote in a blog post. “Everything is generated beyond the input image.”
World Labs' systems transform images into interactive, explorable 3D scenes. Image credit: World Labs
In a demo on World Labs' website, the AI-generated scenes, which anyone with a keyboard and mouse can explore, look impressive, if a bit cartoonish. They are rendered live in the browser and feature controllable cameras with adjustable simulated depth of field (DoF). The stronger the DoF effect, the more blurry the background objects will appear.
World Labs' systems are part of a new category of AI called “world models.” Many of these models can simulate games and 3D environments, but suffer from artifacts and consistency issues. For example, startup Decart's Minecraft-simulating world model, Oasis, has a low resolution and quickly “forgets” the level layout.
In contrast, World Labs' approach ensures that the scene remains the same after it is generated and follows the fundamental laws of physics. That is, the scene has solidity and a sense of depth.
World Labs' systems can also apply interactive effects and animations to the scene, such as changing the color of objects or dynamically lighting the background.
Scene generated by World Labs systems. Image credit: World Labs
“Most generative AI tools create 2D content such as images and videos,” World Labs writes. “Generating in 3D instead offers greater control and consistency. This will change the way we create digital representations of the physical world in movies, games, simulators, and more.”
There is certainly room for improvement. The World Labs scene is not fully explorable. Movement is limited to a small area. (If you try to move outside of it, you will hit the boundary.) You may also experience rendering errors, such as objects blending together unnaturally.
However, World Labs says this is only an “initial preview.”
“We are working hard to improve the size and fidelity of our generated worlds and experiment with new ways for users to interact with them,” the startup wrote on its blog.
Some of the demos on the World Labs site. Effects can be applied to scenes in real time. Image credit: World Labs
Founded earlier this year, World Labs has raised $230 million in venture capital from investors including Andreessen Horowitz (a16z), Ashton Kutcher, Intel Capital, AMD Ventures, and Eric Schmidt. Valued at more than $1 billion, the company hopes to have its first product ready in 2025.
Beyond interactive scenes, World Labs plans to build useful tools for artists, designers, developers, filmmakers, engineers, and other professionals. It targets a wide range of customers, from video game developers to movie studios.
“We already have the ability to create virtual, interactive worlds, but it takes hundreds of millions of dollars and hundreds of millions of dollars and tons of development time,” World Labs co-founder Justin Johnson said on a recent episode of the a16z podcast. It will take a while,” he said. “[World models] Instead of just getting images and clips, you'll now be able to get fully simulated, vibrant, interactive 3D worlds. ”