AI Papers Reader

Personalized digests of latest AI research

View on GitHub

AI Can Now Compose 3D Videos From Text Prompts

Imagine a world where you can describe a video in words, and AI will bring your vision to life. This future is closer than ever, thanks to a new AI model called C3V (Compositional 3D-aware Video Generation). Researchers at Microsoft Research Asia and the University of Science and Technology of China have developed this groundbreaking model that composes individual 3D concepts into compelling videos.

C3V leverages the power of Large Language Models (LLMs) to break down complex textual prompts into simpler components. Think of it as a film director assigning tasks to different departments, each responsible for a specific aspect of the production. For example, a prompt like “a cat chases a mouse across the kitchen” would be decomposed into “kitchen,” “cat,” and “mouse chasing.”

Each concept is then generated separately using a pre-trained expert model, producing a 3D representation. For the kitchen, a scene model might be used; for the cat and mouse, object models; and for the chasing, a motion model. This approach allows for much more precise control over each concept compared to existing methods that try to generate everything at once.

The next step is to compose these individual concepts into a cohesive whole. This is where C3V gets truly innovative. Instead of relying on LLMs to directly estimate the trajectory of each object, the researchers have devised a “step-by-step” reasoning approach. First, the locations of the starting and ending points of the object’s trajectory are determined. Then, the LLM generates a path between these points, guiding the object’s movement.

Finally, 2D diffusion models are used to refine the composition, ensuring that the final video adheres to a natural image distribution. This process involves optimizing the scales, locations, and rotations of the objects based on the generated trajectory.

The results are impressive. C3V can generate videos with diverse motion and high visual quality. You can control the appearance of specific characters, the movement of viewpoints, and even edit individual concepts after the video is created.

C3V represents a significant leap forward in text-to-video generation. It paves the way for more creative and personalized video content, opening up new possibilities in entertainment, education, and communication. This is just the beginning of a new era in AI-powered video creation, where the power of imagination is only limited by our ability to describe it.