Capturing Studio-Quality Avatars With a Phone Camera

📄 Full Paper

💬 Ask

A new AI technique can generate hyper-realistic, fully animatable 3D avatars from just a few quick videos captured with a smartphone. The new method, presented in a recent paper by researchers at Meta Reality Labs, overcomes several limitations of previous methods.

Traditional photorealistic avatars require complex and expensive studio equipment like the LightStage. While recent approaches can create avatars from phone scans, they have several drawbacks, such as:

• Lighting being baked into the avatars: This means the lighting conditions in the avatar are fixed and cannot be changed.
• Lack of detail: Phone scans often lack detail in areas like the back of the ears.
• Missing regions: Phone captures usually miss certain areas like the back of the head, leading to gaps in the 3D model.

The new method tackles these issues by generating a studio-quality texture map, which represents the surface of the avatar, from a phone capture. The researchers developed a two-step process:

1. Studio-like texture generation: The first step involves using a StyleGAN2, a powerful generative model, to generate a low-resolution texture map with studio-like illumination and inpainted missing regions. The StyleGAN2 is trained on a large dataset of phone-captured texture maps and then fine-tuned using a small set of studio-captured texture maps.

2. Facial detail enhancement: The second step involves enhancing the low-resolution texture map with a diffusion model. Diffusion models are known for their ability to generate high-quality images, and the researchers use a diffusion model specifically designed to generate facial details. The model is trained to add realistic details to the texture map, taking into account the gradient of the phone-captured texture map. This helps to preserve the original facial details while ensuring the final texture map has studio-quality illumination and no missing regions.

Once the studio-quality texture map is generated, it can be used to create a high-quality avatar using a universal prior model like Authentic Volumetric Avatars (AVA). This allows for realistic animation and the ability to render the avatar from any viewpoint.

The researchers conducted several experiments to evaluate the performance of their method. They compared it to other state-of-the-art methods for generating avatars from phone captures and found that their method outperforms previous approaches in several metrics, including visual quality, accuracy, and identity preservation.

This new technique is a significant advancement in the field of avatar creation and has the potential to make high-quality, personalized avatars more accessible to a wider audience. As the technology continues to improve, we can expect to see even more realistic and expressive avatars generated from everyday phone captures.

Chat about this paper

To chat about this paper, you'll need a free Gemini API key from Google AI Studio.

Your API key will be stored securely in your browser's local storage.