FaceShot – A portrait animation generation framework jointly launched by Tongji University and Shanghai AI Lab, among others

What is FaceShot

FaceShot is a novel training-free portrait animation generation framework developed by Tongji University, Shanghai AI Lab, and Nanjing University of Science and Technology. It leverages an appearance-guided landmark matching module and a coordinate-based landmark relocation module to generate precise and robust landmark sequences for a wide range of characters. Utilizing semantic correspondence from latent diffusion models, it enables facial motion generation across diverse character types. These landmark sequences are then input into a pre-trained landmark-driven animation model to produce animated videos. FaceShot breaks the limitations of relying on realistic portrait landmarks, making it suitable for any stylized characters and driving videos. It can also serve as a plugin compatible with any landmark-driven animation model, significantly enhancing overall performance.

Key Features of FaceShot

• Character Animation Generation: Generates smooth and natural facial animations for various types of characters while preserving their original appearance.

• Cross-domain Animation: Supports animating non-human characters (e.g., toys, animals) driven by human videos, expanding the application scope of portrait animation.

• Training-free: Produces high-quality animations without the need for additional training or fine-tuning for each character or driving video.

• Compatibility: Seamlessly integrates as a plugin with any landmark-driven animation model.

Technical Principles of FaceShot

• Appearance-guided Landmark Matching Module: Based on semantic correspondences learned from a latent diffusion model and prior appearance knowledge, this module generates precise facial landmarks for arbitrary characters. Diffusion features are extracted from reference and target images using the DDIM inversion process. Image prompts help reduce appearance differences across domains. Cosine distance is used to ensure semantic consistency in landmark matching, and an appearance gallery is introduced to further optimize the results.

• Coordinate-based Landmark Relocation Module: This module captures subtle facial motions from the driving video through coordinate system transformations, generating aligned landmark sequences. It consists of two stages: global motion and local motion. The global motion stage estimates overall face translation and rotation, while the local motion stage relocates the relative movements of facial components (eyes, mouth, nose, eyebrows, and facial contours) using point-wise transformations. Using simple coordinate transformation formulas, the module accurately captures both global and local facial movements to produce stable landmark sequences.

• Landmark-driven Animation Model: The generated landmark sequences are input into a pre-trained animation model (e.g., MOFA-Video) to produce the final animated video. The landmark sequence is provided as an additional condition to the animation model’s U-Net, ensuring precise motion tracking. This approach enables the model to generate animations that align with the driving video, preserve the character’s visual identity, and achieve high-quality portrait animation.

Project Links

• Official Website: https://faceshot2024.github.io/faceshot/

• GitHub Repository: https://github.com/open-mmlab/FaceShot

• arXiv Paper: https://arxiv.org/pdf/2503.00740

Application Scenarios for FaceShot

• Film and Entertainment: Create vivid animations for characters in movies and TV series, enhancing visual appeal.

• Game Development: Rapidly generate animations for game characters to boost expressiveness and engagement.

• Education: Make educational content more engaging and interactive to improve learning experiences.

• Advertising and Marketing: Generate animated brand mascots to enhance brand image and user engagement.

• VR/AR Applications: Produce virtual character animations to increase immersion and interactivity.