PhysGen3D – An Interactive 3D Scene Creation from a Single Image, Jointly Launched by Tsinghua University and Other Universities

What is PhysGen3D?

PhysGen3D is an innovative framework capable of transforming a single image into an interactive 3D scene and generating physically realistic videos. By integrating image-based geometry and semantic understanding with physics-based simulation, it infers the 3D shape, pose, physical, and lighting properties of objects from a single image to create an image-centric digital twin. Furthermore, it simulates the counterfactual physical behaviors of objects using the Material Point Method (MPM) and seamlessly integrates dynamic effects into the original image, producing visually realistic results.

The main functions of PhysGen3D

• Creating Interactive 3D Scenes from a Single Image: PhysGen3D can convert a single image into an interactive 3D scene, in which users can simulate different physical behaviors.
• Precisely control the initial conditions of objects: Users can specify initial conditions such as the speed and material properties of objects, thereby achieving fine-grained control over the generated video results.
• Generate physically realistic videos: By combining image-based geometry and semantic understanding with physics-based simulation, PhysGen3D can generate videos that are visually realistic in terms of dynamics and lighting while being physically plausible.
• Dense 3D tracking: Enables precise 3D tracking of objects within the scene.
• Video editing: Allows for the exchange of objects between different scenes or the removal of certain objects while keeping their initial positions unchanged.
• Camera control: Supports video generation from different perspectives.
• Generate videos from paintings: Can process other types of inputs, such as generated images and paintings, to create videos.

The Technical Principles of PhysGen3D

• 3D Scene Reconstruction: Starting from a single image, PhysGen3D utilizes advanced image-based geometry and semantic understanding techniques to infer the 3D shape, pose, physical, and lighting properties of objects, while simultaneously reconstructing the geometry and appearance of the background. By integrating outputs from multiple pre-trained vision models, it creates an image-centric digital twin.
• Physical Simulation: In terms of physical simulation, PhysGen3D adopts the Material Point Method (MPM), a point-voxel-based framework designed to simulate counterfactual physical behaviors of objects in images. By accurately inferring the physical properties of objects, PhysGen3D achieves a high level of realism and stability in the simulation environment. Users can specify initial conditions such as object velocity and material properties to achieve fine-grained control over the generated video results.
• Physics-Based Rendering: After completing the dynamic simulation, PhysGen3D calculates vertex motion through motion interpolation and deforms the mesh. It then uses optimized physically based rendering (PBR) materials and performs physics-based rendering with Mitsuba3 under environmental lighting. To avoid transferring the entire static background into the rendering pipeline, PhysGen3D constructs a 3D shadow capture surface and employs a two-pass shadow mapping technique to extract shadow and global illumination effects. Finally, the foreground objects and shadows are composited onto the restored background to generate the final video.

Project address of PhysGen3D

• Project official website: https://by-luckk.github.io/PhysGen3D/
• GitHub repository: https://github.com/by-luckk/PhysGen3D
• arXiv technical paper: https://arxiv.org/pdf/2503.20746

Application scenarios of PhysGen3D

• Film and Television Production and Special Effects: PhysGen3D can be used for special effects generation in film and television production, helping creators quickly build dynamic scenes with physical realism.
• Virtual Reality and Augmented Reality: In virtual reality (VR) and augmented reality (AR) applications, PhysGen3D can generate interactive 3D scenes from a single image, providing users with an immersive experience.
• Education and Training: PhysGen3D can be used in the field of education to help students better understand physical concepts.
• Game Development: PhysGen3D offers new possibilities for game development. Developers can quickly build levels and scenes in games by leveraging its ability to generate dynamic scenes with physical realism from a single image.
• Advertising and Marketing: In the advertising and marketing field, PhysGen3D can be used to create engaging dynamic ad content. It can generate dynamic videos from product photos, showcasing the product’s usage scenarios and physical characteristics to capture consumers’ attention.