SpatialGen – A 3D Scene Generation Model Open-Sourced by Qunhe Technology

What is SpatialGen？

SpatialGen is a 3D scene generation model open-sourced by Qunhe Technology. Built on a diffusion model architecture, it supports generating temporally and spatially consistent multi-view images from text descriptions, reference images, and 3D spatial layouts. These images can then be transformed into 3D Gaussian scenes and rendered into roaming videos. Powered by a massive dataset of indoor 3D scenes, SpatialGen produces visually realistic images where objects maintain accurate spatial properties and physical relationships across different camera angles, enabling users to freely navigate scenes and enjoy immersive experiences. SpatialGen addresses the spatial consistency problem of existing video generation models, providing a powerful tool for AI-driven video creation.

Key Features of SpatialGen

• Multi-view image generation: Generates temporally and spatially consistent multi-view images from text, reference images, and 3D spatial layouts, ensuring objects maintain accurate spatial attributes and physical relations across views.

• 3D Gaussian scene generation: Converts generated multi-view images into 3D Gaussian scenes, supporting rendering of roaming videos for immersive 3D experiences.

• Spatiotemporal consistency: Ensures stable shapes and spatial relationships across frames, solving the spatial inconsistency issue common in video generation models.

• Controllable parametric layout generation: Supports controllable generation based on parametric layouts, enabling richer structural scene control to meet diverse user needs.

Technical Principles of SpatialGen

• Multi-view diffusion model: Built on a diffusion model architecture, SpatialGen samples multiple camera views in 3D space, transforms 3D layouts into 2D semantic and depth maps, and generates RGB images per view conditioned on text and reference images.

• Large-scale high-quality dataset: Trained on Qunhe Technology’s vast indoor 3D scene dataset, ensuring realistic visuals and accurate spatial relations in generated outputs.

• 3D reconstruction algorithms: Converts generated multi-view images into 3D Gaussian scenes, bridging the gap between 2D image generation and 3D scene reconstruction.

• Spatiotemporal consistency techniques: Special algorithms ensure temporal and spatial stability across views, avoiding shifts or inconsistencies between frames, and improving overall video quality.

Project Links

• GitHub: https://github.com/manycore-research/SpatialGen

• HuggingFace: https://huggingface.co/manycore-research/SpatialGen-1.0

Application Scenarios of SpatialGen

• Interior design and renovation: Generate multiple design schemes from user input or floor plans, helping designers quickly visualize and optimize solutions.

• Virtual Reality (VR) & Augmented Reality (AR): Create realistic 3D scenes for immersive VR/AR applications, such as virtual museums or tourist attractions, enhancing user interaction.

• Game development: Rapidly generate 3D environments like indoor spaces or city streets, accelerating game development and enriching scene diversity.

• Robotics training and simulation: Generate 3D environments such as homes or industrial workshops for robot training, providing diverse data to improve adaptability and performance.

• Film and animation production: Produce high-quality 3D scenes and animations, from futuristic cities to historical architecture, improving production efficiency and visual realism.