Matrix-3D – 3D World Model Open-Sourced by Kunlun Wanwei

What is Matrix-3D？

Matrix-3D is a framework developed by Kunlun Wanwei’s Skywork AI team for generating explorable panoramic 3D worlds. It combines panoramic video generation with 3D reconstruction, allowing high-quality, fully explorable 3D scenes to be created from a single image or text prompt. Leveraging a trajectory-guided panoramic video diffusion model and two 3D reconstruction methods (a fast feed-forward network and a high-quality optimization method), Matrix-3D delivers large-scale, highly consistent 3D scene generation. It supports both text and image inputs, offering efficiency and strong generalization capabilities. The accompanying Matrix-Pano dataset provides strong support for research.

Key Features of Matrix-3D

• Panoramic Video Generation – Generates high-quality panoramic videos from a single image or text prompt, with support for user-defined camera trajectories.

• 3D Scene Reconstruction – Offers two reconstruction methods: a fast feed-forward network for real-time needs and an optimization-based approach for high-quality results.

• Multi-Input Support – Accepts both text and image inputs, allowing users to generate corresponding 3D scenes as needed.

• Large-Scale Scene Generation – Produces expansive 3D scenes with full 360° free exploration, surpassing other methods in exploration range.

• High Controllability – Allows users to define custom generation trajectories and endlessly extend existing scenes.

Technical Principles

1. Trajectory-Guided Panoramic Video Generation – Uses mesh renderings as conditional inputs to train a video diffusion model. The model generates panoramic videos based on user-defined camera paths, ensuring spatial consistency and geometric accuracy.

2. Converting Panoramic Videos to 3D Scenes – Based on a Transformer architecture, directly predicts 3D geometry attributes from the latent features of the generated panoramic video, enabling fast reconstruction suitable for real-time applications.

3. Optimization-Based Method – Enhances generated panoramic videos with super-resolution and applies 3D Gaussian Splatting for detail-rich, high-quality 3D scenes—ideal for visually demanding scenarios.

4. Matrix-Pano Dataset – Addresses the scarcity of 3D scene data with a large-scale synthetic dataset containing 116,759 high-quality static panoramic video sequences, each with camera trajectories and annotations. The dataset’s diversity and quality strongly support model training.

5. Panoramic Representation – Uses panoramic images as an intermediate representation, covering a 360° horizontal and 180° vertical view. By stitching panoramic images from multiple positions, it generates panoramic videos containing all necessary information for 3D world creation.

Project Links

• Official Website: https://matrix-3d.github.io/

• GitHub Repository: https://github.com/SkyworkAI/Matrix-3D

• HuggingFace Model Hub: https://huggingface.co/Skywork/Matrix-3D

• Technical Paper: https://github.com/SkyworkAI/Matrix-3D/blob/main/asset/report.pdf

Application Scenarios

• Game Development – Rapidly generate high-quality 3D game environments, shortening development cycles and enhancing personalized player experiences.

• Film Production – Create realistic virtual environments and effects, reducing shooting costs and assisting in storyboard design and scene previews.

• Virtual Reality (VR) & Augmented Reality (AR) – Fully explorable 360° 3D scenes for virtual tourism and AR applications, boosting immersion.

• Robotics Navigation & Autonomous Driving – Generate complex 3D environments for training and testing navigation systems, improving decision-making safety.

• Education & Training – Create virtual laboratories and realistic training simulations for teaching and skills development, enhancing learning effectiveness.