Matrix-Game – The first industrial spatial intelligence large model open-sourced by Kunlun Wanwei

What is Matrix-Game?

Matrix-Game is the industry’s first open-source spatial intelligence model with over 10 billion parameters, developed by Kunlun Wanwei. It is an interactive video generation model within the Matrix-Zero world model framework. Leveraging a two-stage training strategy, Matrix-Game generates coherent and controllable interactive videos based on user input. It excels in fine-grained interaction control, high-fidelity visual and physical consistency, and strong generalization across diverse scenarios. Matrix-Game sets a new benchmark for building general-purpose virtual world infrastructures and is designed for applications in virtual game development, film production, and metaverse content creation.

Key Features of Matrix-Game

• Controllable Video Generation:
  Users can explore and manipulate detailed, physically consistent virtual worlds using simple keyboard or mouse inputs.

• Multi-Scenario Generalization:
  The model generalizes well to a variety of Minecraft game environments (e.g., forests, beaches, deserts, glaciers), with potential for adaptation beyond Minecraft-style games.

• Autoregressive Long-Form Video Generation:
  Supports autoregressive generation of long videos, ensuring smooth transitions between actions and camera angles for coherent temporal and environmental consistency.

• Systematic Evaluation Framework:
  Introduces the GameWorld Score, a comprehensive evaluation standard measuring video quality in terms of visual fidelity, temporal consistency, action controllability, and physical rule understanding.

Technical Principles Behind Matrix-Game

• Two-Stage Training Strategy:
  In the first stage, the model learns environment dynamics and features from large-scale unlabeled Minecraft videos. In the second stage, fine-grained controllable video generation is trained using labeled interactive data from Minecraft and Unreal Engine with keyboard and mouse input signals.

• Image-to-World Modeling:
  Uses a single reference frame as the starting point for interactive video generation without relying on text prompts. It models spatial geometry, object motion, and physical interactions purely from visual signals.

• Autoregressive Video Generation:
  Generates videos in an autoregressive manner, using the last few frames of the previous segment as motion context to ensure smooth temporal continuity. Training incorporates random perturbations, frame drops, and classifier-free guidance to reduce temporal drift and error accumulation.

• Controllable Interaction Design:
  Keyboard actions are represented as discrete tokens, while camera movements are represented as continuous tokens. Based on the GameFactory control module, the architecture integrates a multi-modal Diffusion Transformer and uses classifier-free guidance to enhance responsiveness to user inputs.

Project Links

• Official Homepage: https://matrix-game-homepage.github.io/

• GitHub Repository: https://github.com/SkyworkAI/Matrix-Game

• HuggingFace Model Hub: https://huggingface.co/Skywork/Matrix-Game

• Technical Paper: https://github.com/SkyworkAI/Matrix-Game

Application Scenarios

• Virtual Game Development:
  Quickly generate diverse game maps and dynamic interactive environments to accelerate development and enhance player immersion.

• Film and Metaverse Production:
  Create high-fidelity dynamic scenes for immersive experiences, supporting rapid creative content generation.

• Embodied AI Training:
  Provide rich and varied virtual environments to improve the task performance of embodied AI agents.

• Education and Training:
  Build interactive virtual environments for teaching and vocational training to enhance understanding and practical experience.

• Creative Content Generation:
  Offer a robust foundation for producing creative videos and virtual scene designs, enabling fast prototyping of imaginative ideas.