DeepMesh – A 3D Mesh Generation Framework Developed by Tsinghua University and Nanyang Technological University

What is DeepMesh?

DeepMesh is a 3D mesh generation framework proposed by researchers from Tsinghua University and Nanyang Technological University. It generates high-quality 3D meshes based on reinforcement learning and autoregressive transformers. The framework optimizes mesh generation through two key innovations: first, an efficient pre-training strategy that incorporates a novel tokenization algorithm and an improved data processing pipeline; second, the introduction of reinforcement learning (specifically Direct Preference Optimization, DPO) to align the generated meshes with human preferences. DeepMesh can generate meshes with complex details and precise topological structures conditioned on point clouds and images, outperforming existing methods in terms of accuracy and quality.

The main functions of DeepMesh

• High-quality 3D Mesh Generation: DeepMesh can generate 3D meshes with rich details and accurate topological structures, suitable for various complex geometric shapes.
• Point Cloud Conditional Generation: DeepMesh can generate corresponding 3D meshes based on input point cloud data, suitable for various scenarios ranging from sparse to dense point clouds.
• Image Conditional Generation: DeepMesh supports conditional generation based on images, capable of generating 3D meshes from input 2D images.

The technical principles of DeepMesh

• Self-Regression Transformer: DeepMesh adopts a self-regression transformer as its core architecture, which includes self-attention layers and cross-attention layers. It can gradually generate the faces of a mesh and predict the vertices and faces of the mesh through conditional inputs (such as point clouds or images). For point cloud conditional generation tasks, DeepMesh incorporates a perceptron encoder to extract point cloud features and integrates them into the transformer model.
• Efficient Pre-training Strategy: DeepMesh introduces an improved tokenization algorithm that significantly reduces sequence length through locally aware face traversal and block index coordinate encoding while preserving geometric details. The framework also adopts data organization and processing improvement strategies, filtering low-quality mesh data and enhancing training efficiency through a truncation training strategy.
• Reinforcement Learning and Human Preference Alignment: DeepMesh incorporates Direct Preference Optimization (DPO), collecting preference pairs for reinforcement learning training based on human evaluations and the design of 3D metric scoring criteria. This ensures that the generated meshes are geometrically accurate and visually aligned with human aesthetics.
• End-to-End Differentiable Mesh Representation: DeepMesh supports end-to-end differentiable mesh representation with dynamic topology changes. This differentiability enables the model to be optimized through gradient descent, further improving the quality of the generated meshes.

The project address of DeepMesh

• Project official website: https://zhaorw02.github.io/DeepMesh/
• GitHub repository: https://github.com/zhaorw02/DeepMesh
• arXiv technical paper: https://arxiv.org/pdf/2503.15265

Application scenarios of DeepMesh

• Virtual Environment Construction: DeepMesh can generate realistic 3D mesh models for constructing virtual scenes in virtual reality, such as virtual buildings, virtual cities, etc.
• Dynamic Content Generation: Through reinforcement learning optimization, DeepMesh can dynamically generate 3D models based on real-time data in games, enhancing the immersion and interactivity of the gaming experience.
• Character Animation: DeepMesh can generate high-quality 3D character models, supporting complex animation production needs such as skeletal rigging and animation rendering.
• Dynamic Medical Simulation: Through reinforcement learning optimization, DeepMesh can generate dynamic medical models, such as heart motion simulation, to help doctors better understand organ movement and function.
• Product Modeling: DeepMesh can be used to generate 3D models of industrial products, supporting complex design and manufacturing processes.