LegoGPT – A LEGO brick design model launched by Carnegie Mellon University

What is LegoGPT

LegoGPT is a LEGO brick design model developed by Carnegie Mellon University. It supports the generation of physically stable and buildable LEGO structures based on text prompts. Trained on an autoregressive language model and a large-scale LEGO design dataset, LegoGPT generates LEGO structures step by step according to user input, ensuring both structural stability and constructability.

LegoGPT also supports the addition of textures and colors to the models. The generated designs can be manually assembled or automatically built using robots. LegoGPT provides an efficient and creative way to design LEGO models, advancing AI applications in the physical world.

Main Features of LegoGPT

• Text-to-LEGO Design: Generates physically stable and buildable LEGO models based on user descriptions (e.g., “a chair with four straight legs”).

• Physical Stability Check: Continuously verifies the structural stability of generated LEGO designs. If instability is detected, it automatically rolls back and regenerates the structure.

• Texture and Coloring: Adds textures and colors to the generated LEGO models.

• Automatic Assembly Support: The designs are suitable for both manual and robotic assembly.

Technical Principles of LegoGPT

• Autoregressive Language Model: Uses an autoregressive language model as its core framework, framing the LEGO design task as a sequence generation problem—predicting the placement and size of the next brick step by step to build a complete structure.

• Physics-Aware Rollback Mechanism: During generation, the model continuously checks for physical stability. If an unstable segment is generated, it reverts to the last stable state and regenerates the subsequent structure to ensure final stability.

• Text-to-LEGO Mapping: Leverages natural language processing to extract geometric and structural information from text descriptions and translate them into LEGO placement instructions.

• Training on Large-Scale Dataset: Trained on a large dataset of LEGO structures and corresponding text descriptions, enabling the model to learn how to generate stable and diverse LEGO designs from text inputs.

Project Links for LegoGPT

• Project Website: https://avalovelace1.github.io/LegoGPT/

• GitHub Repository: https://github.com/AvaLovelace1/LegoGPT/

• arXiv Technical Paper: https://arxiv.org/pdf/2505.05469

• Online Demo on Hugging Face: https://huggingface.co/spaces/cmu-gil/LegoGPT-Demo

Application Scenarios of LegoGPT

• Creative Design: Helps designers quickly generate and test LEGO prototypes.

• Educational Tool: Stimulates student creativity and supports learning in physics and spatial reasoning.

• Toy Development: Enables LEGO enthusiasts to create personalized models.

• Automated Assembly: Supports robotic LEGO assembly for enhanced efficiency.

• Virtual Reality: Generates realistic LEGO models for use in VR and gaming.