Vibe Draw – An AI 3D Modeling Tool: Convert Doodle Sketches into 3D Models with One Click

What is Vibe Draw?

Vibe Draw is an open-source AI-powered 3D modeling tool that allows users to transform doodles and sketches drawn on a 2D canvas into stunning 3D models. Users can refine their models through text prompts or by continuing to draw, and export them in standard formats (e.g., .glTF) with just one click. Vibe Draw breaks down technical barriers, enabling anyone to easily bring their 3D ideas to life without requiring professional skills.

The front end of Vibe Draw is built using technologies such as Next.js, React, and Three.js, while the back end is constructed with FastAPI and Celery, supporting real-time updates and asynchronous task processing.

The main functions of Vibe Draw

• Sketch to 3D Model: Users can freely draw sketches on an infinite 2D canvas, and the AI will automatically recognize and generate 3D models.
• Model to 3D Scene: Users can drag the generated 3D models into a 3D space and arrange them freely to quickly build a complete 3D scene.
• One-Click Export: Supports exporting in glTF format.
• AI Sketch Optimization: The AI automatically optimizes sketches while preserving the user’s spatial composition.
• Text-to-3D Model: Users can directly input text prompts to generate 3D models.
• Model Editability: Generated models can be edited through “re-sketching” or by providing new text prompts.

The Technical Principle of Vibe Draw

• Front-end Technology: The front-end is built with Next.js and React, providing a responsive and user-friendly interface. It integrates Three.js to achieve efficient rendering of 3D models. TLDraw serves as a powerful 2D drawing canvas, supporting free drawing and graphic editing. Zustand is utilized for state management to ensure efficient synchronization and management of the application’s state. Overall, it delivers a smooth interactive experience to users.
• Backend Technology: The backend is built on FastAPI, a high-performance API framework, to handle frontend requests and complex backend logic. Celery is used as an asynchronous task queue to manage time-consuming AI operations, such as sketch optimization and 3D model conversion. Redis provides publish/subscribe functionality for real-time updates and task result storage. Leveraging SSE (Server-Sent Events) technology, it enables real-time feedback on task progress to users, enhancing interactivity and user experience.
• AI-Driven Conversion: The core functionality relies on advanced AI algorithms to automatically optimize users’ rough 2D sketches into more refined drawings, with support for further conversion into 3D models. The AI model enables automatic modeling of complex shapes and structures. Users can further optimize and adjust the 3D models based on text prompts, achieving efficient transformation from creative sketches to the 3D world.

The project address of Vibe Draw

• GitHub Repository: https://github.com/martin226/vibe-draw

Application scenarios of Vibe Draw

• Creative Design: Quickly convert sketches into 3D models, aiding designers in swiftly validating their creative ideas.
• Education Assistance: Intuitively demonstrate the conversion from 2D to 3D, helping students learn spatial structures and modeling principles.
• Game Development: Rapidly generate game scenes and character prototypes, accelerating pre-development design.
• Architectural Design: Transform hand-drawn architectural sketches into 3D models for easier communication of plans with clients.
• Personal Creation: Enable low-threshold creative expression, suitable for general users’ entertainment and sharing.