MagicColor – A Multi-Instance Line Drawing Coloring Framework Launched by the Hong Kong University of Science and Technology

What is MagicColor?

MagicColor is a multi-instance line art coloring framework introduced by the Hong Kong University of Science and Technology, supporting efficient colorization of line art. Leveraging a self-supervised training strategy and an instance guidance module, MagicColor addresses the challenge of insufficient multi-instance data, achieving precise instance-level color control. Compared to traditional methods, MagicColor can colorize multi-instance line art in a single forward pass, significantly improving coloring efficiency. It is well-suited for applications in animation production and digital art creation.

The main functions of MagicColor

• Multi-instance Line Art Coloring: MagicColor can process multiple instances of line art coloring tasks simultaneously, assigning accurate colors to each instance.
• Automated Coloring: Automatically converts line art into colorful images.
• Reference Image Driven: Users provide a reference image, and MagicColor colors the line art based on the color and style of the reference image, ensuring that the output image is visually consistent with the reference image.
• Instance-level Control: Precisely controls the color of each instance, maintaining details and color accuracy in complex multi-instance scenarios.
• Edge Enhancement: The model focuses on the edges and high-frequency areas of the image during coloring to improve coloring accuracy and visual quality.

The technical principle of MagicColor

• Foundations of Diffusion Models: MagicColor is based on a pre-trained diffusion model. Leveraging the powerful generative capabilities and semantic consistency maintenance of diffusion models, it converts sketches into color images.
• Self-Supervised Training Strategy: To address the issue of insufficient multi-instance training data, MagicColor employs a two-stage self-supervised training strategy. In the first stage, training is conducted using a single reference image. In the second stage, multi-instance data are generated through operations such as random fusion and scaling to further optimize the model’s multi-instance coloring ability.
• Instance Guidance Module: This module extracts instance features from the reference image and aligns them with the sketch, enabling precise instance-level color control. Features extracted using DINOv2 and ROI alignment techniques are utilized to ensure accurate color transfer of each instance to the target sketch.
• Edge Loss and Color Matching: To enhance the visual quality of coloring, MagicColor introduces an edge loss function, allowing the model to focus on the edges and structural details of the image. Based on color matching techniques, the model establishes a pixel-level correspondence between the reference image and the target sketch, ensuring accurate color transfer.
• Dual UNet Architecture: MagicColor combines two UNet architectures, one for processing the reference image and the other for generating the final color image. This enables the model to better integrate the color information from the reference image and the structural information from the sketch, resulting in high-quality coloring results.

Project address of MagicColor

• Project Website: https://yinhan-zhang.github.io/color/
• GitHub Repository: https://github.com/YinHan-Zhang/MagicColor
• arXiv Technical Paper: https://arxiv.org/pdf/2503.16948

Application scenarios of MagicColor

• Animation Production: Quickly colorize multi-character sketches to improve production efficiency and maintain a consistent color style.
• Digital Art Creation: Rapidly apply colors to sketches, inspire creativity, and generate multiple color variations.
• Game Development: Quickly generate color versions of characters and scenes, enhance development efficiency, and produce stylized assets.
• Education and Training: Serve as a teaching tool to help students quickly understand color theory and improve their color application skills.
• Advertising and Media: Rapidly generate high-quality colored sketches to meet advertising design needs, enhancing visual appeal and brand consistency.