What is MirageLSD?
MirageLSD is the world’s first Live-Stream Diffusion (real-time stream diffusion) AI video model launched by the Decart AI team. It enables infinite-duration real-time video generation with latency as low as 40 milliseconds and supports smooth output at 24 frames per second. Using Diffusion Forcing technology and history-enhanced training, it solves the error accumulation problems found in traditional autoregressive models during long-duration generation, achieving unlimited video generation. Leveraging technologies such as Hopper-optimized Mega Kernels, architecture-aware pruning, and Shortcut Distillation, MirageLSD significantly boosts generation speed while maintaining high visual quality, enabling true real-time interaction.
Main Features of MirageLSD
-
Infinite-Duration Real-Time Video Generation: MirageLSD can generate video streams of unlimited length with latency under 40 milliseconds, supporting real-time generation at 24 FPS, effectively addressing error accumulation issues common in traditional video generation models during long sequences.
-
Real-Time Interactivity: Users can provide prompts, make transitions, and edit during video generation, enabling a continuous interactive experience.
-
Low-Latency Processing: Through optimization techniques such as Hopper-optimized Mega Kernels and architecture-aware pruning, the model achieves ultra-low latency processing of 40 milliseconds, supporting real-time video generation.
-
Wide Application Scenarios: Applicable in live streaming, video calls, game development, animation production, and more, allowing real-time transformation of ordinary scenes into user-specified styles or environments.
Technical Principles Behind MirageLSD
-
Diffusion Forcing Technology: Enables frame-by-frame denoising, allowing the model to generate individual frames without needing the full video context, thus achieving frame-level generation.
-
History-Enhanced Training: Incorporates noise data from historical frames during training so the model can predict and correct input errors, enabling infinite generation.
-
Optimization Strategies:
-
Hopper-Optimized Mega Kernels: Optimized for NVIDIA Hopper GPU architecture to reduce per-layer model latency.
-
Architecture-Aware Pruning: Adjusts model parameter sizes to fit GPU architecture and reduce computational load.
-
Shortcut Distillation: Trains smaller models to mimic the denoising trajectory of larger models, reducing the diffusion steps required for generation.
-
Project Link for MirageLSD
- Technical Paper: https://about.decart.ai/publications/mirage
Application Scenarios of MirageLSD
-
Live Streaming and Video Calls: Real-time transformation of ordinary video calls or live streams into user-specified scenes, such as turning real-world scenes into sci-fi worlds.
-
Game Development: Real-time conversion of game visuals into different artistic styles, for example, transforming regular combat scenes into lightsaber duels.
-
Animation Production and Virtual Dressing: Provides real-time visual effects support for animation creation and virtual costume changes.
