MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes

While controllable generative models for images and videos have achieved remarkable success, high-quality and controllable generation for 3D scenes, particularly in unbounded scenarios like autonomous driving, remains underdeveloped due to high data acquisition requirements. In this paper, we introduce MagicDrive3D, a novel framework for controllable 3D street scene generation that integrates video-based view synthesis with 3D representation (3DGS) generation. This approach supports multi-condition control, including road maps, 3D objects, and text descriptions. Unlike previous methods that acquire 3D representation before training generative models, MagicDrive3D begins by training a multi-view video generation model to synthesize diverse street views. This innovative approach leverages routinely collected autonomous driving data (e.g., nuScenes), significantly reducing data acquisition challenges and enhancing the richness of 3D scene generation. In the 3DGS generation step, we introduce Fault-Tolerant Gaussian Splatting to address minor errors in the generated content. We also introduce monocular depth for better initialization prior, and appearance modeling to handle exposure discrepancies across different viewpoints. Experiments demonstrate that MagicDrive3D generates diverse, high-quality 3D driving scenes that support any-view rendering and enhance downstream tasks like BEV segmentation, showcasing its potential for autonomous driving simulation and beyond.