Introduction
Autonomous driving systems rely heavily on visual perception, which can degrade significantly under adverse weather conditions such as rain, fog, and snow. In such environments, camera-based object detection often suffers from reduced visibility, motion blur, and occlusion. Radar sensors, on the other hand, are more robust to weather but provide sparse and less detailed spatial information.
In this project, we investigate multi-modal fusion of camera images and radar signals to improve object detection performance in adverse driving scenarios. We propose a framework that leverages the complementary strengths of both modalities, combining the rich semantic information from images with the robustness of radar measurements. Our approach focuses on effective feature alignment and fusion strategies to enhance detection accuracy under challenging conditions.
Through extensive experiments, we demonstrate that multi-modal fusion significantly improves detection robustness and reliability compared to single-modality baselines, particularly in low-visibility environments.
Methods
We need to generate samples with fewer categories, so we designed the following workflow. The pipeline for synthetic driving data generation is illustrated in the figure. Starting from real driving data, we first extract structured scene representations under multiple condition channels, including geometry, layout, and object-level abstractions. These intermediate representations serve as controllable condition inputs that disentangle scene structure from environmental factors. A vision-language model (VLM) is then used to encode high-level semantic prompts describing diverse driving conditions, such as weather (e.g., rain, snow, fog), time of day (morning, noon, night), and complex scenarios (e.g., fire scenes, construction sites, dust storms). These condition embeddings are integrated through a Cosmos Transfer module, which maps the structured representations and semantic conditions into a unified latent space and generates corresponding synthetic driving data. This design enables flexible and scalable synthesis of diverse, high-fidelity driving scenes while preserving the underlying spatial consistency of real-world data.
As we previously mentioned, various sensors perform differently across different scenarios; therefore—to leverage the strengths of each—we have designed a fusion architecture. The architecture of the proposed multi-modal perception framework is illustrated in the figure below. The system takes synchronized inputs from three sensors, including camera images, LiDAR point clouds, and RADAR signals. For the camera branch, image features are first extracted and then transformed into a bird’s-eye-view (BEV) representation through a camera-to-BEV projection module. For the LiDAR and RADAR branches, raw point-based inputs are encoded and flattened into BEV-aligned feature maps. These modality-specific BEV features are subsequently fused via element-wise aggregation to form a unified BEV representation that captures complementary spatial and semantic information across sensors. The fused BEV features are then processed by a 2D backbone network to further enhance spatial context and feature expressiveness. Finally, a 3D detection head operates on the refined BEV features to predict object locations and categories in 3D space. This design effectively leverages the strengths of different sensing modalities and enables robust perception under diverse and challenging driving conditions.
Experiments
The results are summarized in the figure from both qualitative and quantitative perspectives. We first visualize the diversity of the generated data, where our method synthesizes a wide range of adverse weather conditions (e.g., overcast, rain, and snow) as well as long-tail scenarios (e.g., construction zones, airport runways, fire scenes, and dust storms) based on real driving scenes. Compared to the original real-world data, the synthetic dataset exhibits a significantly more balanced distribution across different conditions, effectively enriching underrepresented cases such as night, rain, and rare events.
Quantitatively, we evaluate the impact of the proposed synthetic data on 3D object detection performance. By augmenting the training set with the generated data, our method achieves substantial improvements over the baseline. Specifically, the mean Average Precision (mAP) increases from 0.747 to 0.949, while key error metrics are consistently reduced, including translation error (0.851 → 0.778), scale error (0.794 → 0.733), and orientation error (1.035 → 0.672). Velocity error remains comparable. These results demonstrate that incorporating diverse and realistic synthetic data effectively improves model robustness and generalization, particularly in challenging and long-tail driving scenarios.
Below is the 3D object detection results on real-world autonomous driving datasets They are presented across four representative urban scenarios, including residential streets, intersections, and crowded pedestrian zones. The proposed method demonstrates robust performance in detecting and localizing multiple categories of objects, such as vehicles (orange bounding boxes) and pedestrians (blue/red bounding boxes), under diverse conditions—including partial occlusion, varying distances, and complex lighting. Across all cases, the predicted 3D bounding boxes show high alignment with the actual positions, sizes, and orientations of the targets, effectively reducing false negatives and positives even for small, distant, or heavily occluded objects. These results validate the method’s reliability and generalization capability in real-world driving environments, making it suitable for downstream perception tasks in autonomous systems.
Resources
2026 Spring Semester
Team
