To improve the performance of the detector, we could use the synthesis strategy. Here, we take clear images and transform them to simulate various bad weather conditions. By training our detector on these bad weather images, we prepare our detector to recognize and interpret objects under bad weather conditions.
The second strategy is de-weathering. Instead of training the system to deal with bad weather, we revert our bad weather images back into clear ones, allowing the detector to detect effectively.
Physics-based Foggy Image Generation
We utilized a physics-based method to generate foggy images, which can synthesize more training data. This model states that the color of a scene point E in fog or haze, observed by a color camera, is given by a linear combination of the direction of the scene point R’s color as seen on a clear day, and the direction of airlight color E-infinity in fog or haze. So we generate synthetic foggy images using the depth map generated by the Depth Anything method, and also adjust the amount of fog by the scattering coefficient beta.
Here are the input images and their corresponding foggy images. We can add realistic fog to clear images, which aids in further training and refining our models.
Another progress is that we finetuned the Faster R-CNN detector using both real driving datasets and generated foggy images, and testing on the DENSE dataset’s heavy foggy images, either with dehazing or without dehazing.
Diffusion-based Foggy Image Generation
Another synthesizing approach we explored is a diffusion-based method. We began by collecting a sufficient amount of foggy data using the CARLA simulator. Next, we fine-tuned a stable diffusion model to perform unpaired image translation. In this setup, the source domain consists of images captured under clear weather conditions, while the target domain comprises the simulated foggy images. To implement this, we used a cycleGAN-like architecture.
The results, as shown below, demonstrate that this diffusion-based method is capable of generating photorealistic foggy images, provided there is enough unpaired training data. The generated images accurately simulate fog, with thicker fog in the distance and clearer visibility up close, similar to the results achieved with the physics-based model.
Using NeRF to Defog
Our method’s overall pipeline is illustrated in the figure. Assuming the NeRF output consists of color and depth under clear weather conditions, we introduced a learnable parameter, beta (scattering coefficient), and used the atmospheric scattering model to convert these into foggy colors. Finally, we computed the loss of color and depth for backpropagation.
