Experiments

Datasets

We collected 2 sets of data: one for training the weather classification model and one for training the restoration model. In the ideal case, we would like our data to match the conditions seen on the road as closely as possible. This is not always achievable due to the use of metrics such as PSNR to optimize the restoration models, where both ground truth and perturbed images are needed for comparison. Thus, we must generate some data synthetically in conjunction with real-world datasets.

Aggregated real-world data

The data used for weather classification must be aggregated according to the weather-type. Currently to our knowledge, there are no large-scale datasets taken from vehicles that contain weather type annotations. We are working to combine multiple datasets together for both downstream tasks. In order to improve on generalizability of the models, we include data from both on and off road bad weather conditions. At the moment, the DAWN and Multi Weather Image (MWI) Datasets are combined together for weather classification tasks and we hope to continue adding more datasets.

Synthetic data

To optimize the restoration model we require clear images without degradation along with weather degraded image of the same scene. Hence, we will have to take clear images from any on road dataset and generate weather degradations in it. Also, since we aim on performing evaluation of the restored images on downstream tasks, we require the ground truth annotations of these downstream tasks. We plan of performing our evaluation on object detection and hence, we took the 2D object-KITTI dataset. Its bounding box annotations can be used for a task driven loss and downstream evaluation. We obtained corresponding synthetic fog and rain images. The original dataset had 15000 images which we divided into 13000 for the train dataset(5000 fog, 5000 rain and 3000 clear) and 2000 for the test dataset(750 fog, 750 rain and 500 clear). A third category of clear images is added for the restoration model to learn not to perform any changes to the image when it does not have any degradation.

For synthetic fog generation we used the Atmospheric Scattering model[1]. We generated foggy images with 5 different fog densities of 0.6, 0.9, 1.2, 1.5 and 1.8.

For synthetic rain generation we used a combination of GAN and PBR method[2]. We generated images with 5 different rain densities.

Weather Classification Model

Our weather classification is designed for two purposes: first to segment out different weather types to be sent to distinct downstream restoration models, and second (once the model is fully trained) to create weather type annotations for newly sourced data. 

Our weather classification module consists of a resnet50 backbone model and classifies the data into the following classes: (1) rain, (2) snow, (3) fog/haze, (4) sand, and (5) clear. We first trained the weather classification model without adding any feature extraction methods. Next, we also trained the model by using features such as saturation, contrast and dark channel[3]. These features are represented as histograms and concatenated at the linear layers.

Image Restoration Model

For image restoration we use a single transformer encoder-decoder model based on TransWeather[4] to restore an image degraded by any type of weather condition. The TransWeather model has three parts to it:

1. Transformer encoder: A degraded input image is given to a transformer encoder which learns coarse and fine information across each stage.
2. The transformer decoder block uses the encoded features as keys and values and uses learnable weather type query embeddings as queries.
3. These extracted features are then passed through a convolutional projection block to get a clean image.

The loss used to train this model is a sum of a restoration L1 loss and perceptual loss. The restoration L1 loss makes sure that the model trains to produce restored images as close as clear images and to remove the weather degradations. The perceptual loss is the mean square error between the features extracted from the ground truth clear image and the output restored image.

Experiments on TransWeather

1. We first replicated the TransWeather model with the available best weights and tested it on a single weather type of rain. This provided good results on synthetic rain data but did not give clear restored images when tested on real world Dawn dataset. This might be due to domain shift. Fine-tuning the model on real-world dataset is the good next option, but this is not possible due to unavailability of clear and degraded image from same scene.
2. The next step we took was to collect synthetic weather degraded data that closely match real world data. We first check the performance of the model on a single weather-type, and later plan to extend it to all weather types. We trained the TransWeather model on our synthetically generated fog images(5000 train and 750 test) from scratch.
3. Now, we wanted to extend this to multiple weather types. For this we trained the model from scratch on a dataset of synthetically generated fog images(5000 train and 750 test), synthetically generated rain images(5000 train and 750 test) and clear images(3000 train and 500 test). The evaluation metrics after testing the trained model on the combined test dataset were much lesser the evaluation metrics obtained with just fog dataset. This behaviour can either be attributed to poor performance on restoration of images degraded by rain or inability of the decoder to find the weather type. This is where the integration of a weather classifier might help.

References

[1] S.G. Narasimhan and S.K. Nayar. Contrast restoration of weather degraded images. IEEE Transactions on Pattern Analysis and Machine Intelligence,25(6):713–724, 2003.

[2] Maxime Tremblay, Shirsendu Sukanta Halder, Raoul de Charette, and Jean-Fran ̧cois Lalonde. Rain rendering for evaluating and improving robustness to bad weather. CoRR, abs/2009.03683, 2020.

[3] Zhang, Zheng, et al. “Scene-free multi-class weather classification on single images.” Neurocomputing

[4] Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M. Patel. Transweather: Transformer-based restoration of images degraded by adverse weather conditions. CoRR, abs/2111.14813, 2021.