Motivation
Autonomous driving has been a daunting challenge for the last couple decades. Some argue that we are close to level 5 automation, but in reality there are still additional challenges to overcome. One major challenge is the ability to persist optimal perceptual performance across different weather types and lighting conditions. Currently, autonomous vehicle operations and popular benchmarks focus on improving performance in clear, daytime conditions. In order to deploy self-driving at scale, autonomy performance must not degrade in areas where weather conditions are more volatile.
Adverse weather conditions cause artifacts that can confuse perception systems trained on clear weather data. Rain can cause vertical streaks across images or adherent raindrops to sensors, fog masks details and disperses light on the roadways, and snow can completely occlude lane lines and objects on the road. Without properly accounting for these scenarios in training, perception models may exhibit unpredictable behavior or miss important information in the scene.
Problem Statement
For our project, we propose a 3-pronged approach to tackle this challenge: (1) dataset aggregation, (2) weather type and degree classification, and (3) task-driven image restoration. The first portion combats the data scarcity of on-road bad weather data. We also follow physics-based atmospheric scattering models to generate synthetic data used to train our generative restoration models. Next, weather type and degree give cues about the intensity of weather to the restoration framework.
Finally, our restoration model is a generative transformer-based model that uses both a similarity and perceptual loss to drive the model to perform equally well on all weather types. Our hope is that by combining these efforts, we can maintain optimal performance across all weather conditions.
References
[1] M. J. Mirza et al., “Robustness of Object Detectors in Degrading Weather Conditions,” 2021 IEEE
[2] Y. Lee, J. Jeon, Y. Ko, B. Jeon and M. Jeon, “Task-Driven Deep Image Enhancement Network for Autonomous Driving in Bad Weather,” 2021 IEEE