Runway segmentation
Method
We first fine-tuned YOLO family segmentation models on two public datasets: RLD[1] and BARS[2]. This allowed us to find the optimal hyperparameters. We also tested different sizes of models to find the best accuracy-latency tradeoff.
This was our baseline, from which we built upon the three additional modules of VALNet[1]. An ablation study assesses the efficacy of each module.
Results
Yolo baseline:
| Precision | Recall | mAP50 | AP75 |
| 0.93 | 0.91 | 0.92 | 0.93 |
Interpretations:
- Precision, more important than recall in that safety-critical case, is satisfying.
- More specifically, AP75’s results is good news, since our later experiments showed that our feature extraction tend to works much better for IoU>75%
- Most false positive are when the runway is extremely far or when another runway is detected in the airport
Improvement with VALNet’s modules :
The implementation of the additional modules did not bring significant improvements over YOLO’s baseline. This may be due to the fact that our dataset did not have enough complex scenarios to show the added value of the modules.
VALNet Evaluation
We implement based on the provided description from the publication and evaluate the relative performance of VALNet in comparison to the YOLOv8-seg benchmark. Preliminary results show that VALNet outperforms smaller versions of YOLOv8-seg in terms of AP but are not aligned with the results presented in the paper.
A drawback of precision metrics is that they do not relay information about the geometric properties of the segmentation. Qualitative examinations show that VALNet can accurately estimate runway geometry at close distances (<1km). A noticed failure case is that VALNet will label tall pillar-like structures like ATC towers as false positives.
Figure 1: example of good segmentation from VALNet. Note that the segmentation results captures the geometry of the runway.
Figure 2: example of bad segmentation from VALNet with additional inaccurate segmentation
Feature extraction
Method
We apply geometric post-processing on the segmentation mask to extract region of interests like:
Central line:
- Useful for lateral offset and yaw estimation
- Centerline extraction algorithm: the dominant runway direction is first estimated via PCA on the mask pixels. Then, a set of lines parallel to this direction is sampled, and for each line, pixels within the 5th to 95th percentile are selected. Midpoints are computed along these scanlines, and a final centerline is robustly fitted using RANSAC to avoid outliers.
- Direct centerline estimation using deep learning models has also been explored as an alternative to this geometric pipeline.
Corner points:
- Useful for potential homography estimation
- Douglas–Peucker algorithm
Results
Observed results on public dataset (BARS + RLD):
- Median lateral offset: 2.04 px
- Median angle: 1.56 rad
Interpretations:
- The method works well, except for outliers
- Most outliers are from a segmentation mask with IoU<70%. This is the reason why AP75 was taken as the main metrics for runway segmentation
XPlane-12 Dataset
Public datasets for runway segmentation such as RLD and BARS do not contain pose labels. We develop a plugin for XPlane-12 that performs automated landings with randomized conditions and collects pose information (3D position, roll, pitch, and yaw) of the aircraft relative to the runway. For preliminary experiments, we use 250 example descents captured at different runways around the world during the daytime with random environment settings.
Figure 3: example of an RGB image from an automated descent captured using our XPlane-12 plugin
Learning-based Baseline for Lateral Displacement Estimation
Figure 4: rough sketch of the baseline’s architecture for lateral displacement estimation
In practice, it is possible to estimate roll and pitch using inertial sensors on the aircraft. An important focus of the project is to estimate lateral displacement of the plane relative to the aircraft. We develop a baseline for lateral displacement estimation utilizing the ground truth roll and pitch, as well as the RGB camera image and the ground truth runway segmentation, by using a fully-connected network. Preliminary results suggest that this approach is able to overfit to the training data but isn’t able to generalize to validation; using our dataset generation tool to generate more data would help to alleviate this problem.
Figure 5: estimated lateral displacement on a video containing training samples
Figure 6: estimated lateral displacement on a video containing validation samples