Reconstruction of Dense 3D Point Cloud from Images
We aimed at lifting 2D images into 3D space for precise obstacle point clouds. The input consists of image sequences from the Roadwork Dataset. First, we feed the images into the Metric3D network to extract the metric depth of each image. Since the depth information can be inconsistent across frames, we then run Droid-SLAM to jointly optimize the camera poses and depth information. After these two steps, we can generate a point cloud like the one shown here. The pipeline is shown below.
Here’s an example of our results.
Filtering
To get the positions and points of obstacles in road work areas, we process the 3D point cloud and validate points through projections across multiple 2D image masks. Points visible in a specific number of consecutive frames are retained as valid. Here is the demonstration of our SLAM-generated point cloud and our filtered obstacles. We see that by utilizing 2D segmentation labels, we successfully retain the points corresponding to obstacles while discarding irrelevant points. Below is some visualization.
Geo-alignment
We also apply a geo-alignment algorithm, so that the camera poses and point cloud are restored to the correct real-world scale. Figure (a) is The camera trajectories before geo-alignment. Figure (b) is The camera trajectories after geo-alignment. The red trajectories represent the SLAM poses, while the blue trajectories depict the geo-aligned COLMAP poses.
