Background
Simultaneous Localization And Mapping (SLAM) is a well-studied problem in robotics, with many existing approaches and solutions. Although indoor SLAM is a mostly solved problem, SLAM in outdoor, unstructured environments is significantly more challenging, with no well-established solutions. This project evaluated the performance of four state-of-the-art open-source SLAM algorithms across three different outdoor environments to learn how well (or poorly) each performed and how much effort was needed to adapt these algorithms for the environment.
Approach
We evaluated three different SLAM algorithms: Lightweight and Ground-Optimized LIDAR Odometry and Mapping (LeGO-LOAM), Oriented features from accelerated segment test and Rotated Binary robust independent elementary features for SLAM (ORB-SLAM2), and Google Cartographer. Each algorithm was tested across three different environments: SwRI’s test track (wide-open with minimal features), SwRI’s campus (urban area with dense buildings), and SwRI’s off-road trails.
Accomplishments
The algorithm performance was compared using a medium length route (approximately 1.6 km) for each environment. The comparison showed that none of the algorithms worked particularly well, with most algorithms accumulating at least 10 m of error after a single loop. LeGO-LOAM performed the worst, accumulating between 150 m and 1,010 m error, while Cartographer performed the best with 9.79 m to 118.51 m of accumulated error. Performance varied across different environments, but not as expected; in several cases wide-open areas proved worse than off-road areas. Cartographer performed the worst in wide-open spaces, while LeGO-LOAM and ORB-SLAM2 performed most poorly on the off-road trails.
Our primary conclusion is that none of the approaches we evaluated worked “out-of-the-box” and took considerable effort to tune and make functional. After this tuning, we were able to get two out of three algorithms to produce decent results (Figure 1 shows performance at SwRI’s test track) for some of the three environments, but no algorithm or set of parameters worked reliably in all environments.
Figure 1: Final reconstructed routes using different SLAM algorithms: (A) LeGO-LOAM, (B) ORB-SLAM2, and (C) Cartographer. Note that LeGO-LOAM performed the worst, failing to maintain shape, ORB-SLAM2 maintained overall shape decently (until final leg), and Cartographer had the smoothest final path, but underestimated turns around corners, which warped the final shape.