Background
As automated vehicles (AVs) are introduced onto public roads, regulators will need reliable methods of testing their performance. This task poses many challenges. For example, an AV’s software is often proprietary, making the car’s internal data flows inaccessible to independent evaluators. Additionally, different classes of vehicles on different roadways will be expected to perform differently, meaning testing approaches should be flexible. The objective of this project was to determine whether data collected from externally mounted autonomy sensor systems could be used to generate useful metrics related to AV behavior and safety without access to the underlying software or sensor platform.
Approach
We developed metrics based around lane keeping, speed control, and obstacle avoidance using information available from the Ranger Evaluation Kit (REK) and Engineering Technology Demonstrator 2 (ETD2). These sensor suites included sensors such as a camera, global positioning system (GPS) antennae, inertial measurement units (IMUs), and a lidar. We expressed the metrics in code that would take the outputs from the sensor suites and export a metrics report. We evaluated the test approach by executing driving scenarios well and poorly to compare the resulting performance metrics. For example, in the diagram below, which describes good and poor execution of the passing scenario, the black, red, and blue icons represent the shuttle, MKZ, and Acura from the picture. Extreme care was taken to prevent any accidents in the poor driving scenarios. We also collected REK data on multiple platforms to gauge the applicability of this testing approach to vehicles of different sizes.
Figure 1: Good vs poor driving for passing scenario.
Figure 2: Vehicle platforms used in the passing scenario; the middle vehicle is fitted with the sensor suites.
Accomplishments
We demonstrated that data from the sensors listed above could be combined in a modular way to create metrics in key AV performance areas, such as speed control, without having to access any of the test vehicle’s hardware or software internals. We wrote code to calculate these metrics and export the results into a human-readable metrics report. Finally, we demonstrated that the REK sensor suite could be mounted on a variety of platforms, making the testing approach widely applicable.