Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

Daniel J. Fremont, Edward Kim, Yash Vardhan Pant, Sanjit A. Seshia, Atul Acharya, Xantha Bruso, Paul Wells, Steve Lemke, Qiang Lu, and Shalin Mehta. Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World. In 23rd IEEE International Conference on Intelligent Transportation Systems (ITSC), September 2020.

Download

[pdf] 

Abstract

We present a new approach to automated scenario-based testing of the safety of autonomous vehicles, especially those using advanced artificial intelligence-based components, spanning both simulation-based evaluation as well as testing in the real world. Our approach is based on formal methods, combining formal specification of scenarios and safety properties, algorithmic test case generation using formal simulation, test case selection for track testing, executing test cases on the track, and analyzing the resulting data. Experiments with a real autonomous vehicle at an industrial testing facility support our hypotheses that (i) formal simulation can be effective at identifying test cases to run on the track, and (ii) the gap between simulated and real worlds can be systematically evaluated and bridged.

BibTeX

@inproceedings{fremont-itsc20,
 author    = {Daniel J. Fremont and
               Edward Kim and
               Yash Vardhan Pant and
               Sanjit A. Seshia and
               Atul Acharya and
               Xantha Bruso and
               Paul Wells and
               Steve Lemke and
               Qiang Lu and
               Shalin Mehta},
  title     = {Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World},
  booktitle = {23rd {IEEE} International Conference on Intelligent Transportation Systems (ITSC)}, 
  month = sep,
  year = {2020},
  abstract = {We present a new approach to automated scenario-based testing of the safety of autonomous vehicles, especially those using advanced artificial intelligence-based components, spanning both simulation-based evaluation as well as testing in the real world. Our approach is based on formal methods, combining formal specification of scenarios and safety properties, algorithmic test case generation using formal simulation, test case selection for track testing, executing test cases on the track, and analyzing the resulting data. Experiments with a real autonomous vehicle at an industrial testing facility support our hypotheses that (i) formal simulation can be effective at identifying test cases to run on the track, and (ii) the gap between simulated and real worlds can be systematically evaluated and bridged.},
}