Master’s Thesis Presentation • Software Engineering — WiseBench: A Motion Planning Benchmarking Framework for Autonomous Vehicles

Friday, September 25, 2020 10:00 am - 10:00 am EDT (GMT -04:00)

Please note: This master’s thesis presentation will be given online.

Marko Ilievski, Master’s candidate
David R. Cheriton School of Computer Science

Supervisor: Professor Krzysztof Czarnecki

Rapid advances in every sphere of autonomous driving technology have intensified the need to be able to benchmark and compare different approaches. While many benchmarking tools tailored to different sub-systems of an autonomous vehicle, such as perception, already exist, certain aspects of autonomous driving still lack the necessary depth and diversity in suitable benchmarking approaches. Autonomous vehicle motion planning is one such example. While motion planning benchmarking tools are abundant in the robotics community in general, they largely tend to lack the specificity and scope required to rigorously compare algorithms that are tailored to the autonomous vehicle domain. Furthermore, approaches that are targeted at autonomous vehicle motion planning are generally either not involved enough to distinguish subtle differences between different approaches, or they are unable to scale across problems and operational design domains of varying complexity. In this work, we aim to address these issues by proposing WiseBench, an autonomous vehicle motion planning benchmark framework that approaches the problems described in a comprehensive manner.

WiseBench incorporates a robust set of simulation, scenario-suite, and comparison metrics requirements, and is implemented using a carefully crafted set of scenarios and robust comparison metrics that operate within an in-house simulation environment, all of which satisfy the specified requirements. The benchmark proved to be successful in comparing and contrasting two different autonomous vehicle motion planners, and were shown to be an effective measure of passenger comfort and safety in a real-life experiment. The main contributions of our work on WiseBench thus include: a scenario creation methodology for the representative scenario suite, a comparison methodology to evaluate different motion planning algorithms, and a proof-of-concept implementation of the WiseBench framework as a whole.

To join this master’s thesis presentation on Zoom, please go to