Skip to main content
placeholder image

Metamorphic Fuzz Testing of Autonomous Vehicles

Conference Paper


Abstract


  • Driving simulation is the primary approach for testing the software components of autonomous vehicles. This paper presents an automated testing method, termed metamorphic fuzz testing (MFT), in the context of simulation testing of autonomous driving. MFT differs from existing fuzzing techniques in the following two stages: First, it can generate "unrealistic"scenarios where scenes of the virtual world are refreshed frequently (so obstacles can suddenly appear / disappear) - -this is to test the self-driving vehicle's robustness in the face of unexpected situations. In the second stage, MFT uses metamorphic relations as a filtering or debugging tool to distinguish between genuine failures and false alarms yielded in the first stage. We conduct empirical studies using the real-life Baidu Apollo self-driving system, recording a genuine failure rate of 3.7%. We have reported some of the detected failures to the Apollo team and received their confirmation. Our testing method is platform-independent and, therefore, can be applied to other autonomous driving systems and advanced driver-assistance systems (ADAS).

Publication Date


  • 2020

Publisher


Citation


  • Han, J. C., & Zhou, Z. Q. (2020). Metamorphic Fuzz Testing of Autonomous Vehicles. In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020 (pp. 380-385). doi:10.1145/3387940.3392252

Scopus Eid


  • 2-s2.0-85093078440

Web Of Science Accession Number


Start Page


  • 380

End Page


  • 385

Abstract


  • Driving simulation is the primary approach for testing the software components of autonomous vehicles. This paper presents an automated testing method, termed metamorphic fuzz testing (MFT), in the context of simulation testing of autonomous driving. MFT differs from existing fuzzing techniques in the following two stages: First, it can generate "unrealistic"scenarios where scenes of the virtual world are refreshed frequently (so obstacles can suddenly appear / disappear) - -this is to test the self-driving vehicle's robustness in the face of unexpected situations. In the second stage, MFT uses metamorphic relations as a filtering or debugging tool to distinguish between genuine failures and false alarms yielded in the first stage. We conduct empirical studies using the real-life Baidu Apollo self-driving system, recording a genuine failure rate of 3.7%. We have reported some of the detected failures to the Apollo team and received their confirmation. Our testing method is platform-independent and, therefore, can be applied to other autonomous driving systems and advanced driver-assistance systems (ADAS).

Publication Date


  • 2020

Publisher


Citation


  • Han, J. C., & Zhou, Z. Q. (2020). Metamorphic Fuzz Testing of Autonomous Vehicles. In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020 (pp. 380-385). doi:10.1145/3387940.3392252

Scopus Eid


  • 2-s2.0-85093078440

Web Of Science Accession Number


Start Page


  • 380

End Page


  • 385