Skip to main content
placeholder image

Metamorphic Testing: Testing the Untestable

Journal Article


Abstract


  • What if we could know that a program is buggy, even if we could not tell whether or not its observed output is correct? This is one of the key strengths of metamorphic testing, a technique where failures are not revealed by checking an individual concrete output, but by checking the relations among the inputs and outputs of multiple executions of the program under test. Two decades after its introduction, metamorphic testing has become a fully-fledged testing technique with successful applications in multiple domains, including online search engines, autonomous machinery, compilers, Web APIs, and deep learning programs, among others. This article serves as a hands-on entry point for newcomers to metamorphic testing, describing examples, possible applications, and current limitations, providing readers with the basics for the application of the technique in their own projects.

Authors


  •   Segura, Sergio (external author)
  •   Towey, Dave (external author)
  •   Zhi Quan (George) Zhou
  •   Chen, Tsong Yueh (external author)

Publication Date


  • 2019

Citation


  • Segura, S., Towey, D., Zhou, Z. Quan. & Chen, T. Yueh. (2019). Metamorphic Testing: Testing the Untestable. IEEE Software, Online First 1-12.

Scopus Eid


  • 2-s2.0-85058627958

Number Of Pages


  • 11

Start Page


  • 1

End Page


  • 12

Volume


  • Online First

Place Of Publication


  • United States

Abstract


  • What if we could know that a program is buggy, even if we could not tell whether or not its observed output is correct? This is one of the key strengths of metamorphic testing, a technique where failures are not revealed by checking an individual concrete output, but by checking the relations among the inputs and outputs of multiple executions of the program under test. Two decades after its introduction, metamorphic testing has become a fully-fledged testing technique with successful applications in multiple domains, including online search engines, autonomous machinery, compilers, Web APIs, and deep learning programs, among others. This article serves as a hands-on entry point for newcomers to metamorphic testing, describing examples, possible applications, and current limitations, providing readers with the basics for the application of the technique in their own projects.

Authors


  •   Segura, Sergio (external author)
  •   Towey, Dave (external author)
  •   Zhi Quan (George) Zhou
  •   Chen, Tsong Yueh (external author)

Publication Date


  • 2019

Citation


  • Segura, S., Towey, D., Zhou, Z. Quan. & Chen, T. Yueh. (2019). Metamorphic Testing: Testing the Untestable. IEEE Software, Online First 1-12.

Scopus Eid


  • 2-s2.0-85058627958

Number Of Pages


  • 11

Start Page


  • 1

End Page


  • 12

Volume


  • Online First

Place Of Publication


  • United States