Skip to main content
placeholder image

Metamorphic testing of named entity recognition systems: A case study

Journal Article


Abstract


  • Named entity recognition (NER) is a widely used natural language processing technique; it plays a key role in information extraction from sentences. To be able to test the correctness of NER systems is important, but it is expensive because an automated test oracle is normally unavailable. To address the oracle problem, this study proposes to apply metamorphic testing (MT). The authors conduct a case study with Litigant, an industrial NER system of the Ant Group, and show that MT can effectively detect real-life bugs in the absence of an ideal oracle. The authors further investigate the causes for a series of entity recognition failures detected. Outcomes of this research further justify the application of MT to the natural language processing domain as well as provide hints for practitioners to improve the quality process of their NER systems.

Publication Date


  • 2022

Citation


  • Xu, Y., Zhou, Z. Q., Zhang, X., Wang, J., & Jiang, M. (2022). Metamorphic testing of named entity recognition systems: A case study. IET Software, 16(4), 386-404. doi:10.1049/sfw2.12058

Scopus Eid


  • 2-s2.0-85127557628

Start Page


  • 386

End Page


  • 404

Volume


  • 16

Issue


  • 4

Abstract


  • Named entity recognition (NER) is a widely used natural language processing technique; it plays a key role in information extraction from sentences. To be able to test the correctness of NER systems is important, but it is expensive because an automated test oracle is normally unavailable. To address the oracle problem, this study proposes to apply metamorphic testing (MT). The authors conduct a case study with Litigant, an industrial NER system of the Ant Group, and show that MT can effectively detect real-life bugs in the absence of an ideal oracle. The authors further investigate the causes for a series of entity recognition failures detected. Outcomes of this research further justify the application of MT to the natural language processing domain as well as provide hints for practitioners to improve the quality process of their NER systems.

Publication Date


  • 2022

Citation


  • Xu, Y., Zhou, Z. Q., Zhang, X., Wang, J., & Jiang, M. (2022). Metamorphic testing of named entity recognition systems: A case study. IET Software, 16(4), 386-404. doi:10.1049/sfw2.12058

Scopus Eid


  • 2-s2.0-85127557628

Start Page


  • 386

End Page


  • 404

Volume


  • 16

Issue


  • 4