Skip to main content
placeholder image

A Monte Carlo Method for Metamorphic Testing of Machine Translation Services

Conference Paper


Abstract


  • © 2018 Association for Computing Machinery. With the growing popularity of machine translation services, it has become increasingly important to be able to assess their quality. However, the test oracle problem makes it difficult to conduct automated testing. In this paper, we propose a Monte Carlo method, in combination with metamorphic testing, to overcome the oracle problem. Using this method, we assessed the quality of three popular machine translation services' namely, Google Translate, Microsoft Translator, and Youdao Translate. We set the source language to be English, and the target languages included Chinese, French, Japanese, Korean, Portuguese, Russian, Spanish, and Swedish. A sample of 33,600 observations (involving a total of 100,800 actual translations) was collected and analyzed using a 3 56 factorial design. Based on this data, our model found Google Translate to be the best (in terms of the metamorphic relation used) for each and every target language considered. A trend for Indo- European languages producing better results was also identified.

Authors


  •   Pesu, Daniel (external author)
  •   Zhen, Jingfeng (external author)
  •   Zhi Quan (George) Zhou
  •   Towey, Dave (external author)

Publication Date


  • 2018

Citation


  • Pesu, D., Zhen, J., Zhou, Z. & Towey, D. (2018). A Monte Carlo Method for Metamorphic Testing of Machine Translation Services. 2018 ACM/IEEE International Workshop on Metamorphic Testing (pp. 38-45). New York: ACM.

Scopus Eid


  • 2-s2.0-85051257640

Start Page


  • 38

End Page


  • 45

Place Of Publication


  • New York

Abstract


  • © 2018 Association for Computing Machinery. With the growing popularity of machine translation services, it has become increasingly important to be able to assess their quality. However, the test oracle problem makes it difficult to conduct automated testing. In this paper, we propose a Monte Carlo method, in combination with metamorphic testing, to overcome the oracle problem. Using this method, we assessed the quality of three popular machine translation services' namely, Google Translate, Microsoft Translator, and Youdao Translate. We set the source language to be English, and the target languages included Chinese, French, Japanese, Korean, Portuguese, Russian, Spanish, and Swedish. A sample of 33,600 observations (involving a total of 100,800 actual translations) was collected and analyzed using a 3 56 factorial design. Based on this data, our model found Google Translate to be the best (in terms of the metamorphic relation used) for each and every target language considered. A trend for Indo- European languages producing better results was also identified.

Authors


  •   Pesu, Daniel (external author)
  •   Zhen, Jingfeng (external author)
  •   Zhi Quan (George) Zhou
  •   Towey, Dave (external author)

Publication Date


  • 2018

Citation


  • Pesu, D., Zhen, J., Zhou, Z. & Towey, D. (2018). A Monte Carlo Method for Metamorphic Testing of Machine Translation Services. 2018 ACM/IEEE International Workshop on Metamorphic Testing (pp. 38-45). New York: ACM.

Scopus Eid


  • 2-s2.0-85051257640

Start Page


  • 38

End Page


  • 45

Place Of Publication


  • New York