Zhou, Zhi Quan (George)

Overview

Dr Zhi Quan (George) Zhou received the BSc degree in Computer Science from Peking University, China, and the PhD degree in Software Engineering from The University of Hong Kong. His current research interests include software testing and debugging; cybersecurity and security testing; the interplay among software testing, machine learning, and big data; and self-driving vehicles. His research has been reported in Nature (https://go.nature.com/2TdWoM6). He was selected for a Virtual Earth Award by Microsoft Research, Redmond, USA, and a 2018 Researcher of the Year Gold Disruptor Award by the Australian Computer Society.

Zhou was the main contributor to some of the earliest research papers on metamorphic testing, and was one of the few pioneers who opened up and established the metamorphic testing research field. In 2016, he co-founded and chaired the IEEE/ACM International Workshop on Metamorphic Testing (MET '16), in conjunction with the 38th International Conference on Software Engineering (ICSE '16) in Austin, USA. As of July 2019, the annual MET workshop was recognized as one of the "popular conferences, workshops and journals related to software engineering" (p. 8, https://arxiv.org/abs/1907.09427).

Zhou was an invited keynote speaker at ICSE MET 2017 held in Buenos Aires, Argentina, and at the IEEE International Conference on Artificial Intelligence Testing in San Francisco, 2019 (slides), where he was also invited to organize the Special Track on Metamorphic Testing and Artificial Intelligence. He was an ACM SIGSOFT Webinar speaker and an ICSE '18 Technical Briefings speaker. Some of his work has been presented at the Journal-First sessions of ICSE '16, ICSE '19, and ICSE '20. He is a Program Committee Chair of the IEEE International Conference on Artificial Intelligence Testing 2020 (Keble College, Oxford University, UK).

Zhou has been invited to serve as a reviewer/adviser for various organizations including the Australian Research Council, the Research Grants Council of Hong Kong, and the Air Force Office of Scientific Research of the United States Air Force, and for various international journals including IEEE Transactions on Software Engineering, on Computers, on Reliability, on Services Computing, on Emerging Topics in Computational Intelligence, and so on. He has served as a guest editor for the Journal of Systems and Software.

Top Publications

Year	Title
2019	Metamorphic Testing of Driverless Cars Published in Communications of the ACM
2019	Metamorphic Relations for Enhancing System Understanding and Use Published in IEEE Transactions on Software Engineering
2019	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published in IEEE Transactions on Software Engineering
2016	Metamorphic testing for software quality assessment: a study of search engines Published in IEEE Transactions on Software Engineering
2011	Semi-proving: an integrated method for program proving, testing, and debugging Published in IEEE Transactions on Software Engineering
2018	Metamorphic Testing: A Review of Challenges and Opportunities Published in ACM Computing Surveys
2012	Automated functional testing of online search services Published in Software Testing, Verification and Reliability
2015	A revisit of three studies related to random testing Published in Science China Information Sciences
2018	A cost-effective software testing strategy employing online feedback information Published in Information Sciences
2016	Metamorphic testing for cybersecurity Published in Computer
2019	A Case Against Mission-Critical Applications of Machine Learning Published in Communications of the ACM

Research Areas

Research Overview

Software engineering, software testing and debugging, verification and validation, user experience, citation analysis.

Available as Research Supervisor

Selected Publications

Journal Article

Year	Title
2022	Metamorphic testing of named entity recognition systems: A case study Published In IET Software
2020	A Case Against Mission-Critical Applications of Machine Learning Published In Communications of the ACM
2020	A feedback control approach for preventing system resource exhaustion caused by software aging Published In Journal of Internet Technology
2020	Metamorphic Relations for Enhancing System Understanding and Use Published In IEEE Transactions on Software Engineering
2020	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published In IEEE Transactions on Software Engineering
2020	Metamorphic Testing of Driverless Cars Published In Communications of the ACM
2020	Metamorphic Testing of Mapping Software Published In Lecture Notes in Business Information Processing
2020	Metamorphic Testing: Testing the Untestable Published In IEEE Software
2020	A workload-based nonlinear approach for predicting available computing resources Published In Journal of Systems Engineering and Electronics
2020	Beating Random Test Case Prioritization Published In IEEE Transactions on Reliability
2020	Input Test Suites for Program Repair: A Novel Construction Method Based on Metamorphic Relations Published In IEEE Transactions on Reliability
2020	Software ageing process as an evolving dynamic system Published In IET Software
2019	A Case Against Mission-Critical Applications of Machine Learning Published In Communications of the ACM
2019	A feedback control approach for preventing system resource exhaustion caused by software aging Published In Journal of Internet Technology
2019	Metamorphic Relations for Enhancing System Understanding and Use Published In IEEE Transactions on Software Engineering
2019	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published In IEEE Transactions on Software Engineering
2019	Metamorphic Testing of Driverless Cars Published In Communications of the ACM
2019	Metamorphic Testing of Mapping Software Published In Lecture Notes in Business Information Processing
2019	Metamorphic Testing: Testing the Untestable Published In IEEE Software
2018	A cost-effective software testing strategy employing online feedback information Published In Information Sciences
2018	Adaptive Random Testing in Detecting Layout Faults of Web Applications Published In International Journal of Software Engineering and Knowledge Engineering
2018	Introduction to the Special Issue on Test Oracles Published In Journal of Systems and Software
2018	Metamorphic Testing: A Review of Challenges and Opportunities Published In ACM Computing Surveys
2016	High resolution SOM approach to improving anomaly detection in intrusion detection systems Published In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
2016	Metamorphic testing for cybersecurity Published In Computer
2016	Metamorphic testing for software quality assessment: a study of search engines Published In IEEE Transactions on Software Engineering
2015	A revisit of three studies related to random testing Published In Science China Information Sciences
2015	Using neural networks to forecast available system resources: an approach and empirical investigation Published In International Journal of Software Engineering and Knowledge Engineering
2012	Automated functional testing of online search services Published In Software Testing, Verification and Reliability
2012	Design and performance of a minimal real-time operating system in a safe language: Experience with java on the sun SPOT Published In Journal of Software
2011	On adaptive random testing through iterative partitioning Published In Journal of Information Science and Engineering
2011	Semi-proving: an integrated method for program proving, testing, and debugging Published In IEEE Transactions on Software Engineering
2010	Automated functional testing of online search services Published In Software Testing, Verification and Reliability
2010	Metamorphic testing of decision support systems: A case study Published In IET Software
2007	On Favourable Conditions for Adaptive Random Testing Published In International Journal of Software Engineering and Knowledge Engineering

Conference Paper

Year	Title
2020	An extended abstract of "metamorphic testing: Testing the untestable" Published In Proceedings - International Computer Software and Applications Conference
2020	Metamorphic Relations for Data Validation: A Case Study of Translated Text Messages Published In IEEE/ACM 4th International Workshop on Metamorphic Testing (MET 2019)
2020	Metamorphic exploration of an unsupervised clustering program Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2020	Metamorphic relations for detection of performance anomalies Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2020	Preparing Software Quality Assurance Professionals: Metamorphic Exploration for Machine Learning Published In TALE 2019 - 2019 IEEE International Conference on Engineering, Technology and Education
2020	The impact of a dot: Case studies of a noise metamorphic relation pattern Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2020	A Testing Tool for Machine Learning Applications Published In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020
2020	ARCAMETES: A Learning Approach for Metamorphic Exploration and Testing Published In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020
2020	Metamorphic Fuzz Testing of Autonomous Vehicles Published In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020
2020	Metamorphic Robustness Testing of Google Translate Published In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020
2019	An extended abstract of "metamorphic testing: Testing the untestable" Published In Proceedings - International Computer Software and Applications Conference
2019	Metamorphic Relations for Data Validation: A Case Study of Translated Text Messages Published In IEEE/ACM 4th International Workshop on Metamorphic Testing (MET 2019)
2019	Metamorphic exploration of an unsupervised clustering program Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2019	Metamorphic relations for detection of performance anomalies Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2019	Preparing Software Quality Assurance Professionals: Metamorphic Exploration for Machine Learning Published In TALE 2019 - 2019 IEEE International Conference on Engineering, Technology and Education
2019	The impact of a dot: Case studies of a noise metamorphic relation pattern Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2018	A Monte Carlo Method for Metamorphic Testing of Machine Translation Services Published In 2018 ACM/IEEE International Workshop on Metamorphic Testing
2018	Enhancing Supervised Classifications with Metamorphic Relations Published In Proceedings - International Conference on Software Engineering
2018	Metamorphic Testing for Adobe Analytics Data Collection JavaScript Library Published In 2018 ACM/IEEE International Conference on Software Engineering
2018	Metamorphic testing 20 years later: A hands-on introduction Published In International Conference on Software Engineering
2018	Metamorphic testing for machine translations: MT4MT Published In Proceedings - 25th Australasian Software Engineering Conference, ASWEC 2018
2018	Metamorphic testing of navigation software: A pilot study with Google Maps Published In Proceedings of the 51st Annual Hawaii International Conference on System Sciences (HICSS-51)
2017	Metamorphic Testing for Adobe Data Analytics Software Published In 2nd International Workshop on Metamorphic Testing
2017	Metamorphic Testing: Beyond Testing Numerical Computations Published In IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET)
2016	Message from the Workshop Chairs MET 2016 Published In Proceedings - International Conference on Software Engineering
2016	Metamorphic testing: A new student engagement approach for a new software testing paradigm Published In Proceedings of 2016 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE)
2014	Adaptive random testing for image comparison in regression web testing Published In 2014 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2014
2014	Testing model transformation programs using metamorphic testing Published In 26th International Conference on Software Engineering & Knowledge Engineering (SEKE 2014)
2013	On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions Published In 13th International Conference on Quality Software
2013	Similarity-based search for model checking: a pilot study with java pathfinder Published In 13th International Conference on Quality Software
2012	Function substitution: Towards constraint solving in software testing Published In QSIC 2012: 12th International Conference on Quality Software
2012	Metamorphic testing: Applications and integration with other methods: Tutorial synopsis Published In QSIC 2012: 12th International Conference on Quality Software
2012	On the fault-detection capabilities of adaptive random test case prioritization: Case studies with large test suites Published In Proceedings of the 45th Hawaii International Conference on System Sciences
2010	Using coverage information to guide test case selection in Adaptive Random Testing Published In IEEE International Computer Software and Applications Conference Workshops
2009	Improving software testing cost-effectiveness through dynamic partitioning Published In International Conference on Quality Software
2008	CSCI: a LEAP into the future Published In International Conference on Information and Emerging Technologies
2007	Automated software testing and analysis: techniques, practices and tools Published In Hawaii International Conference on System Sciences
2007	Developing real-time systems in Java on Macintosh Published In Apple University Consortium Academic and Developers Conference
2007	Enhanced random testing for programs with high dimensional input domains Published In International Conference on Software Engineering
2007	Software development of embedded systems on Macintosh Published In Apple University Consortium Academic and Developers Conference
2005	An effective testing method for end-user programmers Published In Proceedings of the 1st Workshop on End-User Software Engineering, WEUSE 2005

Investigator On

Year	Title
2021	Context-aware verification and validation framework for autonomous driving Awarded by: Australian Research Council (ARC) Funding Scheme: Discovery Projects
2017 - 2020	Design and Deployment of Practical Anonymous Access Systems Awarded by: Australian Research Council (ARC) Funding Scheme: Linkage Projects
2010 - 2015	EATandDREAM: Effective automatic Testing and Debugging for REAl-life eMbedded wireless communications software Awarded by: Australian Research Council (ARC) Funding Scheme: Linkage Projects
2007	On testing non-testable information retrieval systems with geographic components on the web Awarded by: Microsoft Research Funding Scheme: Virtual Earth Academic Research Collaboration Award

Impact Story

International rankings
Summary
An independent study conducted by American researchers identified Zhou as one of the top five researchers internationally in the important research field of test oracles [1]. Another independent study conducted by European researchers identified Zhou as one of the top four researchers internationally in metamorphic testing [2]. References: [1] R. A. P. Oliveira, U. Kanewala, and P. A. Nardi. "Automated test oracles: State of the art, taxonomies, and trends," In Advances in Computers, volume 95, pp. 113-199. Elsevier, 2014. Available online: <a href="http://iii.library.uow.edu.au/record=b2303118" target="_blank" rel="noopener">http://iii.library.uow.edu.au/record=b2303118</a> [2] S. Segura, G. Fraser, A. Sanchez, and A. Ruiz-Cortes, "A survey on metamorphic testing", IEEE Transactions on Software Engineering, 42(9), pp. 805-824, 2016. Available online: <a href="http://dx.doi.org/10.1109/TSE.2016.2532875" target="_blank" rel="noopener">http://dx.doi.org/10.1109/TSE.2016.2532875</a>

Detail View

Detail View Summary View
Detail of Impact
The following URL links to an invited webinar that introduces metamorphic testing, produced by the ACM: <a href="https://event.on24.com/wcc/r/1451736/8B5B5925E82FC9807CF83C84834A6F3D" target="_blank" rel="noopener">https://event.on24.com/wcc/r/1451736/8B5B5925E82FC9807CF83C84834A6F3D </a>
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Collaboration with industry
Summary
Zhou has been working with the IT industry on a series of software testing projects. Those industry partners include Adobe Inc., Microsoft Corporation, Partech Systems, and the Baidu Apollo autonomous driving team, among others. An independent review of Zhou's joint work with Adobe Inc. can be found <a href="https://semiwiki.com/eda/284509-innovation-in-verification-april-2020/" title="Innovation in Verification April 2020" target="_blank" rel="noopener">here</a>, where Paul Cunningham (GM of the Verification Group at Cadence) commented that "I like this. I see it as a way to do statistical anomaly-based QA. You compare a lot of runs, looking at distributions to spot bugs. I see a lot of applications: anything performance-related, heuristic-based, machine-learning-based will be naturally statistical. Distribution analyses can then reveal more complex issues than pass/fail analyses. MT gives us tools to find those kinds of problem." And Jim Hogan commented that "This is the first of the papers we've looked at in this series which to me is more than just a feature. This paper would definitely be worth putting money behind, trying to get to production. It looks like a product, perhaps a new class of verification tool. It might even work as a startup." Interested readers may refer to the full discussion <a href="https://semiwiki.com/eda/284509-innovation-in-verification-april-2020/" target="_blank" rel="noopener">Innovation in Verification April 2020</a>.

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Exposing Hidden Defects in Citation Statistics and Journal Impact Factors
Summary
Zhou initiated a "metamorphic robustness testing" method and led his research team to use this method to verify the major citation database systems <a href="https://www.scopus.com" target="_blank" rel="noopener">Scopus</a> and <a href="https://clarivate.com/products/web-of-science/" target="_blank" rel="noopener">Web of Science</a>, revealing robustness problems in citation counts and journal impact factors computed by these systems. A paper titled "Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors" by Zhi Quan Zhou, T.H. Tse, and Matt Witheridge has been published in IEEE Transactions on Software Engineering, the top journal in the field (URL for download: <a href="https://doi.org/10.1109/TSE.2019.2915065" target="_blank" rel="noopener">https://doi.org/10.1109/TSE.2019.2915065</a>).The high impact of the results lies in the finding that the presence of simple hyphens in a paper title adversely affects the citation statistics. Thus, metrics based on citation counts, which are commonly used for staff evaluations in universities, are not as robust as widely believed. This will be of interest not only to professors seeking tenure or promotion, but also to the senior management such as presidents, deans, and heads. The surprising results are applicable to all faculties in any university. Zhou et al.'s findings have been reported by the international media, including <a href="https://documents.uow.edu.au/~zhiquan/TheHyphenMan.pdf" target="_blank" rel="noopener">The Australian</a>, <a href="https://www.psychologytoday.com/intl/blog/the-athletes-way/201906/one-small-hyphen-in-title-could-diminish-academic-success" target="_blank" rel="noopener">Psychology Today</a><a href="https://www.sciencealert.com/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals" target="_blank" rel="noopener"></a>, <a href="https://cacm.acm.org/news/237400-a-single-punctuation-mark-has-been-skewing-our-entire-system-of-scientific-ranking/fulltext" target="_blank" rel="noopener">Communications of the ACM</a>, and <a href="https://www.eurekalert.org/" target="_blank" rel="noopener">EurekAlert!</a>, among others. On the Internet, the implications of Zhou et al.'s findings have been discussed from diverse perspectives, some of which extend beyond the conventional computing and citation domains. For example, see "<a href="https://www.slashgear.com/brave-new-world-predictions-are-coming-true-with-hyphens-31578773/" target="_blank" rel="noopener">Brave New World predictions are coming true with hyphens</a>" written by <a href="https://www.slashgear.com/author/chrisburns/" target="_blank" rel="noopener">Chris Burns</a> on <a href="https://www.slashgear.com/brave-new-world-predictions-are-coming-true-with-hyphens-31578773/" target="_blank" rel="noopener">SlashGear</a>. Inquiries have been made from various communities including, among others, the community of patients with <a href="https://www.3cr.org.au/chronicallychilled/episode-201901021800/asher-wolf-ehlers-danlos-syndrome" target="_blank" rel="noopener">Ehlers-Danlos Syndrome</a> (a <a href="https://en.wikipedia.org/wiki/List_of_syndromes" target="_blank" rel="noopener">condition</a> that has a hyphenated name and is known to be under-reported and under-researched). On 5 June 2019, the <a href="http://www.webofsciencegroup.com/" target="_blank" rel="noopener">Web of Science Group</a> posted a <a href="https://clarivate.com/webofsciencegroup/download/36464/" target="_blank" rel="noopener">response</a> to Zhou et al.'s paper on the <a href="https://clarivate.com/blog/science-research-connect/web-of-science-group-response-to-metamorphic-robustness-testing-exposing-hidden-defects-in-citation-statistics-and-journal-impact-factors/" target="_blank" rel="noopener">Clarivate Analytics website</a>. Feedback from the authors was reported by <a href="https://www.sciencealert.com/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals" target="_blank" rel="noopener">ScienceAlert</a>. On 7 August 2019, Zhou et al.'s work was reported by <a href="https://go.nature.com/2TdWoM6" target="_blank" rel="noopener">Nature</a> (https://go.nature.com/2TdWoM6).

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Detected and reported fatal bugs in self-driving cars eight days before the Uber crash in March 2018
Summary
On March 10, 2018, Zhou and his MPhil student Liqun Sun detected a critical fault in the real-world self-driving system <a href="http://apollo.auto" target="_blank" rel="noopener">Baidu Apollo</a>, and they reported this finding to the Apollo team on the same day (this was 8 days before <a href="https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg" target="_blank" rel="noopener">a fatal accident of a Uber's self-driving vehicle in Tempe, Arizona</a>). The fault caused the self-driving car's LiDAR perception system to ignore pedestrians or other obstacles on the roadway. The Apollo team confirmed the fault on March 19, 2018 (24 hours after <a href="https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg" target="_blank" rel="noopener">the Uber accident</a>, where the vehicle involved used exactly the same type of LiDAR sensor as Apollo). This study was reported in an article titled "Metamorphic Testing of Driverless Cars" by Zhou and Sun, published in <a href="https://cacm.acm.org/" target="_blank" rel="noopener">Communications of the ACM</a>, ACM's flagship magazine (<a href="https://doi.org/10.1145/3241979" target="_blank" rel="noopener">original paper</a>; <a href="https://doi.org/10.1145/3332409" target="_blank" rel="noopener">follow-up letter</a>). Zhou covered this story in his <a href="http://doi.org/10.5281/zenodo.2636979" target="_blank" rel="noopener">keynote speech</a> at the <a href="http://www.ieeeaitests.com/html/speakers.html" target="_blank" rel="noopener">IEEE International Conference on Artificial Intelligence Testing</a> (AITest 2019) held in San Francisco East Bay, California.

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators

Available as Research Supervisor

Advisees

Graduate Advising Relationship

Degree	Research Title	Advisee
Doctor of Philosophy	Dispersity-Based Test Case Prioritization	Liu, Chen
Doctor of Philosophy	Metamorphic Testing applied to citation systems and other border applications	Witheridge, Matt
Doctor of Philosophy (Integrated)	Smart Testing Approaches for intelligent Autonomous Systems: A Research Proposal	Zhu, Junting
Doctor of Philosophy	Software testing of web and cloud based applications	Han, Jiacheng
Doctor of Philosophy	A case for including metamorphic relations (MRs) into the testing standard for Advanced Driving Assist System (ADAS) and autonomous driving	Iqbal, Muhammad
Doctor of Philosophy (Integrated)	On Antomated Testing of Web Applications	Selay, Elmin

Education And Training

Ph.D. in Software Engineering, University of Hong Kong, Department of Computer Science 1998 - 2003

Awards And Honors

Researcher of the Year Gold Disruptor Award, conferred by Australian Computer Society, 2018
Virtual Earth Award, conferred by Microsoft Research, Redmond, USA, 2007

Keywords

- software engineering
- software testing
- test oracles
- autonomous driving
- cybersecurity

Orcid Id

http://orcid.org/0000-0003-1769-9367 (confirmed)

Scopus Id

56566917900

Top Publications

Year	Title
2019	Metamorphic Testing of Driverless Cars Published in Communications of the ACM
2019	Metamorphic Relations for Enhancing System Understanding and Use Published in IEEE Transactions on Software Engineering
2019	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published in IEEE Transactions on Software Engineering
2016	Metamorphic testing for software quality assessment: a study of search engines Published in IEEE Transactions on Software Engineering
2011	Semi-proving: an integrated method for program proving, testing, and debugging Published in IEEE Transactions on Software Engineering
2018	Metamorphic Testing: A Review of Challenges and Opportunities Published in ACM Computing Surveys
2012	Automated functional testing of online search services Published in Software Testing, Verification and Reliability
2015	A revisit of three studies related to random testing Published in Science China Information Sciences
2018	A cost-effective software testing strategy employing online feedback information Published in Information Sciences
2016	Metamorphic testing for cybersecurity Published in Computer
2019	A Case Against Mission-Critical Applications of Machine Learning Published in Communications of the ACM

Research Areas

Research Overview

Software engineering, software testing and debugging, verification and validation, user experience, citation analysis.

Selected Publications

Journal Article

Year	Title
2022	Metamorphic testing of named entity recognition systems: A case study Published In IET Software
2020	A Case Against Mission-Critical Applications of Machine Learning Published In Communications of the ACM
2020	A feedback control approach for preventing system resource exhaustion caused by software aging Published In Journal of Internet Technology
2020	Metamorphic Relations for Enhancing System Understanding and Use Published In IEEE Transactions on Software Engineering
2020	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published In IEEE Transactions on Software Engineering
2020	Metamorphic Testing of Driverless Cars Published In Communications of the ACM
2020	Metamorphic Testing of Mapping Software Published In Lecture Notes in Business Information Processing
2020	Metamorphic Testing: Testing the Untestable Published In IEEE Software
2020	A workload-based nonlinear approach for predicting available computing resources Published In Journal of Systems Engineering and Electronics
2020	Beating Random Test Case Prioritization Published In IEEE Transactions on Reliability
2020	Input Test Suites for Program Repair: A Novel Construction Method Based on Metamorphic Relations Published In IEEE Transactions on Reliability
2020	Software ageing process as an evolving dynamic system Published In IET Software
2019	A Case Against Mission-Critical Applications of Machine Learning Published In Communications of the ACM
2019	A feedback control approach for preventing system resource exhaustion caused by software aging Published In Journal of Internet Technology
2019	Metamorphic Relations for Enhancing System Understanding and Use Published In IEEE Transactions on Software Engineering
2019	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published In IEEE Transactions on Software Engineering
2019	Metamorphic Testing of Driverless Cars Published In Communications of the ACM
2019	Metamorphic Testing of Mapping Software Published In Lecture Notes in Business Information Processing
2019	Metamorphic Testing: Testing the Untestable Published In IEEE Software
2018	A cost-effective software testing strategy employing online feedback information Published In Information Sciences
2018	Adaptive Random Testing in Detecting Layout Faults of Web Applications Published In International Journal of Software Engineering and Knowledge Engineering
2018	Introduction to the Special Issue on Test Oracles Published In Journal of Systems and Software
2018	Metamorphic Testing: A Review of Challenges and Opportunities Published In ACM Computing Surveys
2016	High resolution SOM approach to improving anomaly detection in intrusion detection systems Published In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
2016	Metamorphic testing for cybersecurity Published In Computer
2016	Metamorphic testing for software quality assessment: a study of search engines Published In IEEE Transactions on Software Engineering
2015	A revisit of three studies related to random testing Published In Science China Information Sciences
2015	Using neural networks to forecast available system resources: an approach and empirical investigation Published In International Journal of Software Engineering and Knowledge Engineering
2012	Automated functional testing of online search services Published In Software Testing, Verification and Reliability
2012	Design and performance of a minimal real-time operating system in a safe language: Experience with java on the sun SPOT Published In Journal of Software
2011	On adaptive random testing through iterative partitioning Published In Journal of Information Science and Engineering
2011	Semi-proving: an integrated method for program proving, testing, and debugging Published In IEEE Transactions on Software Engineering
2010	Automated functional testing of online search services Published In Software Testing, Verification and Reliability
2010	Metamorphic testing of decision support systems: A case study Published In IET Software
2007	On Favourable Conditions for Adaptive Random Testing Published In International Journal of Software Engineering and Knowledge Engineering

Conference Paper

Year	Title
2020	An extended abstract of "metamorphic testing: Testing the untestable" Published In Proceedings - International Computer Software and Applications Conference
2020	Metamorphic Relations for Data Validation: A Case Study of Translated Text Messages Published In IEEE/ACM 4th International Workshop on Metamorphic Testing (MET 2019)
2020	Metamorphic exploration of an unsupervised clustering program Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2020	Metamorphic relations for detection of performance anomalies Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2020	Preparing Software Quality Assurance Professionals: Metamorphic Exploration for Machine Learning Published In TALE 2019 - 2019 IEEE International Conference on Engineering, Technology and Education
2020	The impact of a dot: Case studies of a noise metamorphic relation pattern Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2020	A Testing Tool for Machine Learning Applications Published In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020
2020	ARCAMETES: A Learning Approach for Metamorphic Exploration and Testing Published In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020
2020	Metamorphic Fuzz Testing of Autonomous Vehicles Published In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020
2020	Metamorphic Robustness Testing of Google Translate Published In Proceedings - 2020 IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020
2019	An extended abstract of "metamorphic testing: Testing the untestable" Published In Proceedings - International Computer Software and Applications Conference
2019	Metamorphic Relations for Data Validation: A Case Study of Translated Text Messages Published In IEEE/ACM 4th International Workshop on Metamorphic Testing (MET 2019)
2019	Metamorphic exploration of an unsupervised clustering program Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2019	Metamorphic relations for detection of performance anomalies Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2019	Preparing Software Quality Assurance Professionals: Metamorphic Exploration for Machine Learning Published In TALE 2019 - 2019 IEEE International Conference on Engineering, Technology and Education
2019	The impact of a dot: Case studies of a noise metamorphic relation pattern Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019
2018	A Monte Carlo Method for Metamorphic Testing of Machine Translation Services Published In 2018 ACM/IEEE International Workshop on Metamorphic Testing
2018	Enhancing Supervised Classifications with Metamorphic Relations Published In Proceedings - International Conference on Software Engineering
2018	Metamorphic Testing for Adobe Analytics Data Collection JavaScript Library Published In 2018 ACM/IEEE International Conference on Software Engineering
2018	Metamorphic testing 20 years later: A hands-on introduction Published In International Conference on Software Engineering
2018	Metamorphic testing for machine translations: MT4MT Published In Proceedings - 25th Australasian Software Engineering Conference, ASWEC 2018
2018	Metamorphic testing of navigation software: A pilot study with Google Maps Published In Proceedings of the 51st Annual Hawaii International Conference on System Sciences (HICSS-51)
2017	Metamorphic Testing for Adobe Data Analytics Software Published In 2nd International Workshop on Metamorphic Testing
2017	Metamorphic Testing: Beyond Testing Numerical Computations Published In IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET)
2016	Message from the Workshop Chairs MET 2016 Published In Proceedings - International Conference on Software Engineering
2016	Metamorphic testing: A new student engagement approach for a new software testing paradigm Published In Proceedings of 2016 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE)
2014	Adaptive random testing for image comparison in regression web testing Published In 2014 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2014
2014	Testing model transformation programs using metamorphic testing Published In 26th International Conference on Software Engineering & Knowledge Engineering (SEKE 2014)
2013	On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions Published In 13th International Conference on Quality Software
2013	Similarity-based search for model checking: a pilot study with java pathfinder Published In 13th International Conference on Quality Software
2012	Function substitution: Towards constraint solving in software testing Published In QSIC 2012: 12th International Conference on Quality Software
2012	Metamorphic testing: Applications and integration with other methods: Tutorial synopsis Published In QSIC 2012: 12th International Conference on Quality Software
2012	On the fault-detection capabilities of adaptive random test case prioritization: Case studies with large test suites Published In Proceedings of the 45th Hawaii International Conference on System Sciences
2010	Using coverage information to guide test case selection in Adaptive Random Testing Published In IEEE International Computer Software and Applications Conference Workshops
2009	Improving software testing cost-effectiveness through dynamic partitioning Published In International Conference on Quality Software
2008	CSCI: a LEAP into the future Published In International Conference on Information and Emerging Technologies
2007	Automated software testing and analysis: techniques, practices and tools Published In Hawaii International Conference on System Sciences
2007	Developing real-time systems in Java on Macintosh Published In Apple University Consortium Academic and Developers Conference
2007	Enhanced random testing for programs with high dimensional input domains Published In International Conference on Software Engineering
2007	Software development of embedded systems on Macintosh Published In Apple University Consortium Academic and Developers Conference
2005	An effective testing method for end-user programmers Published In Proceedings of the 1st Workshop on End-User Software Engineering, WEUSE 2005

Investigator On

Year	Title
2021	Context-aware verification and validation framework for autonomous driving Awarded by: Australian Research Council (ARC) Funding Scheme: Discovery Projects
2017 - 2020	Design and Deployment of Practical Anonymous Access Systems Awarded by: Australian Research Council (ARC) Funding Scheme: Linkage Projects
2010 - 2015	EATandDREAM: Effective automatic Testing and Debugging for REAl-life eMbedded wireless communications software Awarded by: Australian Research Council (ARC) Funding Scheme: Linkage Projects
2007	On testing non-testable information retrieval systems with geographic components on the web Awarded by: Microsoft Research Funding Scheme: Virtual Earth Academic Research Collaboration Award

Impact Story

International rankings
Summary
An independent study conducted by American researchers identified Zhou as one of the top five researchers internationally in the important research field of test oracles [1]. Another independent study conducted by European researchers identified Zhou as one of the top four researchers internationally in metamorphic testing [2]. References: [1] R. A. P. Oliveira, U. Kanewala, and P. A. Nardi. "Automated test oracles: State of the art, taxonomies, and trends," In Advances in Computers, volume 95, pp. 113-199. Elsevier, 2014. Available online: <a href="http://iii.library.uow.edu.au/record=b2303118" target="_blank" rel="noopener">http://iii.library.uow.edu.au/record=b2303118</a> [2] S. Segura, G. Fraser, A. Sanchez, and A. Ruiz-Cortes, "A survey on metamorphic testing", IEEE Transactions on Software Engineering, 42(9), pp. 805-824, 2016. Available online: <a href="http://dx.doi.org/10.1109/TSE.2016.2532875" target="_blank" rel="noopener">http://dx.doi.org/10.1109/TSE.2016.2532875</a>

Detail View

Detail View Summary View
Detail of Impact
The following URL links to an invited webinar that introduces metamorphic testing, produced by the ACM: <a href="https://event.on24.com/wcc/r/1451736/8B5B5925E82FC9807CF83C84834A6F3D" target="_blank" rel="noopener">https://event.on24.com/wcc/r/1451736/8B5B5925E82FC9807CF83C84834A6F3D </a>
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Collaboration with industry
Summary
Zhou has been working with the IT industry on a series of software testing projects. Those industry partners include Adobe Inc., Microsoft Corporation, Partech Systems, and the Baidu Apollo autonomous driving team, among others. An independent review of Zhou's joint work with Adobe Inc. can be found <a href="https://semiwiki.com/eda/284509-innovation-in-verification-april-2020/" title="Innovation in Verification April 2020" target="_blank" rel="noopener">here</a>, where Paul Cunningham (GM of the Verification Group at Cadence) commented that "I like this. I see it as a way to do statistical anomaly-based QA. You compare a lot of runs, looking at distributions to spot bugs. I see a lot of applications: anything performance-related, heuristic-based, machine-learning-based will be naturally statistical. Distribution analyses can then reveal more complex issues than pass/fail analyses. MT gives us tools to find those kinds of problem." And Jim Hogan commented that "This is the first of the papers we've looked at in this series which to me is more than just a feature. This paper would definitely be worth putting money behind, trying to get to production. It looks like a product, perhaps a new class of verification tool. It might even work as a startup." Interested readers may refer to the full discussion <a href="https://semiwiki.com/eda/284509-innovation-in-verification-april-2020/" target="_blank" rel="noopener">Innovation in Verification April 2020</a>.

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Exposing Hidden Defects in Citation Statistics and Journal Impact Factors
Summary
Zhou initiated a "metamorphic robustness testing" method and led his research team to use this method to verify the major citation database systems <a href="https://www.scopus.com" target="_blank" rel="noopener">Scopus</a> and <a href="https://clarivate.com/products/web-of-science/" target="_blank" rel="noopener">Web of Science</a>, revealing robustness problems in citation counts and journal impact factors computed by these systems. A paper titled "Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors" by Zhi Quan Zhou, T.H. Tse, and Matt Witheridge has been published in IEEE Transactions on Software Engineering, the top journal in the field (URL for download: <a href="https://doi.org/10.1109/TSE.2019.2915065" target="_blank" rel="noopener">https://doi.org/10.1109/TSE.2019.2915065</a>).The high impact of the results lies in the finding that the presence of simple hyphens in a paper title adversely affects the citation statistics. Thus, metrics based on citation counts, which are commonly used for staff evaluations in universities, are not as robust as widely believed. This will be of interest not only to professors seeking tenure or promotion, but also to the senior management such as presidents, deans, and heads. The surprising results are applicable to all faculties in any university. Zhou et al.'s findings have been reported by the international media, including <a href="https://documents.uow.edu.au/~zhiquan/TheHyphenMan.pdf" target="_blank" rel="noopener">The Australian</a>, <a href="https://www.psychologytoday.com/intl/blog/the-athletes-way/201906/one-small-hyphen-in-title-could-diminish-academic-success" target="_blank" rel="noopener">Psychology Today</a><a href="https://www.sciencealert.com/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals" target="_blank" rel="noopener"></a>, <a href="https://cacm.acm.org/news/237400-a-single-punctuation-mark-has-been-skewing-our-entire-system-of-scientific-ranking/fulltext" target="_blank" rel="noopener">Communications of the ACM</a>, and <a href="https://www.eurekalert.org/" target="_blank" rel="noopener">EurekAlert!</a>, among others. On the Internet, the implications of Zhou et al.'s findings have been discussed from diverse perspectives, some of which extend beyond the conventional computing and citation domains. For example, see "<a href="https://www.slashgear.com/brave-new-world-predictions-are-coming-true-with-hyphens-31578773/" target="_blank" rel="noopener">Brave New World predictions are coming true with hyphens</a>" written by <a href="https://www.slashgear.com/author/chrisburns/" target="_blank" rel="noopener">Chris Burns</a> on <a href="https://www.slashgear.com/brave-new-world-predictions-are-coming-true-with-hyphens-31578773/" target="_blank" rel="noopener">SlashGear</a>. Inquiries have been made from various communities including, among others, the community of patients with <a href="https://www.3cr.org.au/chronicallychilled/episode-201901021800/asher-wolf-ehlers-danlos-syndrome" target="_blank" rel="noopener">Ehlers-Danlos Syndrome</a> (a <a href="https://en.wikipedia.org/wiki/List_of_syndromes" target="_blank" rel="noopener">condition</a> that has a hyphenated name and is known to be under-reported and under-researched). On 5 June 2019, the <a href="http://www.webofsciencegroup.com/" target="_blank" rel="noopener">Web of Science Group</a> posted a <a href="https://clarivate.com/webofsciencegroup/download/36464/" target="_blank" rel="noopener">response</a> to Zhou et al.'s paper on the <a href="https://clarivate.com/blog/science-research-connect/web-of-science-group-response-to-metamorphic-robustness-testing-exposing-hidden-defects-in-citation-statistics-and-journal-impact-factors/" target="_blank" rel="noopener">Clarivate Analytics website</a>. Feedback from the authors was reported by <a href="https://www.sciencealert.com/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals" target="_blank" rel="noopener">ScienceAlert</a>. On 7 August 2019, Zhou et al.'s work was reported by <a href="https://go.nature.com/2TdWoM6" target="_blank" rel="noopener">Nature</a> (https://go.nature.com/2TdWoM6).

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Detected and reported fatal bugs in self-driving cars eight days before the Uber crash in March 2018
Summary
On March 10, 2018, Zhou and his MPhil student Liqun Sun detected a critical fault in the real-world self-driving system <a href="http://apollo.auto" target="_blank" rel="noopener">Baidu Apollo</a>, and they reported this finding to the Apollo team on the same day (this was 8 days before <a href="https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg" target="_blank" rel="noopener">a fatal accident of a Uber's self-driving vehicle in Tempe, Arizona</a>). The fault caused the self-driving car's LiDAR perception system to ignore pedestrians or other obstacles on the roadway. The Apollo team confirmed the fault on March 19, 2018 (24 hours after <a href="https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg" target="_blank" rel="noopener">the Uber accident</a>, where the vehicle involved used exactly the same type of LiDAR sensor as Apollo). This study was reported in an article titled "Metamorphic Testing of Driverless Cars" by Zhou and Sun, published in <a href="https://cacm.acm.org/" target="_blank" rel="noopener">Communications of the ACM</a>, ACM's flagship magazine (<a href="https://doi.org/10.1145/3241979" target="_blank" rel="noopener">original paper</a>; <a href="https://doi.org/10.1145/3332409" target="_blank" rel="noopener">follow-up letter</a>). Zhou covered this story in his <a href="http://doi.org/10.5281/zenodo.2636979" target="_blank" rel="noopener">keynote speech</a> at the <a href="http://www.ieeeaitests.com/html/speakers.html" target="_blank" rel="noopener">IEEE International Conference on Artificial Intelligence Testing</a> (AITest 2019) held in San Francisco East Bay, California.

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators

Advisees

Graduate Advising Relationship

Degree	Research Title	Advisee
Doctor of Philosophy	Dispersity-Based Test Case Prioritization	Liu, Chen
Doctor of Philosophy	Metamorphic Testing applied to citation systems and other border applications	Witheridge, Matt
Doctor of Philosophy (Integrated)	Smart Testing Approaches for intelligent Autonomous Systems: A Research Proposal	Zhu, Junting
Doctor of Philosophy	Software testing of web and cloud based applications	Han, Jiacheng
Doctor of Philosophy	A case for including metamorphic relations (MRs) into the testing standard for Advanced Driving Assist System (ADAS) and autonomous driving	Iqbal, Muhammad
Doctor of Philosophy (Integrated)	On Antomated Testing of Web Applications	Selay, Elmin

Education And Training

Ph.D. in Software Engineering, University of Hong Kong, Department of Computer Science 1998 - 2003

Awards And Honors

Researcher of the Year Gold Disruptor Award, conferred by Australian Computer Society, 2018
Virtual Earth Award, conferred by Microsoft Research, Redmond, USA, 2007

Keywords

- software engineering
- software testing
- test oracles
- autonomous driving
- cybersecurity

Orcid Id

http://orcid.org/0000-0003-1769-9367 (confirmed)

Scopus Id

56566917900

Overview

Overview

Top Publications

Research Areas

Research Overview

Available as Research Supervisor

Publications

Selected Publications

Journal Article

Conference Paper

Funding

Investigator On

Impact

Impact Story

International rankings

Collaboration with industry

Exposing Hidden Defects in Citation Statistics and Journal Impact Factors

Detected and reported fatal bugs in self-driving cars eight days before the Uber crash in March 2018

Supervision

Available as Research Supervisor

Advisees

Graduate Advising Relationship

Background

Education And Training

Awards And Honors

Keywords

Identity

Orcid Id

Scopus Id

Overview

Top Publications

Research Areas

Research Overview

Publications

Selected Publications

Journal Article

Conference Paper

Funding

Investigator On

Impact

Impact Story

International rankings

Collaboration with industry

Exposing Hidden Defects in Citation Statistics and Journal Impact Factors

Detected and reported fatal bugs in self-driving cars eight days before the Uber crash in March 2018

Supervision

Advisees

Graduate Advising Relationship

Background

Education And Training

Awards And Honors

Keywords

Identity

Orcid Id

Scopus Id

Visualisations

Research Areas