Zhi Quan (George) Zhou

Overview

Dr Zhi Quan (George) Zhou received the BSc degree in Computer Science from Peking University, China, and the PhD degree in Software Engineering from The University of Hong Kong. His current research interests include software testing and debugging; cybersecurity and security testing; the interplay among software testing, machine learning, and big data; and self-driving vehicles. His research has been reported in Nature (https://go.nature.com/2TdWoM6). He was selected for a Virtual Earth Award by Microsoft Research, Redmond, USA, and a 2018 Researcher of the Year Gold Disruptor Award by the Australian Computer Society.

Zhou was the main contributor to some of the earliest research papers on metamorphic testing, and was one of the few pioneers who opened up and established the metamorphic testing research field. In 2016, he co-founded and chaired the IEEE/ACM 1st International Workshop on Metamorphic Testing (MET '16), in conjunction with the 38th International Conference on Software Engineering (ICSE '16) in Austin, USA. As of July 2019, the annual MET workshop was recognized as one of the "popular conferences, workshops and journals related to software engineering" (p. 8, https://arxiv.org/abs/1907.09427).

Zhou was an invited keynote speaker at ICSE MET 2017 held in Buenos Aires, Argentina, and at the IEEE International Conference on Artificial Intelligence Testing in San Francisco, 2019 (slides), where he was also invited to organize the Special Track on Metamorphic Testing and Artificial Intelligence. He was an ACM SIGSOFT Webinar speaker and an ICSE '18 Technical Briefings speaker. Part of his work has been presented at the Journal-First sessions of ICSE '16, ICSE '19, and ICSE '20. He is a Program Committee Chair of the IEEE International Conference on Artificial Intelligence Testing 2020 (Keble College, Oxford University, UK).

Zhou has been invited to serve as a reviewer/adviser for various organizations including the Australian Research Council, the Research Grants Council of Hong Kong, and the Air Force Office of Scientific Research of the United States Air Force, and for various international journals including IEEE Transactions on Software Engineering, on Computers, on Reliability, on Services Computing, on Emerging Topics in Computational Intelligence, and so on. He has served as a guest editor for the Journal of Systems and Software.

Top Publications

Year	Title
2019	Metamorphic Testing of Driverless Cars Published in Communications of the ACM
2019	Metamorphic Relations for Enhancing System Understanding and Use Published in IEEE Transactions on Software Engineering
2019	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published in IEEE Transactions on Software Engineering
2016	Metamorphic testing for software quality assessment: a study of search engines Published in IEEE Transactions on Software Engineering
2011	Semi-proving: an integrated method for program proving, testing, and debugging Published in IEEE Transactions on Software Engineering
2018	Metamorphic Testing: A Review of Challenges and Opportunities Published in ACM Computing Surveys
2012	Automated functional testing of online search services Published in Software Testing, Verification and Reliability
2015	A revisit of three studies related to random testing Published in Science China Information Sciences
2018	A cost-effective software testing strategy employing online feedback information Published in Information Sciences
2016	Metamorphic testing for cybersecurity Published in Computer
2019	A Case Against Mission-Critical Applications of Machine Learning Published in Communications of the ACM

Research Areas

Research Overview

Software engineering, software testing and debugging, verification and validation, user experience, citation analysis.

Available as Research Supervisor

Selected Publications

Journal Article

Year	Title
2020	A workload-based nonlinear approach for predicting available computing resources Published In Journal of Systems Engineering and Electronics Volume 31 Pages 224 - 230 Altmetric
2020	Beating Random Test Case Prioritization Published In IEEE Transactions on Reliability Volume Online First Pages 1 - 22 ISSN 0018-9529 Altmetric
2019	A Case Against Mission-Critical Applications of Machine Learning Published In Communications of the ACM Volume 62 Page p. 9 ISSN 0001-0782 Altmetric
2019	A feedback control approach for preventing system resource exhaustion caused by software aging Published In Journal of Internet Technology Volume 20 Pages 1513 - 1522 ISSN 1607-9264 Altmetric
2019	Metamorphic Relations for Enhancing System Understanding and Use Published In IEEE Transactions on Software Engineering Volume Online First Pages 1 - 40 ISSN 0098-5589 Altmetric
2019	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published In IEEE Transactions on Software Engineering Volume Online First Pages 1 - 22 ISSN 0098-5589 Altmetric
2019	Metamorphic Testing of Driverless Cars Published In Communications of the ACM Volume 62 Pages 61 - 67 ISSN 0001-0782 Altmetric
2019	Metamorphic Testing of Mapping Software Published In Lecture Notes in Business Information Processing Volume 347 Pages 1 - 20 Altmetric
2019	Metamorphic Testing: Testing the Untestable Published In IEEE Software Volume Online First Pages 1 - 12 ISSN 0740-7459 Altmetric
2018	A cost-effective software testing strategy employing online feedback information Published In Information Sciences Volume 422 Pages 318 - 335 ISSN 0020-0255 Altmetric
2018	Adaptive Random Testing in Detecting Layout Faults of Web Applications Published In International Journal of Software Engineering and Knowledge Engineering Volume 28 Pages 1399 - 1428 ISSN 0218-1940 Altmetric
2018	Introduction to the Special Issue on Test Oracles Published In Journal of Systems and Software Volume 136 Page p. 187 ISSN 0164-1212 Altmetric
2018	Metamorphic Testing: A Review of Challenges and Opportunities Published In ACM Computing Surveys Volume 51 Pages 4:1 - 4:27 ISSN 0360-0300 Altmetric
2016	High resolution SOM approach to improving anomaly detection in intrusion detection systems Published In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) Volume 9992 Pages 191 - 199 Altmetric
2016	Metamorphic testing for cybersecurity Published In Computer Volume 49 Pages 48 - 55 ISSN 0018-9162 Altmetric
2016	Metamorphic testing for software quality assessment: a study of search engines Published In IEEE Transactions on Software Engineering Volume 42 Pages 260 - 280 ISSN 0098-5589 Altmetric
2015	A revisit of three studies related to random testing Published In Science China Information Sciences Volume 58 Pages 052104:1 - 052104:9 ISSN 1674-733X Altmetric
2015	Using neural networks to forecast available system resources: an approach and empirical investigation Published In International Journal of Software Engineering and Knowledge Engineering Volume 25 Pages 781 - 802 ISSN 0218-1940 Altmetric
2012	Automated functional testing of online search services Published In Software Testing, Verification and Reliability Volume 22 Pages 221 - 243 ISSN 0960-0833 Altmetric
2012	Design and performance of a minimal real-time operating system in a safe language: Experience with java on the sun SPOT Published In Journal of Software Volume 7 Pages 1835 - 1844 ISSN 1796-217X Altmetric
2011	On adaptive random testing through iterative partitioning Published In Journal of Information Science and Engineering Volume 27 Pages 1449 - 1472 ISSN 1016-2364
2011	Semi-proving: an integrated method for program proving, testing, and debugging Published In IEEE Transactions on Software Engineering Volume 37 Pages 109 - 125 ISSN 0098-5589 Altmetric
2010	Automated functional testing of online search services Published In Software Testing, Verification and Reliability Volume Aug Pages 1 - 29 ISSN 0960-0833 Altmetric
2010	Metamorphic testing of decision support systems: A case study Published In IET Software Volume 4 Pages 294 - 301 ISSN 1751-8806 Altmetric
2007	On Favourable Conditions for Adaptive Random Testing Published In International Journal of Software Engineering and Knowledge Engineering Volume 17 Pages 805 - 825 ISSN 0218-1940 Altmetric

Conference Paper

Year	Title
2019	Metamorphic Relations for Data Validation: A Case Study of Translated Text Messages Published In IEEE/ACM 4th International Workshop on Metamorphic Testing (MET 2019) Pages 70 - 75 Altmetric
2019	Metamorphic exploration of an unsupervised clustering program Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019 Pages 48 - 54 Altmetric
2019	Metamorphic relations for detection of performance anomalies Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019 Pages 63 - 69 Altmetric
2019	The impact of a dot: Case studies of a noise metamorphic relation pattern Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019 Pages 17 - 23 Altmetric
2018	A Monte Carlo Method for Metamorphic Testing of Machine Translation Services Published In 2018 ACM/IEEE International Workshop on Metamorphic Testing Pages 38 - 45 Altmetric
2018	Enhancing Supervised Classifications with Metamorphic Relations Published In Proceedings - International Conference on Software Engineering Pages 46 - 53 Altmetric
2018	Metamorphic Testing for Adobe Analytics Data Collection JavaScript Library Published In 2018 ACM/IEEE International Conference on Software Engineering Pages 34 - 37 Altmetric
2018	Metamorphic testing 20 years later: A hands-on introduction Published In International Conference on Software Engineering Pages 538 - 539 Altmetric
2018	Metamorphic testing for machine translations: MT4MT Published In Proceedings - 25th Australasian Software Engineering Conference, ASWEC 2018 Pages 96 - 100 Altmetric
2018	Metamorphic testing of navigation software: A pilot study with Google Maps Published In Proceedings of the 51st Annual Hawaii International Conference on System Sciences (HICSS-51) Pages 5687 - 5696
2017	Metamorphic Testing for Adobe Data Analytics Software Published In 2nd International Workshop on Metamorphic Testing Pages 21 - 27 Altmetric
2017	Metamorphic Testing: Beyond Testing Numerical Computations Published In IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET) Pages 1 - 1 Altmetric
2016	Metamorphic testing: A new student engagement approach for a new software testing paradigm Published In Proceedings of 2016 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE) Pages 218 - 228 Altmetric
2014	Adaptive random testing for image comparison in regression web testing Published In 2014 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2014 Pages 1 - 7 Altmetric
2014	Testing model transformation programs using metamorphic testing Published In 26th International Conference on Software Engineering & Knowledge Engineering (SEKE 2014) Pages 94 - 99
2013	On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions Published In 13th International Conference on Quality Software Pages 153 - 162 Altmetric
2013	Similarity-based search for model checking: a pilot study with java pathfinder Published In 13th International Conference on Quality Software Pages 238 - 244 Altmetric
2012	Function substitution: Towards constraint solving in software testing Published In QSIC 2012: 12th International Conference on Quality Software Pages 31 - 40 Altmetric
2012	Metamorphic testing: Applications and integration with other methods: Tutorial synopsis Published In QSIC 2012: 12th International Conference on Quality Software Pages 285 - 288 Altmetric
2012	On the fault-detection capabilities of adaptive random test case prioritization: Case studies with large test suites Published In Proceedings of the 45th Hawaii International Conference on System Sciences Pages 5584 - 5593 Altmetric
2010	Using coverage information to guide test case selection in Adaptive Random Testing Published In IEEE International Computer Software and Applications Conference Workshops Pages 208 - 213 Altmetric
2009	Improving software testing cost-effectiveness through dynamic partitioning Published In International Conference on Quality Software Pages 249 - 258 Altmetric
2008	CSCI: a LEAP into the future Published In International Conference on Information and Emerging Technologies Pages 1 - 14
2007	Automated software testing and analysis: techniques, practices and tools Published In Hawaii International Conference on System Sciences Pages 260 - 260 Altmetric
2007	Developing real-time systems in Java on Macintosh Published In Apple University Consortium Academic and Developers Conference Pages 11.1 - 11.9
2007	Enhanced random testing for programs with high dimensional input domains Published In International Conference on Software Engineering Pages 135 - 140
2007	Software development of embedded systems on Macintosh Published In Apple University Consortium Academic and Developers Conference Pages 10.1 - 10.9

Investigator On

Year	Title
2007	On testing non-testable information retrieval systems with geographic components on the web Awarded by: Microsoft Research Funding Scheme: Virtual Earth Academic Research Collaboration Award
	Design and Deployment of Practical Anonymous Access Systems Awarded by: Australian Research Council (ARC) Funding Scheme: Linkage Projects
	EATandDREAM: Effective automatic Testing and Debugging for REAl-life eMbedded wireless communications software Awarded by: Australian Research Council (ARC) Funding Scheme: Linkage Projects

Impact Story

International rankings
Summary
An independent study conducted by American researchers identified Zhou as one of the top five researchers internationally in the important research field of test oracles [1]. Another independent study conducted by European researchers identified Zhou as one of the top four researchers internationally in metamorphic testing [2]. References: [1] R. A. P. Oliveira, U. Kanewala, and P. A. Nardi. "Automated test oracles: State of the art, taxonomies, and trends," In Advances in Computers, volume 95, pp. 113-199. Elsevier, 2014. Available online: <a href="http://iii.library.uow.edu.au/record=b2303118" target="_blank" rel="noopener">http://iii.library.uow.edu.au/record=b2303118</a> [2] S. Segura, G. Fraser, A. Sanchez, and A. Ruiz-Cortes, "A survey on metamorphic testing", IEEE Transactions on Software Engineering, 42(9), pp. 805-824, 2016. Available online: <a href="http://dx.doi.org/10.1109/TSE.2016.2532875" target="_blank" rel="noopener">http://dx.doi.org/10.1109/TSE.2016.2532875</a>

Detail View

Detail View Summary View
Detail of Impact
The following URL links to an invited webinar that introduces metamorphic testing, produced by the ACM: <a href="https://event.on24.com/wcc/r/1451736/8B5B5925E82FC9807CF83C84834A6F3D" target="_blank" rel="noopener">https://event.on24.com/wcc/r/1451736/8B5B5925E82FC9807CF83C84834A6F3D </a>
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Collaboration with industry
Summary
Zhou has been working with the IT industry on a series of software testing projects. Those industry partners include Adobe Inc., Microsoft Corporation, Partech Systems, and the Baidu Apollo autonomous driving team, among others. An independent review of Zhou's joint work with Adobe Inc. can be found <a href="https://semiwiki.com/eda/284509-innovation-in-verification-april-2020/" title="Innovation in Verification April 2020" target="_blank" rel="noopener">here</a>, where Paul Cunningham (GM of the Verification Group at Cadence) commented that "I like this. I see it as a way to do statistical anomaly-based QA. You compare a lot of runs, looking at distributions to spot bugs. I see a lot of applications: anything performance-related, heuristic-based, machine-learning-based will be naturally statistical. Distribution analyses can then reveal more complex issues than pass/fail analyses. MT gives us tools to find those kinds of problem." And Jim Hogan commented that "This is the first of the papers we've looked at in this series which to me is more than just a feature. This paper would definitely be worth putting money behind, trying to get to production. It looks like a product, perhaps a new class of verification tool. It might even work as a startup." Interested readers may refer to the full discussion <a href="https://semiwiki.com/eda/284509-innovation-in-verification-april-2020/" target="_blank" rel="noopener">Innovation in Verification April 2020</a>.

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Exposing Hidden Defects in Citation Statistics and Journal Impact Factors
Summary
Zhou initiated a "metamorphic robustness testing" method and led his research team to use this method to verify the major citation database systems <a href="https://www.scopus.com" target="_blank" rel="noopener">Scopus</a> and <a href="https://clarivate.com/products/web-of-science/" target="_blank" rel="noopener">Web of Science</a>, revealing robustness problems in citation counts and journal impact factors computed by these systems. A paper titled "Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors" by Zhi Quan Zhou, T.H. Tse, and Matt Witheridge has been published in IEEE Transactions on Software Engineering, the top journal in the field (URL for download: <a href="https://doi.org/10.1109/TSE.2019.2915065" target="_blank" rel="noopener">https://doi.org/10.1109/TSE.2019.2915065</a>).The high impact of the results lies in the finding that the presence of simple hyphens in a paper title adversely affects the citation statistics. Thus, metrics based on citation counts, which are commonly used for staff evaluations in universities, are not as robust as widely believed. This will be of interest not only to professors seeking tenure or promotion, but also to the senior management such as presidents, deans, and heads. The surprising results are applicable to all faculties in any university. Zhou et al.'s findings have been reported by international news media, including <a href="https://documents.uow.edu.au/~zhiquan/TheHyphenMan.pdf" target="_blank" rel="noopener">The Australian</a>, <a href="https://www.psychologytoday.com/intl/blog/the-athletes-way/201906/one-small-hyphen-in-title-could-diminish-academic-success" target="_blank" rel="noopener">Psychology Today</a><a href="https://www.sciencealert.com/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals" target="_blank" rel="noopener"></a>, <a href="https://cacm.acm.org/news/237400-a-single-punctuation-mark-has-been-skewing-our-entire-system-of-scientific-ranking/fulltext" target="_blank" rel="noopener">Communications of the ACM</a>, and <a href="https://www.eurekalert.org/" target="_blank" rel="noopener">EurekAlert!</a>, among others. On the Internet, the implications of Zhou et al.'s findings have been discussed from diverse perspectives, some of which extend beyond the conventional computing and citation domains. For example, see "<a href="https://www.slashgear.com/brave-new-world-predictions-are-coming-true-with-hyphens-31578773/" target="_blank" rel="noopener">Brave New World predictions are coming true with hyphens</a>" written by <a href="https://www.slashgear.com/author/chrisburns/" target="_blank" rel="noopener">Chris Burns</a> on <a href="https://www.slashgear.com/brave-new-world-predictions-are-coming-true-with-hyphens-31578773/" target="_blank" rel="noopener">SlashGear</a>. Inquiries have been made from various communities including, among others, the community of patients with <a href="https://www.3cr.org.au/chronicallychilled/episode-201901021800/asher-wolf-ehlers-danlos-syndrome" target="_blank" rel="noopener">Ehlers-Danlos Syndrome</a> (a <a href="https://en.wikipedia.org/wiki/List_of_syndromes" target="_blank" rel="noopener">condition</a> that has a hyphenated name and is known to be under-reported and under-researched). On 5 June 2019, the <a href="http://www.webofsciencegroup.com/" target="_blank" rel="noopener">Web of Science Group</a> posted a <a href="https://clarivate.com/webofsciencegroup/download/36464/" target="_blank" rel="noopener">response</a> to Zhou et al.'s paper on the <a href="https://clarivate.com/blog/science-research-connect/web-of-science-group-response-to-metamorphic-robustness-testing-exposing-hidden-defects-in-citation-statistics-and-journal-impact-factors/" target="_blank" rel="noopener">Clarivate Analytics website</a>. Feedback from the authors was reported by <a href="https://www.sciencealert.com/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals" target="_blank" rel="noopener">ScienceAlert</a>. On 7 August 2019, Zhou et al.'s work was reported by <a href="https://go.nature.com/2TdWoM6" target="_blank" rel="noopener">Nature</a> (https://go.nature.com/2TdWoM6).

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Detected and reported fatal bugs in self-driving cars eight days before the Uber crash in March 2018
Summary
On March 10, 2018, Zhou and his MPhil student Liqun Sun detected a critical fault in the real-world self-driving system <a href="http://apollo.auto" target="_blank" rel="noopener">Baidu Apollo</a>, and they reported this finding to the Apollo team on the same day (this was 8 days before <a href="https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg" target="_blank" rel="noopener">a fatal accident of a Uber's self-driving vehicle in Tempe, Arizona</a>). The fault caused the self-driving car's LiDAR perception system to ignore pedestrians or other obstacles on the roadway. The Apollo team confirmed the fault on March 19, 2018 (24 hours after <a href="https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg" target="_blank" rel="noopener">the Uber accident</a>, where the vehicle involved used exactly the same type of LiDAR sensor as Apollo). This study was reported in an article titled "Metamorphic Testing of Driverless Cars" by Zhou and Sun, published in <a href="https://cacm.acm.org/" target="_blank" rel="noopener">Communications of the ACM</a>, ACM's flagship magazine (<a href="https://doi.org/10.1145/3241979" target="_blank" rel="noopener">original paper</a>; <a href="https://doi.org/10.1145/3332409" target="_blank" rel="noopener">follow-up letter</a>). Zhou covered this story in his <a href="http://doi.org/10.5281/zenodo.2636979" target="_blank" rel="noopener">keynote speech</a> at the <a href="http://www.ieeeaitests.com/html/speakers.html" target="_blank" rel="noopener">IEEE International Conference on Artificial Intelligence Testing</a> (AITest 2019) held in San Francisco East Bay, California.

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators

Available as Research Supervisor

Advisees

Graduate Advising Relationship

Degree	Research Title	Advisee
Doctor of Philosophy	Un Similarity-Based Software Text case Prioritisation	Liu, Chen
Doctor of Philosophy	Metamorphic Testing applied to citation systems and other border applications	Witheridge, Matt
Doctor of Philosophy (Integrated)	A New Approach to the Detection of Software Security Vulnerabilities	Zhu, Junting
Doctor of Philosophy	Testing Navigation Systems	Brown, Josh
Doctor of Philosophy	A Machine Learning Based Approach for SoftwareTesting.	Wu, Sarah
Doctor of Philosophy	Software testing of web and cloud based applications	Han, Jiacheng
Doctor of Philosophy	A case for including metamorphic relations (MRs) into the testing standard for Advanced Driving Assist System (ADAS) and autonomous driving	Iqbal, Muhammad
Master of Philosophy - EIS	A Case Study of Web Security on Social Media Advertising	Liu, Yelin
Doctor of Philosophy (Integrated)	On Antomated Testing of Web Applications	Selay, Elmin

Education And Training

Ph.D. in Software Engineering, University of Hong Kong, Department of Computer Science 1998 - 2003

Awards And Honors

Researcher of the Year Gold Disruptor Award, conferred by Australian Computer Society, 2008
Virtual Earth Award, conferred by Microsoft Research, Redmond, USA, 2007

Keywords

- software engineering
- software testing
- test oracles
- autonomous driving
- cybersecurity

Orcid Id

http://orcid.org/0000-0003-1769-9367 (confirmed)

Scopus Id

56566917900

Top Publications

Year	Title
2019	Metamorphic Testing of Driverless Cars Published in Communications of the ACM
2019	Metamorphic Relations for Enhancing System Understanding and Use Published in IEEE Transactions on Software Engineering
2019	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published in IEEE Transactions on Software Engineering
2016	Metamorphic testing for software quality assessment: a study of search engines Published in IEEE Transactions on Software Engineering
2011	Semi-proving: an integrated method for program proving, testing, and debugging Published in IEEE Transactions on Software Engineering
2018	Metamorphic Testing: A Review of Challenges and Opportunities Published in ACM Computing Surveys
2012	Automated functional testing of online search services Published in Software Testing, Verification and Reliability
2015	A revisit of three studies related to random testing Published in Science China Information Sciences
2018	A cost-effective software testing strategy employing online feedback information Published in Information Sciences
2016	Metamorphic testing for cybersecurity Published in Computer
2019	A Case Against Mission-Critical Applications of Machine Learning Published in Communications of the ACM

Research Areas

Research Overview

Software engineering, software testing and debugging, verification and validation, user experience, citation analysis.

Selected Publications

Journal Article

Year	Title
2020	A workload-based nonlinear approach for predicting available computing resources Published In Journal of Systems Engineering and Electronics Volume 31 Pages 224 - 230 Altmetric
2020	Beating Random Test Case Prioritization Published In IEEE Transactions on Reliability Volume Online First Pages 1 - 22 ISSN 0018-9529 Altmetric
2019	A Case Against Mission-Critical Applications of Machine Learning Published In Communications of the ACM Volume 62 Page p. 9 ISSN 0001-0782 Altmetric
2019	A feedback control approach for preventing system resource exhaustion caused by software aging Published In Journal of Internet Technology Volume 20 Pages 1513 - 1522 ISSN 1607-9264 Altmetric
2019	Metamorphic Relations for Enhancing System Understanding and Use Published In IEEE Transactions on Software Engineering Volume Online First Pages 1 - 40 ISSN 0098-5589 Altmetric
2019	Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors Published In IEEE Transactions on Software Engineering Volume Online First Pages 1 - 22 ISSN 0098-5589 Altmetric
2019	Metamorphic Testing of Driverless Cars Published In Communications of the ACM Volume 62 Pages 61 - 67 ISSN 0001-0782 Altmetric
2019	Metamorphic Testing of Mapping Software Published In Lecture Notes in Business Information Processing Volume 347 Pages 1 - 20 Altmetric
2019	Metamorphic Testing: Testing the Untestable Published In IEEE Software Volume Online First Pages 1 - 12 ISSN 0740-7459 Altmetric
2018	A cost-effective software testing strategy employing online feedback information Published In Information Sciences Volume 422 Pages 318 - 335 ISSN 0020-0255 Altmetric
2018	Adaptive Random Testing in Detecting Layout Faults of Web Applications Published In International Journal of Software Engineering and Knowledge Engineering Volume 28 Pages 1399 - 1428 ISSN 0218-1940 Altmetric
2018	Introduction to the Special Issue on Test Oracles Published In Journal of Systems and Software Volume 136 Page p. 187 ISSN 0164-1212 Altmetric
2018	Metamorphic Testing: A Review of Challenges and Opportunities Published In ACM Computing Surveys Volume 51 Pages 4:1 - 4:27 ISSN 0360-0300 Altmetric
2016	High resolution SOM approach to improving anomaly detection in intrusion detection systems Published In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) Volume 9992 Pages 191 - 199 Altmetric
2016	Metamorphic testing for cybersecurity Published In Computer Volume 49 Pages 48 - 55 ISSN 0018-9162 Altmetric
2016	Metamorphic testing for software quality assessment: a study of search engines Published In IEEE Transactions on Software Engineering Volume 42 Pages 260 - 280 ISSN 0098-5589 Altmetric
2015	A revisit of three studies related to random testing Published In Science China Information Sciences Volume 58 Pages 052104:1 - 052104:9 ISSN 1674-733X Altmetric
2015	Using neural networks to forecast available system resources: an approach and empirical investigation Published In International Journal of Software Engineering and Knowledge Engineering Volume 25 Pages 781 - 802 ISSN 0218-1940 Altmetric
2012	Automated functional testing of online search services Published In Software Testing, Verification and Reliability Volume 22 Pages 221 - 243 ISSN 0960-0833 Altmetric
2012	Design and performance of a minimal real-time operating system in a safe language: Experience with java on the sun SPOT Published In Journal of Software Volume 7 Pages 1835 - 1844 ISSN 1796-217X Altmetric
2011	On adaptive random testing through iterative partitioning Published In Journal of Information Science and Engineering Volume 27 Pages 1449 - 1472 ISSN 1016-2364
2011	Semi-proving: an integrated method for program proving, testing, and debugging Published In IEEE Transactions on Software Engineering Volume 37 Pages 109 - 125 ISSN 0098-5589 Altmetric
2010	Automated functional testing of online search services Published In Software Testing, Verification and Reliability Volume Aug Pages 1 - 29 ISSN 0960-0833 Altmetric
2010	Metamorphic testing of decision support systems: A case study Published In IET Software Volume 4 Pages 294 - 301 ISSN 1751-8806 Altmetric
2007	On Favourable Conditions for Adaptive Random Testing Published In International Journal of Software Engineering and Knowledge Engineering Volume 17 Pages 805 - 825 ISSN 0218-1940 Altmetric

Conference Paper

Year	Title
2019	Metamorphic Relations for Data Validation: A Case Study of Translated Text Messages Published In IEEE/ACM 4th International Workshop on Metamorphic Testing (MET 2019) Pages 70 - 75 Altmetric
2019	Metamorphic exploration of an unsupervised clustering program Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019 Pages 48 - 54 Altmetric
2019	Metamorphic relations for detection of performance anomalies Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019 Pages 63 - 69 Altmetric
2019	The impact of a dot: Case studies of a noise metamorphic relation pattern Published In Proceedings - 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing, MET 2019 Pages 17 - 23 Altmetric
2018	A Monte Carlo Method for Metamorphic Testing of Machine Translation Services Published In 2018 ACM/IEEE International Workshop on Metamorphic Testing Pages 38 - 45 Altmetric
2018	Enhancing Supervised Classifications with Metamorphic Relations Published In Proceedings - International Conference on Software Engineering Pages 46 - 53 Altmetric
2018	Metamorphic Testing for Adobe Analytics Data Collection JavaScript Library Published In 2018 ACM/IEEE International Conference on Software Engineering Pages 34 - 37 Altmetric
2018	Metamorphic testing 20 years later: A hands-on introduction Published In International Conference on Software Engineering Pages 538 - 539 Altmetric
2018	Metamorphic testing for machine translations: MT4MT Published In Proceedings - 25th Australasian Software Engineering Conference, ASWEC 2018 Pages 96 - 100 Altmetric
2018	Metamorphic testing of navigation software: A pilot study with Google Maps Published In Proceedings of the 51st Annual Hawaii International Conference on System Sciences (HICSS-51) Pages 5687 - 5696
2017	Metamorphic Testing for Adobe Data Analytics Software Published In 2nd International Workshop on Metamorphic Testing Pages 21 - 27 Altmetric
2017	Metamorphic Testing: Beyond Testing Numerical Computations Published In IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET) Pages 1 - 1 Altmetric
2016	Metamorphic testing: A new student engagement approach for a new software testing paradigm Published In Proceedings of 2016 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE) Pages 218 - 228 Altmetric
2014	Adaptive random testing for image comparison in regression web testing Published In 2014 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2014 Pages 1 - 7 Altmetric
2014	Testing model transformation programs using metamorphic testing Published In 26th International Conference on Software Engineering & Knowledge Engineering (SEKE 2014) Pages 94 - 99
2013	On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions Published In 13th International Conference on Quality Software Pages 153 - 162 Altmetric
2013	Similarity-based search for model checking: a pilot study with java pathfinder Published In 13th International Conference on Quality Software Pages 238 - 244 Altmetric
2012	Function substitution: Towards constraint solving in software testing Published In QSIC 2012: 12th International Conference on Quality Software Pages 31 - 40 Altmetric
2012	Metamorphic testing: Applications and integration with other methods: Tutorial synopsis Published In QSIC 2012: 12th International Conference on Quality Software Pages 285 - 288 Altmetric
2012	On the fault-detection capabilities of adaptive random test case prioritization: Case studies with large test suites Published In Proceedings of the 45th Hawaii International Conference on System Sciences Pages 5584 - 5593 Altmetric
2010	Using coverage information to guide test case selection in Adaptive Random Testing Published In IEEE International Computer Software and Applications Conference Workshops Pages 208 - 213 Altmetric
2009	Improving software testing cost-effectiveness through dynamic partitioning Published In International Conference on Quality Software Pages 249 - 258 Altmetric
2008	CSCI: a LEAP into the future Published In International Conference on Information and Emerging Technologies Pages 1 - 14
2007	Automated software testing and analysis: techniques, practices and tools Published In Hawaii International Conference on System Sciences Pages 260 - 260 Altmetric
2007	Developing real-time systems in Java on Macintosh Published In Apple University Consortium Academic and Developers Conference Pages 11.1 - 11.9
2007	Enhanced random testing for programs with high dimensional input domains Published In International Conference on Software Engineering Pages 135 - 140
2007	Software development of embedded systems on Macintosh Published In Apple University Consortium Academic and Developers Conference Pages 10.1 - 10.9

Investigator On

Year	Title
2007	On testing non-testable information retrieval systems with geographic components on the web Awarded by: Microsoft Research Funding Scheme: Virtual Earth Academic Research Collaboration Award
	Design and Deployment of Practical Anonymous Access Systems Awarded by: Australian Research Council (ARC) Funding Scheme: Linkage Projects
	EATandDREAM: Effective automatic Testing and Debugging for REAl-life eMbedded wireless communications software Awarded by: Australian Research Council (ARC) Funding Scheme: Linkage Projects

Impact Story

International rankings
Summary
An independent study conducted by American researchers identified Zhou as one of the top five researchers internationally in the important research field of test oracles [1]. Another independent study conducted by European researchers identified Zhou as one of the top four researchers internationally in metamorphic testing [2]. References: [1] R. A. P. Oliveira, U. Kanewala, and P. A. Nardi. "Automated test oracles: State of the art, taxonomies, and trends," In Advances in Computers, volume 95, pp. 113-199. Elsevier, 2014. Available online: <a href="http://iii.library.uow.edu.au/record=b2303118" target="_blank" rel="noopener">http://iii.library.uow.edu.au/record=b2303118</a> [2] S. Segura, G. Fraser, A. Sanchez, and A. Ruiz-Cortes, "A survey on metamorphic testing", IEEE Transactions on Software Engineering, 42(9), pp. 805-824, 2016. Available online: <a href="http://dx.doi.org/10.1109/TSE.2016.2532875" target="_blank" rel="noopener">http://dx.doi.org/10.1109/TSE.2016.2532875</a>

Detail View

Detail View Summary View
Detail of Impact
The following URL links to an invited webinar that introduces metamorphic testing, produced by the ACM: <a href="https://event.on24.com/wcc/r/1451736/8B5B5925E82FC9807CF83C84834A6F3D" target="_blank" rel="noopener">https://event.on24.com/wcc/r/1451736/8B5B5925E82FC9807CF83C84834A6F3D </a>
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Collaboration with industry
Summary
Zhou has been working with the IT industry on a series of software testing projects. Those industry partners include Adobe Inc., Microsoft Corporation, Partech Systems, and the Baidu Apollo autonomous driving team, among others. An independent review of Zhou's joint work with Adobe Inc. can be found <a href="https://semiwiki.com/eda/284509-innovation-in-verification-april-2020/" title="Innovation in Verification April 2020" target="_blank" rel="noopener">here</a>, where Paul Cunningham (GM of the Verification Group at Cadence) commented that "I like this. I see it as a way to do statistical anomaly-based QA. You compare a lot of runs, looking at distributions to spot bugs. I see a lot of applications: anything performance-related, heuristic-based, machine-learning-based will be naturally statistical. Distribution analyses can then reveal more complex issues than pass/fail analyses. MT gives us tools to find those kinds of problem." And Jim Hogan commented that "This is the first of the papers we've looked at in this series which to me is more than just a feature. This paper would definitely be worth putting money behind, trying to get to production. It looks like a product, perhaps a new class of verification tool. It might even work as a startup." Interested readers may refer to the full discussion <a href="https://semiwiki.com/eda/284509-innovation-in-verification-april-2020/" target="_blank" rel="noopener">Innovation in Verification April 2020</a>.

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Exposing Hidden Defects in Citation Statistics and Journal Impact Factors
Summary
Zhou initiated a "metamorphic robustness testing" method and led his research team to use this method to verify the major citation database systems <a href="https://www.scopus.com" target="_blank" rel="noopener">Scopus</a> and <a href="https://clarivate.com/products/web-of-science/" target="_blank" rel="noopener">Web of Science</a>, revealing robustness problems in citation counts and journal impact factors computed by these systems. A paper titled "Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors" by Zhi Quan Zhou, T.H. Tse, and Matt Witheridge has been published in IEEE Transactions on Software Engineering, the top journal in the field (URL for download: <a href="https://doi.org/10.1109/TSE.2019.2915065" target="_blank" rel="noopener">https://doi.org/10.1109/TSE.2019.2915065</a>).The high impact of the results lies in the finding that the presence of simple hyphens in a paper title adversely affects the citation statistics. Thus, metrics based on citation counts, which are commonly used for staff evaluations in universities, are not as robust as widely believed. This will be of interest not only to professors seeking tenure or promotion, but also to the senior management such as presidents, deans, and heads. The surprising results are applicable to all faculties in any university. Zhou et al.'s findings have been reported by international news media, including <a href="https://documents.uow.edu.au/~zhiquan/TheHyphenMan.pdf" target="_blank" rel="noopener">The Australian</a>, <a href="https://www.psychologytoday.com/intl/blog/the-athletes-way/201906/one-small-hyphen-in-title-could-diminish-academic-success" target="_blank" rel="noopener">Psychology Today</a><a href="https://www.sciencealert.com/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals" target="_blank" rel="noopener"></a>, <a href="https://cacm.acm.org/news/237400-a-single-punctuation-mark-has-been-skewing-our-entire-system-of-scientific-ranking/fulltext" target="_blank" rel="noopener">Communications of the ACM</a>, and <a href="https://www.eurekalert.org/" target="_blank" rel="noopener">EurekAlert!</a>, among others. On the Internet, the implications of Zhou et al.'s findings have been discussed from diverse perspectives, some of which extend beyond the conventional computing and citation domains. For example, see "<a href="https://www.slashgear.com/brave-new-world-predictions-are-coming-true-with-hyphens-31578773/" target="_blank" rel="noopener">Brave New World predictions are coming true with hyphens</a>" written by <a href="https://www.slashgear.com/author/chrisburns/" target="_blank" rel="noopener">Chris Burns</a> on <a href="https://www.slashgear.com/brave-new-world-predictions-are-coming-true-with-hyphens-31578773/" target="_blank" rel="noopener">SlashGear</a>. Inquiries have been made from various communities including, among others, the community of patients with <a href="https://www.3cr.org.au/chronicallychilled/episode-201901021800/asher-wolf-ehlers-danlos-syndrome" target="_blank" rel="noopener">Ehlers-Danlos Syndrome</a> (a <a href="https://en.wikipedia.org/wiki/List_of_syndromes" target="_blank" rel="noopener">condition</a> that has a hyphenated name and is known to be under-reported and under-researched). On 5 June 2019, the <a href="http://www.webofsciencegroup.com/" target="_blank" rel="noopener">Web of Science Group</a> posted a <a href="https://clarivate.com/webofsciencegroup/download/36464/" target="_blank" rel="noopener">response</a> to Zhou et al.'s paper on the <a href="https://clarivate.com/blog/science-research-connect/web-of-science-group-response-to-metamorphic-robustness-testing-exposing-hidden-defects-in-citation-statistics-and-journal-impact-factors/" target="_blank" rel="noopener">Clarivate Analytics website</a>. Feedback from the authors was reported by <a href="https://www.sciencealert.com/hyphens-break-our-entire-system-of-scientific-ranking-new-analysis-reveals" target="_blank" rel="noopener">ScienceAlert</a>. On 7 August 2019, Zhou et al.'s work was reported by <a href="https://go.nature.com/2TdWoM6" target="_blank" rel="noopener">Nature</a> (https://go.nature.com/2TdWoM6).

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators
Detected and reported fatal bugs in self-driving cars eight days before the Uber crash in March 2018
Summary
On March 10, 2018, Zhou and his MPhil student Liqun Sun detected a critical fault in the real-world self-driving system <a href="http://apollo.auto" target="_blank" rel="noopener">Baidu Apollo</a>, and they reported this finding to the Apollo team on the same day (this was 8 days before <a href="https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg" target="_blank" rel="noopener">a fatal accident of a Uber's self-driving vehicle in Tempe, Arizona</a>). The fault caused the self-driving car's LiDAR perception system to ignore pedestrians or other obstacles on the roadway. The Apollo team confirmed the fault on March 19, 2018 (24 hours after <a href="https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg" target="_blank" rel="noopener">the Uber accident</a>, where the vehicle involved used exactly the same type of LiDAR sensor as Apollo). This study was reported in an article titled "Metamorphic Testing of Driverless Cars" by Zhou and Sun, published in <a href="https://cacm.acm.org/" target="_blank" rel="noopener">Communications of the ACM</a>, ACM's flagship magazine (<a href="https://doi.org/10.1145/3241979" target="_blank" rel="noopener">original paper</a>; <a href="https://doi.org/10.1145/3332409" target="_blank" rel="noopener">follow-up letter</a>). Zhou covered this story in his <a href="http://doi.org/10.5281/zenodo.2636979" target="_blank" rel="noopener">keynote speech</a> at the <a href="http://www.ieeeaitests.com/html/speakers.html" target="_blank" rel="noopener">IEEE International Conference on Artificial Intelligence Testing</a> (AITest 2019) held in San Francisco East Bay, California.

Detail View

Detail View Summary View
Field of Research
Countries / Regions
Beneficiaries

Beneficiary Quantification Description
External Partners
Evidence of Impact

Evidence Description
UOW Centres / Programs
UOW Researchers / Collaborators

Advisees

Graduate Advising Relationship

Degree	Research Title	Advisee
Doctor of Philosophy	Un Similarity-Based Software Text case Prioritisation	Liu, Chen
Doctor of Philosophy	Metamorphic Testing applied to citation systems and other border applications	Witheridge, Matt
Doctor of Philosophy (Integrated)	A New Approach to the Detection of Software Security Vulnerabilities	Zhu, Junting
Doctor of Philosophy	Testing Navigation Systems	Brown, Josh
Doctor of Philosophy	A Machine Learning Based Approach for SoftwareTesting.	Wu, Sarah
Doctor of Philosophy	Software testing of web and cloud based applications	Han, Jiacheng
Doctor of Philosophy	A case for including metamorphic relations (MRs) into the testing standard for Advanced Driving Assist System (ADAS) and autonomous driving	Iqbal, Muhammad
Master of Philosophy - EIS	A Case Study of Web Security on Social Media Advertising	Liu, Yelin
Doctor of Philosophy (Integrated)	On Antomated Testing of Web Applications	Selay, Elmin

Education And Training

Ph.D. in Software Engineering, University of Hong Kong, Department of Computer Science 1998 - 2003

Awards And Honors

Researcher of the Year Gold Disruptor Award, conferred by Australian Computer Society, 2008
Virtual Earth Award, conferred by Microsoft Research, Redmond, USA, 2007

Keywords

- software engineering
- software testing
- test oracles
- autonomous driving
- cybersecurity

Orcid Id

http://orcid.org/0000-0003-1769-9367 (confirmed)

Scopus Id

56566917900

Overview

Overview

Top Publications

Research Areas

Research Overview

Available as Research Supervisor

Publications

Selected Publications

Journal Article

Conference Paper

Funding

Investigator On

Impact

Impact Story

International rankings

Collaboration with industry

Exposing Hidden Defects in Citation Statistics and Journal Impact Factors

Detected and reported fatal bugs in self-driving cars eight days before the Uber crash in March 2018

Supervision

Available as Research Supervisor

Advisees

Graduate Advising Relationship

Background

Education And Training

Awards And Honors

Keywords

Identity

Orcid Id

Scopus Id

Overview

Top Publications

Research Areas

Research Overview

Publications

Selected Publications

Journal Article

Conference Paper

Funding

Investigator On

Impact

Impact Story

International rankings

Collaboration with industry

Exposing Hidden Defects in Citation Statistics and Journal Impact Factors

Detected and reported fatal bugs in self-driving cars eight days before the Uber crash in March 2018

Supervision

Advisees

Graduate Advising Relationship

Background

Education And Training

Awards And Honors

Keywords

Identity

Orcid Id

Scopus Id

Visualisations

Research Areas