Who Tests Our Tests? An Overview of Mutation Analysis, its Caveats and Pitfalls

Rahul Gopinath

Speaker:  Rahul Gopinath, Ph.D. Candidate, Oregon State University

Date:  Wednesday, February 22
Time:  11am
Location:  Room 1202, CSE Building

Abstract:  A key concern in software engineering is determining the reliability of our software artifacts. Software engineers traditionally rely on extensive testing to ensure that programs are fault free. However, tests are usually also written by human beings, and hence vulnerable to the similar problems as software applications. An inadequate test suite, or one that contains errors, can provide a false sense of security to the programmer or the consumer. Hence a major question in software testing is how to evaluate the quality of test suites.

Following the maxim, "you can’t find a bug in what you don't cover," practitioners often use code coverage as a measure of test suite quality. However, code coverage can not evaluate the quality of our oracles, and given that our oracles are often insufficient or wrong, code coverage, on its own is insufficient, and mutation testing, a stronger fault-based technique is often recommended. In this talk, I will present the basics of mutation testing, and look at why it is effective. Next, I will examine the limitations to mutation analysis, and how these can be mitigated. I will also review how insights from mutation testing can be used in fields such as evaluating type annotations, program repair and assessing the vulnerability of programs. 

Bio:  Rahul Gopinath is a Ph.D. candidate at Oregon State University, where he is advised by Dr. Carlos Jensen and Dr. Alex Groce. He received his MCS from Illinois Institute of Technology, Chicago. His area of research is software analysis and verification, with a particular emphasis on software testing. During his Ph.D. he focused on mutation analysis of software test suites, especially how to make mutation analysis a feasible option for a practicing software tester.  His work has resulted in both theoretical insights and practical improvements in mutation analysis. Before starting his Ph.D., he was a software engineer at Sun Microsystems, where he worked on the iPlanet line of web and proxy servers. He is a prolific contributor to open-source, with contributions towards the Apache HTTPD, Puppet, and the PIT mutation analysis framework. He is also a committer for NixOS, a functional package manager and a Linux distribution, and the author of MuCheck, a mutation analysis framework for Haskell.

Related Research Publications

Faculty host:  CSE Prof. Bill Griswold (wgg@cs.ucsd.edu)