Watch for Mutants

I have a project at work that uses Minitest, but I want to use mutant to test it. Mutant is a gem that does mutation testing. Mutation testing is an important complement to regular automated testing. Regular automated testing asserts that correct code passes your test suite; mutation testing asserts that incorrect code fails your test suite. In this series, I’m going to try to find the best way to mutation test a Rails app that’s covered mostly be minitest.

At a high level, mutation testing works like this:

  1. Parse the code under test.
  2. Generate a set of mutants. Mutants are altered versions of the code under test. For example, one mutant might have a negation before a boolean operator, so that == becomes !=.
  3. Run the test suite on each mutant.
  4. Assert that every mutant causes the failure of at least one test.
  5. Report to the developer any mutants that survive.

These steps help developers expose gaps in their test coverage. For example, if I have method that’s doing some math, and someone changes + to - in two of the lines, the math is almost certainly now incorrect, so one of my tests really ought to turn red.

As I mentioned, mutant is a gem that performs mutation testing on Ruby code. I believe it is the best known and most widely used such gem. However, it is used with the testing gem rspec, and not with the testing gem minitest. I want to learn how difficult it would be to run mutation testing against a project tested with minitest, either by finding a suitable substitute for mutant or, failing that, by forking mutant and making my version work with minitest. There has been a GitHub pull request to add minitest support open for more than a year, but there doesn’t seem to be any momentum behind merging it.

To look for an alternative to mutant, I will search rubygems.org for other mutation testing gems and take any that look full-featured and well-maintained for a test drive. For my purposes, well-maintained means that the GitHub repo has at least a few commits in the past year, and full-featured means that the mutation testing gem executes the 5 steps above, and looks like it can be called in a build pipeline and log sensible output. Of course, anything that doens’t work with minitest can’t be in the running; if I were willing to abandon minitest, I would just use mutant.

The first candidate I looked at was heckle. Unfortunately, heckle depends on ParseTree which has been EOL’d because it relies on Ruby 1.8 internals.

As far as I can tell, to get from steps 1 to 2, a mutation testing library needs to emit an abstract syntax tree, then edit the emitted tree, and then execute the edited tree. This means that most frameworks for mutation testing rely on something that parses ruby without executing it. However, the ruby interpreter has changed a lot over the years, so such libraries are difficult to maintain. The story with heckle seems common – some people build an open source mutation test gem for Ruby, try to get a little momentum going, and then Ruby changes enough that they can’t catch up. For example, the gem mutiny seems to have basically the same problem; it mentions a parser library that complains about Ruby changes, and then generates a warning about any Ruby version after 2.2.x whenever it is run. Another gem mention was mutest, but like mutant it will only work with rspec.

The other gems returned when I searched for “mutation testing” on rubygems are not maintained, or about genetic analysis, or something. It looks like a search miss.

I’ve learned a little bit about the difficulties of writing a gem like mutant today, but I haven’t found anything I can use. It looks to me like I’ve done due diligence to find code I could use instead of writing code, and I’ll have to start writing code pretty soon. I’m a bit intimidated that mutant has had minitest support requested, but hasn’t implemented it yet.

Next time, I’ll fork mutant and try to understand how it hooks into rspec, maybe even modifying the branch from the PR above. My hope is that my fork can re-use all the existing code for generating mutants, and I’ll only need to change code that’s responsible for running the test suite against the mutants, and that the PR will have most of that taken care of. I will also look through some other forks of mutant and see whether anyone else has attempted or succeeded in this.

Till next time, happy learning!

-Will

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s