I have a project at work that uses Minitest, but I want to use
mutant to test it. Mutant is a gem that does mutation testing. Mutation testing is an important complement to regular automated testing. Regular automated testing asserts that correct code passes your test suite; mutation testing asserts that incorrect code fails your test suite. In this series, I’m going to try to find the best way to mutation test a Rails app that’s covered mostly be minitest.
At a high level, mutation testing works like this:
- Parse the code under test.
- Generate a set of mutants. Mutants are altered versions of the code under test. For example, one mutant might have a negation before a boolean operator, so that
- Run the test suite on each mutant.
- Assert that every mutant causes the failure of at least one test.
- Report to the developer any mutants that survive.
These steps help developers expose gaps in their test coverage. For example, if I have method that’s doing some math, and someone changes
- in two of the lines, the math is almost certainly now incorrect, so one of my tests really ought to turn red.
As I mentioned, mutant is a gem that performs mutation testing on Ruby code. I believe it is the best known and most widely used such gem. However, it is used with the testing gem
rspec, and not with the testing gem
minitest. I want to learn how difficult it would be to run mutation testing against a project tested with
minitest, either by finding a suitable substitute for
mutant or, failing that, by forking
mutant and making my version work with
minitest. There has been a GitHub pull request to add
minitest support open for more than a year, but there doesn’t seem to be any momentum behind merging it.
To look for an alternative to
mutant, I will search rubygems.org for other mutation testing gems and take any that look full-featured and well-maintained for a test drive. For my purposes, well-maintained means that the GitHub repo has at least a few commits in the past year, and full-featured means that the mutation testing gem executes the 5 steps above, and looks like it can be called in a build pipeline and log sensible output. Of course, anything that doens’t work with
minitest can’t be in the running; if I were willing to abandon
minitest, I would just use
As far as I can tell, to get from steps 1 to 2, a mutation testing library needs to emit an abstract syntax tree, then edit the emitted tree, and then execute the edited tree. This means that most frameworks for mutation testing rely on something that parses ruby without executing it. However, the ruby interpreter has changed a lot over the years, so such libraries are difficult to maintain. The story with
heckle seems common – some people build an open source mutation test gem for Ruby, try to get a little momentum going, and then Ruby changes enough that they can’t catch up. For example, the gem mutiny seems to have basically the same problem; it mentions a parser library that complains about Ruby changes, and then generates a warning about any Ruby version after 2.2.x whenever it is run. Another gem mention was
mutest, but like
mutant it will only work with
The other gems returned when I searched for “mutation testing” on rubygems are not maintained, or about genetic analysis, or something. It looks like a search miss.
I’ve learned a little bit about the difficulties of writing a gem like
mutant today, but I haven’t found anything I can use. It looks to me like I’ve done due diligence to find code I could use instead of writing code, and I’ll have to start writing code pretty soon. I’m a bit intimidated that
mutant has had
minitest support requested, but hasn’t implemented it yet.
Next time, I’ll fork
mutant and try to understand how it hooks into
rspec, maybe even modifying the branch from the PR above. My hope is that my fork can re-use all the existing code for generating mutants, and I’ll only need to change code that’s responsible for running the test suite against the mutants, and that the PR will have most of that taken care of. I will also look through some other forks of
mutant and see whether anyone else has attempted or succeeded in this.
Till next time, happy learning!