Mutant and Minitest

Last time, I tried to find an easy way of running mutation testing on a rails app that’s tested by minitest. I did not find an easy way, so today I’m going to try to do it the hard way: use an old PR on mutant as a jumping off point to try to make a version of mutant that hooks into minitest.

First, I need to get ready to get to work:

  1. I forked mutant
  2. I cloned the fork locally: git clone git@github.com:willmurphyscode/mutant.git ; cd mutant
  3. I checkout the branch to work on git checkout feature/minitest-integration
  4. I add this to my test app’s gemfile: gem 'mutant', path: '/Projects/ruby/mutant'
  5. bundle install in the test project.

Now, I can at least try to run mutation testing against my fork:

 RAILS_ENV=test bundle exec mutant -r ./config/environment --use minitest Lightbulb

The Ligthbulb at the end is the name of a rails model. I don’t yet know how to tell mutant to test everything, but this command should start working once I’m able to run mutant on my minitests. Unfortunately, this blows up at a require 'test_helper' in any test that I run. For now, I’m going to make the require line more explicit so that I can get to the next error. I change require 'test_helper' to:

require Rails.root.join('test','test_helper')

__FILE__ is a variable set by the Ruby interpreter that holds the path to the currently executing file. All this change does is tell Ruby where the file test_helper.rb is, even if the file requiring it is invoked from an unexpected path, as in the case when it is invoked by mutant and not by rails or rake.

After I do this, mutant runs successfully, and claims that I killed zero out of zero mutants with my test suite. That might be because this file is so empty that there’s really only the results of a few rails g ... commands in it. I’m going to add a method and a lousy test:

# app/models/lightbulb.rb
  def sum(a, b)
    a + b
  end

# test/models/lightbulb_test.rb
  it 'adds numbers' do
    a = 1
    b = 2
    sum = lightbulb.sum(a, b)
    assert sum.is_a? Fixnum
  end

This is a pretty bad test. It tests a method called sum just by asserting that if you pass two integers in, you’ll get an integer back. My hope is that mutant will emit something that does subtraction, or just returns one of the arguments, and notice that it doesn’t call the test to fail. This definitely causes mutant to try to do more. Unfortuantely, it leads to an error. Out of the box, one big difference between minitest and rspec is that rspec tests know what they cover, and minitest tests do not.

Mutation testing can be slow. It runs each test once on each mutant it thinks the test should care about, and it has to generate all the mutants. If it has to run every test on every mutant, it would quickly bog down. Therefore, mutant invokes cover_expression on each test class to check what is covered by that test. This call allows mutant to subset tests efficiently and avoid the O(m * n) work of running every test against every mutant.

Based on examples in mutant, I added the following to my test/test_helper.rb file:

  def self.cover(expression)
    @expression = expression
  end

  def self.cover_expression
    @expression or fail "Cover expression for #{self} is not specified"
  end

And then added, for example, cover 'Foo#' to the top of the class FooTest, for example. Now mutant knows which tests to run against which mutants. I had to add cover 'something' to every test class, because mutant will raise if it can’t tell which code a given test covers.

With the bad tests above, I got the following output:

evil:Lightbulb#sum:/Projects/ruby/testing_stuff/app/models/lightbulb.rb:3:d829c
@@ -1,4 +1,4 @@
 def sum(a, b)
-  a + b
+  a
 end
-----------------------
evil:Lightbulb#sum:/Projects/ruby/testing_stuff/app/models/lightbulb.rb:3:7016f
@@ -1,4 +1,4 @@
 def sum(a, b)
-  a + b
+  b
 end
-----------------------
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [Lightbulb]>
Integration:     Mutant::Integration::Minitest
Jobs:            1
Includes:        []
Requires:        ["./config/environment", "minitest/autorun"]
Subjects:        1
Mutations:       16
Results:         16
Kills:           14
Alive:           2
Runtime:         3.03s
Killtime:        0.66s
Overhead:        358.66%
Mutations/s:     5.28
Coverage:        87.50%

The lines above beginning with evil show mutants that pass all my tests. That is, they represent bugs that I could accidentally introduce and none of my tests would turn red.

Next I will add a real test for sum:

  def test_it_adds_numbers
    a = 1
    b = 2
    expected = a + b
    actual = lightbulb.sum(a, b)
    assert_equal expected, actual
  end

(I’m not worried right now about why sum is an instance variable on Lightbulb.) I run mutant again:

Matcher:         #<Mutant::Matcher::Config match_expressions: [Lightbulb]>
Integration:     Mutant::Integration::Minitest
Jobs:            1
Includes:        []
Requires:        ["./config/environment"]
Subjects:        1
Mutations:       16
Results:         16
Kills:           16
Alive:           0
Runtime:         3.03s
Killtime:        0.64s
Overhead:        370.84%
Mutations/s:     5.29
Coverage:        100.00%

Now my test suite kills 16 out of 16 mutants on the Lightbulb class. That means that mutant was unable to emit a mutant that passed all my tests. This is real progress, and I’m pretty excited.

One reason this project got off to such a good start is that most of the real work had been done on an old mutant PR, and also the creator of mutant was very helpful on the mutation testing slack channel. Also, the very clear discussion on the PR linked above was a big help.

One big disadvantage right now is that describe blocks in minitest don’t know about the cover 'Foo#' call in the class, so I have cover 'Foo#' or whatever in every describe block. This is fine for a small app, but it wouldn’t work for a bigger app; I will need to do something to propagate cover expressions to describe blocks on minitest classes.

Still, we have a pretty excellent proof of concept here. I know that mutant can execute minitest tests just fine. Next time, I’ll try to eliminate the need to have cover 'Foo#' everywhere. Also, mutant uses some parallelization of test runs, and this may not work with tests that affect external state. Next time, I’ll take a look at these two issues.

Till next time, happy learning!

-Will

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s