State and Caching in Minitest

Last time, I learned that using let blocks in minitest cause minitest to define a method on the test class, and I hypothesized that this was to allow tests to share code while preventing them from sharing state. But the method that gets defined caches the results, so that the initializing block is only evaluated on the first call. I want to convince myself that test methods cannot corrupt this cache, or rather, can’t corrupt this cache without doing so on purpose.

For reference, here’s the minitest code that runs when we make a let block:

def let name, &block
  name = name.to_s
  pre, post = "let '#{name}' cannot ", ". Please use another name."
  methods = Minitest::Spec.instance_methods.map(&:to_s) - %w[subject]
  raise ArgumentError, "#{pre}begin with 'test'#{post}" if
    name =~ /\Atest/
  raise ArgumentError, "#{pre}override a method in Minitest::Spec#{post}" if
    methods.include? name

  define_method name do
    @_memoized ||= {}
    @_memoized.fetch(name) { |k| @_memoized[k] = instance_eval(&block) }
  end
end

Here’s how I understand the method: When we evaluate a let block, such as let(:spell_to_cast) { 'fireball' }, we validate the argument, raising if it would either hide an existing method on the class or define a test_ method. Then, we define a method that says, “There’s a cache of method results (@_memoized). If it’s nil, initialize it to an empty hash. Then, try to fetch from it by name, in this case, :spell_to_cast. If that’s nil, execute the block that was passed in, cache the result, and then return the result.

What I want to prove to myself today is that it’s not likely that I’ll accidentally get a reference to a member of @_memoized and mutate it.

To conduct this test I want an obviously mutable model:

class Lightswitch < ActiveRecord::Base
  attr_accessor :turned_on
  
  def turn_on
    self.turned_on = true
  end
end

From here my goal is simple: write believable-looking test code that gets a reference to an instance of Lightswitch in @_memoized, invokes turn_on on it, and then pollutes subsequent tests with the fact that a light switch cache is on. I suspect this is impossible, but I want to understand why it’s impossible. Magic makes me nervous, because magic is always an abstraction that might leak when I don’t want it to.

Here’s the basic setup:

describe 'mutability of @_memoized via lightswitches' do
  let(:switch_thats_off) do
    value = Lightswitch.new
    value.turned_on = false
    value
  end

  it 'turns on' do
    test_subject = switch_thats_off
    test_subject.turn_on
    assert test_subject.turned_on, 'switch should be on'
  end   
end

This is what I would consider a normal use. I want to understand why subsequent calls to switch_thats_off don’t return the mutated instance of Lightswitch from the @_memoized cache.

First, I want to understand how often minitest runs constructors on test classes, so I made this constructor override:

def initialize(*)
  super
  puts 'i was initialized again'
  these_methods = (self.methods - Object.new.methods)
    .select {|m| /test_/ =~ m } # `select` is like C#'s `where`
  puts "I have #{these_methods.length} test methods!"
end

This constructor results in the following output:

..........I was initialized again
I have 3 test methods!
.I was initialized again
I have 3 test methods!
.I was initialized again
I have 3 test methods!

This experiment makes it pretty clear to me that minitest runs the constructor every time it executes a test method. So why all the ceremony around let blocks? Why can’t I just say something like this:

def self.let name, &block
  # TODO: raise if name is invalid or hides another method
  value = block.call 
  define_method name do
    value
  end
end

I put this in my test class, along with some puts statements so that I was sure it was being called, tried to detect a change. I came up with the following (super annoying) snippet of test:

it 'turns on the switch' do
  switch_thats_off.turn_on
end    
it 'turns on the switch' do
  switch_thats_off.turn_on
end    
it 'turns on the switch' do
  switch_thats_off.turn_on
end    
it 'turns on the switch' do
  switch_thats_off.turn_on
end    
it 'turns on the switch' do
  switch_thats_off.turn_on
end

it 'should still start out off' do
  refute switch_thats_off.turned_on, 'should be off'
end

Minitest calls these it blocks in a random order, so in all probability one of the ‘turns the switch on’ methods will run before the ‘should still start out off’ method. It doesn’t. Hiding minitest’s let method causes the tests above to fail in most cases (except for the off chance that the last block gets run before any of the others).

The problem with my implementation is that the block passed to define_method closes over value, so every instance of the test class ends up with a reference to the same instance of Lightswitch, and the test runs interfere. The following method will almost work, but is also not what minitest chose to implement:

def self.let name, &block
  define_method name do
    block.call
  end
end

Can you spot the issue with this one? There are two issues:

First, there’s no caching at all. Test code won’t even be able to mutate this variable within a block. Every call to the method name, whatever it turns out to be called, will execute &block all over again, and return a new object. It won’t behave like a local variable at all; the caller won’t even be able to mutate it and see the change within its own call block.

Second, because there’s no caching at all, the test code won’t be very performant; we’ll have to re-execute &block every time we want the value.

In order to fix these issues, I settled on this issue, which aside from validation is the same as minitest’s:

def self.let name, &block
  define_method name do
    @_my_hash ||= {}
    @_my_hash.fetch(name) {|k| @_my_hash[k] = block.call}
  end
end

A quick aside: the fetch usage in the code snippet above was unfamiliar to me. Basically, it means, “try to get key name from the hash map, otherwise execute the block delimited by {} and return its value. The block in the {} in turn calls the block that was passed in and assigns the resulting value to the appropriate key in the hash.

This snippet, has, I think, the same semantics as a method that either initializes an instance variable or returns its current value if it’s already been initialized.

Summing Up

I started this article wondering how let blocks worked, and discovered two things:

1. minitest re-instantiates the entire test object every time it runs a test_ method or an it block.
2. let blocks work by defining an instance method on the test class that either initializes the value of some variable, caches, and returns it, or returns the cached value.

That answers my initial question, which was, “how is minitest sure that different test_ methods will not interfere with each other?” The answer is “by making a new test object every time a test_ method is run.”

The next thing I want to understand is how nesting describe blocks works; I hope to have a post about that in the near future.

Till next time, happy learning!

-Will