Last time, I learned that using let
blocks in minitest cause minitest to define a method on the test class, and I hypothesized that this was to allow tests to share code while preventing them from sharing state. But the method that gets defined caches the results, so that the initializing block is only evaluated on the first call. I want to convince myself that test methods cannot corrupt this cache, or rather, can’t corrupt this cache without doing so on purpose.
For reference, here’s the minitest code that runs when we make a let
block:
def let name, &block
name = name.to_s
pre, post = "let '#{name}' cannot ", ". Please use another name."
methods = Minitest::Spec.instance_methods.map(&:to_s) - %w[subject]
raise ArgumentError, "#{pre}begin with 'test'#{post}" if
name =~ /\Atest/
raise ArgumentError, "#{pre}override a method in Minitest::Spec#{post}" if
methods.include? name
define_method name do
@_memoized ||= {}
@_memoized.fetch(name) { |k| @_memoized[k] = instance_eval(&block) }
end
end
Here’s how I understand the method: When we evaluate a let
block, such as let(:spell_to_cast) { 'fireball' }
, we validate the argument, raising if it would either hide an existing method on the class or define a test_
method. Then, we define a method that says, “There’s a cache of method results (@_memoized
). If it’s nil, initialize it to an empty hash. Then, try to fetch from it by name, in this case, :spell_to_cast
. If that’s nil, execute the block that was passed in, cache the result, and then return the result.
What I want to prove to myself today is that it’s not likely that I’ll accidentally get a reference to a member of @_memoized
and mutate it.
To conduct this test I want an obviously mutable model:
class Lightswitch < ActiveRecord::Base
attr_accessor :turned_on
def turn_on
self.turned_on = true
end
end
From here my goal is simple: write believable-looking test code that gets a reference to an instance of Lightswitch
in @_memoized
, invokes turn_on
on it, and then pollutes subsequent tests with the fact that a light switch cache is on. I suspect this is impossible, but I want to understand why it’s impossible. Magic makes me nervous, because magic is always an abstraction that might leak when I don’t want it to.
Here’s the basic setup:
describe 'mutability of @_memoized via lightswitches' do
let(:switch_thats_off) do
value = Lightswitch.new
value.turned_on = false
value
end
it 'turns on' do
test_subject = switch_thats_off
test_subject.turn_on
assert test_subject.turned_on, 'switch should be on'
end
end
This is what I would consider a normal use. I want to understand why subsequent calls to switch_thats_off
don’t return the mutated instance of Lightswitch
from the @_memoized
cache.
First, I want to understand how often minitest runs constructors on test classes, so I made this constructor override:
def initialize(*)
super
puts 'i was initialized again'
these_methods = (self.methods - Object.new.methods)
.select {|m| /test_/ =~ m } # `select` is like C#'s `where`
puts "I have #{these_methods.length} test methods!"
end
This constructor results in the following output:
..........I was initialized again
I have 3 test methods!
.I was initialized again
I have 3 test methods!
.I was initialized again
I have 3 test methods!
This experiment makes it pretty clear to me that minitest runs the constructor every time it executes a test method. So why all the ceremony around let blocks? Why can’t I just say something like this:
def self.let name, &block
# TODO: raise if name is invalid or hides another method
value = block.call
define_method name do
value
end
end
I put this in my test class, along with some puts
statements so that I was sure it was being called, tried to detect a change. I came up with the following (super annoying) snippet of test:
it 'turns on the switch' do
switch_thats_off.turn_on
end
it 'turns on the switch' do
switch_thats_off.turn_on
end
it 'turns on the switch' do
switch_thats_off.turn_on
end
it 'turns on the switch' do
switch_thats_off.turn_on
end
it 'turns on the switch' do
switch_thats_off.turn_on
end
it 'should still start out off' do
refute switch_thats_off.turned_on, 'should be off'
end
Minitest calls these it
blocks in a random order, so in all probability one of the ‘turns the switch on’ methods will run before the ‘should still start out off’ method. It doesn’t. Hiding minitest’s let
method causes the tests above to fail in most cases (except for the off chance that the last block gets run before any of the others).
The problem with my implementation is that the block passed to define_method
closes over value
, so every instance of the test class ends up with a reference to the same instance of Lightswitch
, and the test runs interfere. The following method will almost work, but is also not what minitest chose to implement:
def self.let name, &block
define_method name do
block.call
end
end
Can you spot the issue with this one? There are two issues:
First, there’s no caching at all. Test code won’t even be able to mutate this variable within a block. Every call to the method name
, whatever it turns out to be called, will execute &block
all over again, and return a new object. It won’t behave like a local variable at all; the caller won’t even be able to mutate it and see the change within its own call block.
Second, because there’s no caching at all, the test code won’t be very performant; we’ll have to re-execute &block
every time we want the value.
In order to fix these issues, I settled on this issue, which aside from validation is the same as minitest’s:
def self.let name, &block
define_method name do
@_my_hash ||= {}
@_my_hash.fetch(name) {|k| @_my_hash[k] = block.call}
end
end
A quick aside: the fetch
usage in the code snippet above was unfamiliar to me. Basically, it means, “try to get key name
from the hash map, otherwise execute the block delimited by {}
and return its value. The block in the {}
in turn calls the block that was passed in and assigns the resulting value to the appropriate key in the hash.
This snippet, has, I think, the same semantics as a method that either initializes an instance variable or returns its current value if it’s already been initialized.
Summing Up
I started this article wondering how let
blocks worked, and discovered two things:
- minitest re-instantiates the entire test object every time it runs a
test_
method or anit
block. let
blocks work by defining an instance method on the test class that either initializes the value of some variable, caches, and returns it, or returns the cached value.
That answers my initial question, which was, “how is minitest sure that different test_
methods will not interfere with each other?” The answer is “by making a new test object every time a test_
method is run.”
The next thing I want to understand is how nesting describe
blocks works; I hope to have a post about that in the near future.
Till next time, happy learning!
-Will