Running a Minitest Suite tags: minitest testing ruby 2013-05-10

I have enjoyed using Minitest on a new project, but have been struggling with how I should be kicking off my test suite. This is a pretty simple thing in rspec, but wasn't obvious with Minitest. I have explored a few different methods and figured that I might as well document them.

Update: 2013-05-11

Requirements

While looking for a way to run the test suite I had a few specific requirements:

Speed
Running the test suite shouldn’t take longer than running the individual tests themselves.
Simplicity
Reasoning about the process should be VERY simple.
Minimal Output
Running the whole suite shouldn’t clutter my test output: I only want to see the test results.
Ease of Use
It should be very easy to trigger a full test run. (I run my test suite directly from terminal VIM)
Runs all Tests/Specs
It should run the whole test suite.

Methods Tested

The options that I have explored include:

Sample Spec

Throughout this article I will be using the following sample spec file to illustrate the various methods of running a suite:

require 'minitest/spec'

describe "simple failing test" do
  it "fails" do
    assert 1 < 0
  end
end

Using minitest/autorun

Using require 'minitest/autorun' in any given spec file will run any specs after the file has been evaluated. So to run the sample spec file from above you could run the following command:

ruby -r 'minitest/autorun' simple_spec.rb

Which would output the following:

Run options: --seed 2852

# Running tests:

F

Finished tests in 0.000599s, 1669.4491 tests/s, 1669.4491 assertions/s.

  1) Failure:
  simple failing test#test_0001_fails [spec/simple_spec.rb:3]:
  Failed assertion, no message given.

  1 tests, 1 assertions, 1 failures, 0 errors, 0 skips

You could also add require 'minitest/autorun' to the beginning of the file, and run it with:

ruby simple_spec.rb

Meets Requirements?

Speed
Yes, this method is super fast.
Simplicity
Yes, the source for how Minitest.autorun works is pretty straight forward (see here and here).
Minimal Output
Yes, the only output is that of Minitest itself.
Ease of Use
Yes, I can run it from VIM via :!ruby spec/simple_spec.rb.
Runs all Tests/Specs
No, running the tests via this method will only run the test on the files required. So if you just ran ruby spec/simple_spec.rb you would only get the test output for that particular set of specs (this is super useful, just not what I am going for here).

Using rake/testtask

One of the most recommended ways that I have seen is to setup and use Rake::TestTask in your Rakefile. A sample Rakefile might look like:

require 'rake/testtask'

Rake::TestTask.new do |t|
  t.libs << "spec"
  t.test_files = FileList['spec/**/*_spec.rb']
end

By default this gives you a new rake task named 'test' that will run all of the spec files found. The output is much the same with one major exception: when the test suite fails Rake will raise an error and print out the backtrace. Normally, you would want a backtrace from any errors thrown while running your tests, unfortunately this error actually has nothing to do with your test suite, and simply points to the line where Rake::TestTask itself calls Kernel#fail.

Here is a sample of the output:

Run options: --seed 57803

# Running tests:

F

Finished tests in 0.000538s, 1858.7361 tests/s, 1858.7361 assertions/s.

  1) Failure:
simple failing test#test_0001_fails [spec/simple_spec.rb:5]:
Failed assertion, no message given.

1 tests, 1 assertions, 1 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1): [ruby -I"lib:spec" -I"/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib" "/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/rake_test_loader.rb" "spec/simple_spec.rb" ]
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/testtask.rb:104:in `block (3 levels) in define'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/file_utils.rb:45:in `call'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/file_utils.rb:45:in `sh'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/file_utils_ext.rb:37:in `sh'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/file_utils.rb:82:in `ruby'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/file_utils_ext.rb:37:in `ruby'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/testtask.rb:100:in `block (2 levels) in define'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/file_utils_ext.rb:58:in `verbose'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/testtask.rb:98:in `block in define'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/task.rb:246:in `call'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/task.rb:246:in `block in execute'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/task.rb:241:in `each'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/task.rb:241:in `execute'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/task.rb:184:in `block in invoke_with_call_chain'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/task.rb:170:in `invoke'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/application.rb:143:in `invoke_task'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/application.rb:101:in `block (2 levels) in top_level'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/application.rb:101:in `each'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/application.rb:101:in `block in top_level'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/application.rb:110:in `run_with_threads'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/application.rb:95:in `top_level'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/application.rb:73:in `block in run'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/application.rb:160:in `standard_exception_handling'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/gems/rake-10.0.4/lib/rake/application.rb:70:in `run'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/bin/ruby_noexec_wrapper:14:in `eval'
/Users/rjackson/.rvm/gems/ruby-2.0.0-p0/bin/ruby_noexec_wrapper:14:in `<main>'
Tasks: TOP => rake_testtask
(See full trace by running task with --trace)

Meets Requirements?

Speed
Yes, this method is super fast.
Simplicity
No, the source for Rake::TestTask relies on a bunch of rake internal details. (If you understand rake’s internals, then this wouldn’t be an issue for you.)
Minimal Output
No, running the test suite generates a bunch of useless noise.
Ease of Use
Yes, I can run it from VIM via:!rake test.
Runs all Tests/Specs
Yes, this will run all files matching the glob passed in to the FileList.

Simple Script

After dealing with the rake/testtask backtrace for a while I started to think about what actually needed to happen. Minitest itself was doing most of the work, all I need to do is require the right files, and Minitest would take it from there. Leaning on Minitest's philosophy of simplicity I came up with:

Dir.glob('./spec/**/*_spec.rb').each { |file| require file}

This solution is so simple that I had initially completely overlooked it!

Meets Requirements?

Speed
Yes, this method is super fast.
Simplicity
Yes, this method only relies on standard require, and Dir.glob.
Minimal Output
Yes, the only output displayed is from Minitest.
Ease of Use
Yes, I can run it from VIM via :!rake test.
Runs all Tests/Specs
Yes, this will run all files matching the Dir.glob.

Conclusion

It was so easy to get caught up in the pageantry of the one true way to run the tests, but the reality is that all of these methods are perfectly valid (and likely many more). Any of them could work for your particular scenario.

I have truly enjoyed working with Minitest after years with rspec. Please don't get me wrong, rspec is a great tool, but it is very refreshing to be able to read and understand the source of your test framework in a short period of time (minitest is roughly 1/5 of the size of rspec; for details see here).

Keep an eye out for more articles as I continue to explore the world of minitest.

Update: 2013-05-11:

Since a few people have asked about this I figured it may be useful to explain exactly how I am using the little script above. I have embedded it into my spec_helper.rb inside an if __FILE__ == $0 section. That way if I execute the spec_helper.rb directly it will run the whole test suite, but I can still run individual test files (even though they require the spec_helper.rb). So a basic spec_helper.rb would look like this:

require 'minitest/autorun'

if __FILE__ == $0
  $LOAD_PATH.unshift('lib', 'spec')
  Dir.glob('./spec/**/*_spec.rb') { |f| require f }
end

So if I execute ruby spec/spec_helper.rb it will run the whole suite, but if I run something like ruby spec/some_random_spec.rb it will run just that one spec file.

You could also do the same thing with a specific rake task (as mentioned in one of the reddit comments). This might look something like:

task :test do
  $LOAD_PATH.unshift('lib', 'spec')
  Dir.glob('./spec/**/*_spec.rb') { |f| require f }
end
comments powered by Disqus