Test Driven Development with Ruby

By Peter Marklund on the 9th of May 2006

Introduction

I wrote this document for my presentation at the Rails Recipes Meetup in Stockholm on the 9th of May 2006. It covers Test Driven Development in general and how it can be applied to Ruby scripting and web development with Ruby on Rails. Some famliarity with these technologies is a prerequisite to fully understanding the text.

What is Test Driven Development (TDD)?

  • Motivation: 
1. If something can go wrong, it will.
2. If anything simply cannot go wrong, it will anyway.
3. Nothing is ever as simple as it seems.
  • Repeat after me: 
1. Write a test that specifies a tiny bit of functionality
2. Ensure the test fails (you haven't built the functionality yet!)
3. Write only the code necessary to make the test pass
4. Refactor the code, ensuring that it has the simplest design possible
   for the functionality built to date
  • Goal: clean code that works
  • Benefits: a decoupled design and documentation on how the code was intended to be used.

The Rails Testing Landscape

I previously wrote an article on how I test my website with model and controller tests. In this section I add some new lessons that I've learned and complete the picture with integration and helper tests.

The examples in this section are drawn from my homepage. You may view the source code and do an svn checkout from http://svn.marklunds.com/trunk.

Overview 

Testing tool                   Interface
---------------           -----------------------
Selenium            <->    IE, Firefox, Safari, ...
                                 | 
WWW::Mechanize, LWP <->        HTTP
                                 |
                               Rails
                                 |
Integration tests   <->        Routes
                                 |
                               Views
                                 |
Unit tests          <->        Helpers
                                 |
Functional tests    <->      Controllers
                                 |
Unit tests          <->        Models

Integration tests

With integration tests you can simulate whole use cases, aka user stories. An example would be user bob logs in, bob creates an order, bob logs out. Integration tests can also simulate multiple users and help recreate session problems.

As of May 10th 2006, you need to apply the following little patch to make the assert_tag method work (see rails bug 4631) together with integration tests: 

===================================================================
--- vendor/rails/actionpack/lib/action_controller/test_process.rb       (revision 4333)
+++ vendor/rails/actionpack/lib/action_controller/test_process.rb       (working copy)
@@ -416,7 +416,7 @@
     end

     def html_document
-      @html_document ||= HTML::Document.new(@response.body)
+      HTML::Document.new(@response.body)
     end

     def find_tag(conditions)

Here is an example of a simple integration test I use for my homepage: 

require "#{File.dirname(__FILE__)}/../test_helper"

class AdminTest < ActionController::IntegrationTest
  fixtures :users, :posts, :categories, :categories_posts

  # Simulate an admin creating a new post and going back to the front
  # page to view the post
  def test_creating_post
    # View the front page before creating the post
    get_page "/", "peter/index"
    assert_posts 3

    # Go to login page
    get_page "/login/login", "login/login"
    # Check form is present
    assert_tag :tag => 'form',
                   :attributes => 
                      {:action => '/login/login',
                        :method => 'post'
                      }
    
    # Log in
    post_via_redirect "/login/login", :user => {
         :name => 'admin',
        :password => 'abracadabra'
        }
    assert_response :success
    assert_template "admin/admin/index"

    # Visit posts admin index page
    get_page "/admin/post", "admin/post/list"
    
    # Visit page for creating post
    get_page "/admin/post/new", "admin/post/new"

    # Create post
    post_via_redirect "/admin/post/create", :post => {
        :subject => 'Integration testing is great!'
        }
    assert_response :success
    assert_template "admin/post/list"

    # Go back to homepage and view the new post
    get_page "/", "peter/index"
    assert_posts 4
    assert_tag :content => "Integration testing is great!"
  end

  private
  def get_page(url, expect_template)
    get url
    assert_response :success
    assert_template expect_template
  end

  # Assert there are a certain number of posts in the HTML
  def assert_posts(post_count)
    assert_tag :tag => 'div',
      :attributes => {:id => 'content'},
      :children => {:count => post_count,
                          :only => {
                            :tag => 'div',
                            :attributes => {
                              :class => 'post_body'
                            }
                          }
                        }
  end
end

Helper tests

Recipe 44 in the Rails Recipes book teaches us how to write tests for the helper methods we use in our views. For my homepage I only have a single helper method and here is the test case for it: 

require File.dirname(__FILE__) + '/../test_helper'

# Test case for the methods in PeterHelper
class PeterHelperTest < Test::Unit::TestCase
  include ActionView::Helpers::UrlHelper
  include ActionView::Helpers::TextHelper
  include ActionView::Helpers::TagHelper
  include PeterHelper

  def setup
    @controller = PeterController.new
    @controller.instance_eval { @params = {}, @request = ActionController::TestRequest.new }	
    @controller.send(:initialize_current_url)
  end

  def test_full_post_url
    assert_equal "http://marklunds.com/articles/one/23", full_post_url(23)
  end
end

This test case is not particularly exciting, but it serves to demonstrate how to write helper tests.

Controller tests

In the controller tests you check for example that your controller actions fetch the right content from the database, render the right views, and do the right redirects. You can also do validation against the HTML output of the views here using the assert_tag method. Here is an example of a controller test.

Model tests

In model tests you typically check that your validation code is working, go through the CRUD (create, read, update, delete) cycle, and check any methods you've added. There is typically quite a bit of overlap between model and controller tests, and since models in Rails typically have so little code, you may feel that you are testing the Rails framework rather than your own code. However it's better to test too much than too little. You should spell out your assumptions about how the code works in your tests. Here is an example model test.

Example: Refactoring

When writing this document I found the following code in the Post class for the weblog on my homepage: 

  def self.find_by_category(category_id)
    # I couldn't get Rails to do eager (pre) loading for this query
    # so there will be one categories query per post. TODO: make this
    # page (and other pages) paginated. 
    # If the number of categories queries really
    # became a problem I could of course do the eager loading
    # manually by writing the whole SQL query and 
    # instantiating the objects myself.
    find(:all, :order => "created_at DESC", 
               :conditions => ["posts.private = 0
                           AND posts.id = categories_posts.post_id
                           AND categories_posts.category_id = ?",
                   category_id],
               :joins => ', categories_posts')
  end

I figured that Rails had probably fixed the :include problem by now, refactored the method and assured myself that the I hadn't broken anything by re-running my tests: 

# Refactored method:

  def self.find_by_category(category_id)
    find(:all, :order => "created_at DESC", 
               :conditions => ["posts.private = 0
                  AND categories.id = ?", category_id],
               :include => 'categories')
  end

# Tests still pass:

 $ rake
...
Finished in 0.13 seconds.

1 tests, 1 assertions, 0 failures, 0 errors

Example: Manual Testing and Reproducing Bugs

When browsing my website in development with the test fixtures data (tip: the rake task db:fixtures:load loads the fixtures for you) I noticed that the menu in my weblog would list categories with no public posts. Now of course I asked myself how my tests could have missed this. It turns out I had test coverage for this bug only I hadn't realized it was a problem when I wrote the tests so the tests were making erroneous assertions: 

# In peter_controller_test.rb:
  def assert_layout_used
    assert_not_nil assigns(:archive_years)
    assert_equal 2, assigns(:archive_years).size
    assert_not_nil assigns(:categories)
    assert_equal 3, assigns(:categories).size  # This should be 2!

    assert_tag :tag => 'div', :attributes => {:id => 'menu'}
    assert_tag :tag => 'div', :attributes => {:id => 'content'}   
  end

# In category_test.rb:
# Testing shouldn't be in the list here since that category
# has no public posts
  def test_all_categories
    assert_equal ['Business', 'Programming', 'Testing'],
                 Category.all_categories.map { |c| c.name }
  end

Correcting the tests I was able to reproduce the bug with rake. The next step was to fix the bug which was fairly easy, I just needed to add a sub-select condition to the categories query: 

  def self.all_categories
    find(:all, :conditions => 
    ['id in (select distinct category_id
           from categories_posts,
                posts
           where categories_posts.post_id = posts.id
             and posts.private = 0)'], 
      :order => "name")
  end

Tips

  • Write small and focused test methods that check only one aspect of your code.
  • Avoid fragile assertions on human readable text such as error or flash messages.
  • Keep your test code DRY and readable by creating an API or even better a Domain Specific Language (DSL). See Jamis Buck's article on DSLs.
  • If you are using database agnostic migrations you can use SQLite to speed up your test database.
  • Set up a script that runs tests on every source commit or at least a couple of times a day. At one of my previous jobs this was known as the auto-shame script  :-) See the continuous_builder plugin and recipe 46 in the Rails Recipes book.
  • Use the coverage library to find patches of unexecuted code (more about this below).
  • Transactional fixtures can bite you. If you are using transactions within your code they won't work.
  • The RailsTidy plugin adds the assert_tidy method to Test::Unit::TestCase so that you can can validate the HTML of your views in your controller and integration tests.

Test Driven Scripting

In this section I talk about lessons I learned from applying test driven development to command line scripting with Ruby. You may view the source code and do an svn checkout from http://svn.marklunds.com/bin/trunk to better be able to follow the examples.

Background

  • I needed a script for uploading photos to my website
  • What I had was a collection of untested Perl and Bash hacks
  • By converting to command line scripts that invoke Ruby classes I achieved clean and testable codebase and a framework for adding new scripts
  • I use a setup similar to Rails with the Test::Unit library and the rake build tool

Regression Testing

I run tests when I refactor to verify that I haven't changed or broken current functionality. Example: 

# Before refactoring:

    def despacify(dir)
      Dir.entries(dir).each do |filename|
        next if ['.', '..'].include? filename
	filepath = File.join(dir, filename)
        despacify(filepath) if File.directory?(filepath)
        if filename =~ /\s/
          new_filename = filename.gsub(/\s/, '_')
          FileUtils.mv filepath, File.join(dir, new_filename)
        end
      end      
    end


# After refactoring:

    def despacify(dir)
      Find.find(dir) do |path|
        if File.basename(path) =~ /\s/
          new_filename = File.basename(path).gsub(/\s/, '_')
          FileUtils.mv path, File.join(dir, new_filename)
        end
      end      
    end


# Test result:

     1) Failure:
    test_main_usecase(TestDespacifier) [./test/photos/test_despacifier.rb:34]:
    <[".", "..", "file_4"]> expected but was
    <[".", "..", "file 4"]>.

The reason the refactored code fails is an ordering problem. Sometimes the code renames (despacifies) a directory and later tries to rename the files in that directory with the old directory name. Possible resolutions:

  • revert the refactoring
  • simplify the behaviour (despacify only regular files, not directories) so the refactored code works
  • fix the ordering problem.

Reproducing Bugs

When I discover a bug I write a test to reproduce it. Example: 

# Before bug fix:

    # Return given path (which may be absolute or relative to the current directory) 
    # relative to from_path
    def self.make_relative(path, from_path)
      Pathname.new(path).relative_path_from(Pathname.new(from_path)).to_s
    end


# Test to reproduce bug:

  def test_make_relative
    # Reproduce bug when first arg is relative and second absolute
    assert_equal("../2006/stockholm/valborg", 
      Photos::Resizor.make_relative("2006/stockholm/valborg", File.expand_path(PHOTOS_DIR)))
  end


# Test result:

  1) Error:
test_make_relative(TestResizor):
ArgumentError: relative path between absolute and relative path:
#<Pathname:2006/stockholm/valborg>, 
#<Pathname:/home/Peter Marklund/bin/test_resizor_photos_dir>
    /usr/lib/ruby/1.8/pathname.rb:529:in `relative_path_from'


# After bug fix:

    # Return given path (which may be absolute or relative 
    # to the current directory) relative to from_path
    def self.make_relative(path, from_path)
      # There seems to be a bug in the Pathname#relative_path_from 
      # method when first argument is relative and second
      # absolute. If we make both paths absolute it works
      Pathname.new(File.expand_path(path)).
        relative_path_from(Pathname.new(File.expand_path(from_path))).to_s
    end


# Test result:

  Finished in 1.341 seconds.
  
  2 tests, 11 assertions, 0 failures, 0 error

Code Coverage

With the coverage library I can check for patches of unexecuted code. Code that has never run may very well contain misspelled variable or class names or have incorrect syntax. Example: 

gem install coverage
ruby -rcoverage test/run.rb

# Found code not covered:

      if File.exists?(@target_dir)
 	raise(InvalidInputError, 
 	      "Target directory '#{target_dir}' already exists - aborting")
      end



# Test to cover code:

  def test_target_dir_exists
    assert_raise(InvalidInputError) do
      MergeDirs.new([@dir1['name'], @dir2['name'], @dir1['name']])
    end
  end



# Test result exposing the missing "@" in "#{target_dir}": 

   1) Failure:
  test_target_dir_exists(TestMergeDirs) [test/test_merge_dirs.rb:47]:
  <InvalidInputError> exception expected but was
  Class: <NameError>
  Message: <"undefined local variable or method `target_dir' for #<MergeDirs:0x383018>">
  ---Backtrace---
  ./test/../lib/merge_dirs.rb:17:in `initialize'
  test/test_merge_dirs.rb:48:in `test_target_dir_exists'
  test/test_merge_dirs.rb:47:in `test_target_dir_exists'
  ---------------
  
  3 tests, 13 assertions, 1 failures, 0 errors

Test Coverage

I use assertions to check that the code behaves correctly. Remember that code coverage does not equal test coverage. Code coverage only means the code has executed, not that it produced the right results. The number of assertions per lines of code can serve as a crude measure of test coverage. As shown below I have 38 assertions for 324 lines of code, i.e. about one assertion per every 10 line of code. 

  $ rake
  (in /home/Peter Marklund/bin)
  /usr/bin/ruby -Ilib "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb"   "test/test_merge_dirs.rb" "test/photos/test_despacifier.rb" "test/photos/test_publisher.rb"   "test/photos/test_resizor.rb" 
  Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader
  Started
  .........
  Finished in 2.433 seconds.

  9 tests, 38 assertions, 0 failures, 0 errors

  $ rake stats
  (in /home/Peter Marklund/bin)
  +----------------------+-------+-------+---------+---------+-----+-------+
  | Name                 | Lines |   LOC | Classes | Methods | M/C | LOC/M |
  +----------------------+-------+-------+---------+---------+-----+-------+
  | Code                 |   437 |   324 |      10 |      35 |   3 |     7 |
  |   Tests              |   345 |   252 |       4 |      28 |   7 |     7 |
  +----------------------+-------+-------+---------+---------+-----+-------+
  | Total                |   782 |   576 |      14 |      63 |   4 |     7 |
  +----------------------+-------+-------+---------+---------+-----+-------+
    Code LOC: 324     Test LOC: 252     Code to Test Ratio: 1:0.8

You can find gaps in your test coverage by searching for if statements that you could reverse or lines of code that you could comment out without the tests breaking.

Interview with Martin Fowler

Unhurriedness

Martin Fowler:

"There's an impossible-to-express quality about test-first design that gives you a sense of unhurriedness. You are actually moving very quickly, but there's an unhurriedness because you are creating little micro-goals for yourself and satisfying them. At each point you know you are doing one micro-goal piece of work, and it's done when the test passes. That is a very calming thing. It reduces the scope of what you have to think about. You don't have to think about everything you have to do in the class. You just have to think about one little piece of responsibility. You make that work and then you refactor it so everything is very nicely designed."

When to Stop

Bill Venners:

"When do you stop writing tests? You say in Refactoring, "There's a point of diminishing returns with testing, and there's a danger that by writing too many tests you become discouraged and end up not writing any. You should concentrate on where the risk is." How do you know where the risk is?"

Martin Fowler:

"Ask yourself which bits of the program would you be scared to change? One test I've come up with since the Refactoring book is asking if there is any line of code that you could comment out and the tests wouldn't fail? If so, you are either missing a test or you've got an unnecessary line of code. Similarly, take any Boolean expression. Could you just reverse it? What test would fail? If there's not a test failing, then, you've obviously got some more tests to write or some code to remove."

Other Methods for Reducing the Bug Count

Arguably, TDD has been hyped lately and it is not a silver bullet. Remember that TDD is only as good as the tests are. In this section I offer some complementary methods you should consider.

Think!

  • Review and care about your code. Think hard about what could go wrong
  • Be knowledgeable about the tools that you use - read the docs.
  • Understand the problem domain
  • Be thorough, methodical, and patient

Simplify

Complex systems tend to be more error prone than simple ones.

  • Reduce functional scope. Projects have limited time and resouces and the more functional ground you have to cover the lower the quality will be.
  • Elliminate special cases and convoluted logic. Look for FIXME comments in the code such as "Yes I know this is messy, but...".
  • Get rid of external dependencies. A warning flag could be a README files saying "before you do this, remember to run this or that script".
  • Make your code less generic. Don't build for future anticipated requirements.
  • Streamline your implementation. Reduce the line count for your code whilst maintaining readability. Ruby is a great help in this respect.

Work with Others

  • Teach someone how your system works. This is a way to discover if your implementation is convoluted and needs simplifying. It's also a test of your own understanding of the system.
  • Have your code be reviewed by another programmer who also cares about code quality
  • Have someone write tests for your code. We tend to be blind to our own bugs. Typically, there is a set of circumstances that triggers the bug that didn't occur to you when you wrote the code or when you wrote the tests.

Do Manual Testing

Automated tests won't catch all bugs. Just bringing up your site in the browser running on the test database might reveal obvious problems.

Monitor your Site in Production

Schedule a script to check for broken links, do markup validation, and check for expected tags and texts on your pages. This way you can respond quickly when a bug has slipped into production or your server is having problems.

Make sure you get notified about errors and exceptions in your web application. For instructions on how to do this with Rails, see recipe 47 in the Rails Recipes book, or google for rescue_action_in_public.

Resources