When I first started poking around the codebase a few months ago, they’d just started writing a few cucumber selenium integration tests – which of course I want to encourage! – but they weren’t running them on their continuous integration box. And if you aren’t running them on CI, you can’t really call them tests…

It’s not that the developers were lazy; it’s just that Diaspora’s CI box is an ubuntu server instance that doesn’t even have xwindows. There’s no way to attach a display, and without a display, a browser won’t run. So how can you run selenium features that have to actually open a browser window and click on stuff?

I’m glad you asked!

You set up a virtual framebuffer – essentially a simulated display – and run your features in there. Here’s how I got this going.

Step 1: Install Crap

Yeah, you need xwindows, and a bunch of other stuff. I’m not a frequent linux user, so I was surprised that a lot of these commands appeared to fail, with extremely lengthy output about how such-and-such dependency could not be installed, or compiled, or resolved, or kafloozled, or whatever. But then when I ran the programs that supposedly weren’t installed, they worked fine. …thanks, ubuntu?

sudo aptitude install xubuntu-desktop
sudo aptitude install exaile
sudo aptitude install gconf2
sudo aptitude install xvfb
sudo aptitude install firefox

Step 2: Verify Your Virtual Framebuffer

I like the phrase “virtual framebuffer.” It’s very computer science, and using it makes me feel smart. But really, it’s just a display that gets rendered in memory, but not output to any monitor. Programs that require the ability to create windows, such as Firefox, need a display to run in – but they’ll take any old display. If you give Firefox a virtual framebuffer, it’s perfectly happy and treats it like any other display, even though no one can see it.

So let’s verify we have everything set up to run one. This magic incantation starts one on display port :99.

 Xvfb :99 -ac -screen 0 1024x768x16

In another shell, run this one, which opens Firefox to example.org inside the virtual display you’re running in the original shell.

DISPLAY=:99.0 firefox http://example.org

Firefox is now open to example.org, the world’s most boring webpage. Really! Let’s prove it by taking a screenshot in a third shell.

xwd -root -display :99.0 -out xwdout

Your screenshot is now in a file called xwdout. Sadly, this is not an image format that normal image viewers can see, so let’s convert it to a jpeg.

convert xwdout screenshot.jpg

Using your favorite sftp program, retrieve screenshot.jpg and have a look at it. It should look something like this:

That boxy UI, those icons from 1995...it could only be linux.

Ta-da! You have evidence of this so-called “virtual” framebuffer. Now all we have to do is set it up so to start before we run the tests, and stop once the tests are done.

Step 3: Adventures in Shell Scripting

This goes in your /etc/init.d, and will give you ULTIMATE POWER, if ultimate means you can start and stop a virtual framebuffer on display port :99.

Now you just need a rake task that starts the display, passes the port info to the tests, and then stops it.

And just in case you have to get this running with rvm, here’s the shell script that cc.rb calls to kick off a Diaspora build on ruby 1.8.7.

I’ve had to set this up twice, now, but there are a lot of moving parts. If you try this and notice something I forgot, please let me know.

Enjoy your headless cucumbers…

(I wonder what kind of google search results that will get me.)

How do you start, and how do you know you’re finished? What do you test, and where? These questions can be answered hundreds of different ways, but here’s my way.

The first code I write: a feature

As a developer, rather than a designer, I’m always tempted to start with unit tests and work out towards a cucumber feature (“inside-out” testing). But that approach gets me into no end of trouble. I usually end up writing and testing stuff on the model that I don’t ultimately need. Plus once I’m down in the weeds coding, I lose track of the big picture.

So I like to do outside-in testing instead. I start each story I get from tracker with a cucumber feature that expresses how the PM will be able to accept it when I’m done. The feature helps me frame the problem properly, and focus on doing exactly what I need to make it work. Since I come back to it periodically while I’m coding, I keep focused on the higher-level goal. And finally – if I write it first, I can’t skip writing it once I’m done.

Before we get going…

There are certain types of tests I don’t write in this example (and in some cases, at all). Let’s get those out of the way so you don’t have to come up with a scathing comment at the bottom of the post.

• Model tests. In this example, my model doesn’t do anything other than default ActiveRecord behavior, so it doesn’t need any tests. Don’t test rails internals. Once my model has custom behavior, it will have specs, too.
• View tests. I have no tests that verify that my markup is what I expect. That’s because they’re a waste of time. Yes, even with complex views. Verify behavior with cucumber tests, unit-test Javascript with jasmine, and leave the rest to the humans. You’ll waste more developer time maintaining them than it would take humans to verify them. Verifiers are a whole lot cheaper than developers.
• Error case tests. In this example, there are no error cases. The model has no validations, and the table has no constraints. Once there are error cases, I generally put those in the model if I can, in the controller when I have to, and never in the cucumber tests. The latter is mostly a suite-speed consideration – cucumber tests run much more slowly than rspec. Cucumber’s great for for happy path tests; I leave the rest to rspec.

Let’s get going!

The first feature

Say I’m doing a library app and the first story is “User can enter a new book into the system.” Before I write any other code, I write this feature:

Feature: User manages books
  Scenario: User adds a new book
    Given I go to the new book page
    And I fill in "Name" with "War & Peace"
    And I fill in "Description" with "Long Russian novel"
    When I press "Create"
    Then I should be on the book list page
    And I should see "War & Peace"

Starting the fail-fix cycle

I run it using cucumber features, and it fails on the first line – Given I go to the new book page – because cucumber doesn’t know where the “new book page” is. So I add that to the cucumber paths helper.

    when /the new book page/
      new_book_path

Now when I run cucumber, it fails because it can’t find new_book_path. So I add that to routes.rb:

  map.resources :books,  nly => [:new]

Now when I run cucumber, it complains that it can’t find the BooksController. That means it’s time to dive down to rspec controller tests.

My first spec experience

I create books_controller_spec.rb in spec/controllers, and add a test for the new method:

require File.expand_path(File.join(File.dirname(__FILE__), '..', 'spec_helper.rb'))
describe BooksController do
  describe "#new" do
    it "should be successful" do
      get :new
      response.should be_success
    end
  end
end

When I run this spec, it complains that there is no BooksController. Fixed:

class BooksController < ApplicationController
end

I re-run the spec and get "no action responded to new." So I add the new method.

class BooksController < ApplicationController
  def new
  end
end

Now the spec passes! Time to check back with cucumber.

Getting past the first line

I read through my cucumber feature again:

Feature: User manages books
  Scenario: User adds a new book
    Given I go to the new book page
    And I fill in "Name" with "War & Peace"
    And I fill in "Description" with "Long Russian novel"
    When I press "Create"
    Then I should be on the book list page
    And I should see "War & Peace"

Last time I ran it, it failed on the first line because it couldn't find the BooksController. This time, same location, but it says it can't find the view. So whiny! To placate it, I create an empty view called new.html.erb and run it again.

Now cucumber gets past line 1 (huzzah!!) and fails on line 2 (And I fill in "Name" with "War & Peace") with the message that it can't find a field called Name to fill in. So I add a standard rails form to the view.

<%- form_for @book do |f| -%>
    <%= f.label :name %>
    <%= f.text_field :name %>
    <%= f.label :description %>
    <%= f.text_area :description %>
    <%= f.submit "Create" %>
<%- end -%>

Uh oh. Cucumber is mad at me because there is no @book object. Back to rspec for me!

rspec: The Return

In my controller's new method, I need to create a book object that the form will use. I first add a test for that in the controller spec:

require File.expand_path(File.join(File.dirname(__FILE__), '..', 'spec_helper.rb'))
describe BooksController do
  describe "#new" do
    before do
      get :new
    end
    it "should be successful" do
      response.should be_success
    end
    it "should create a book object" do
      assigns(:book).should_not be_nil
    end
  end
end

This fails the right way - it says assigns(:book) is nil. So then I add the creation of the book object to the controller.

class BooksController < ApplicationController
  def new
    @book = Book.new
  end
end

Now the spec fails, saying it can't find the Book class. It has a point - I haven't created the model yet. Fixed:

class Book < ActiveRecord::Base
end

Now it fails saying it can't find the books table. So I write a migration that creates that.

class CreateBooksTable < ActiveRecord::Migration
  def self.up
    create_table :books do |t|
      t.string :name
      t.text :description
    end
  end
  def self.down
    drop_table :books
  end
end

Once I do rake db:migrate and rake db:test:prepare, I re-run my controller spec....and it passes! Back to the cucumber feature!

Cucumber...again.

In our last episode, cucumber was visibly annoyed because there was no @book object for the form to operate on. I run it again to see if it's still sulking.

Yep. This time it tells me that it can't find books_path. form_for tries to submit to the create path by default, which I haven't added yet. I add it to the routes.

  map.resources :books,  nly => [:new, :create]

This time, when I run cucumber, it gets through the first three lines (woo hoo!) and fails on the 4th, saying no action responded to create. Back to the rspec-cave, batman!

rspec: The Sequel to The Return

I add a controller spec for the create method.

  describe "#create" do
    it "should create a new book" do
      post :create, "book" => {"name" => "Jane Eyre", "description" => "Something Victorian"}
      assigns(:book).should_not be_nil
      assigns(:book).name.should == "Jane Eyre"
    end
  end

When I run it, I get the same message as in cucumber: no action responded to create. So I create the create:

class BooksController < ApplicationController
  def new
    @book = Book.new
  end
  def create
  end
end

Now when I re-run the spec, it fails saying that assigns(:book) is nil, which makes sense. I put in the guts of create to make that pass.

  def create
    @book = Book.new(params[:book])
    @book.save
  end

Now rspec passes! Back to cucumber.

So...cucumber. We meet again.

When I re-run the feature, it says I'm missing a template for create, which is correct. However, in this case, I don't want to make a template for create - I want to redirect to the book list page. So once again, I'm back with rspec.

rspec: Back so soon?

I add that expectation to the controller spec for create.

    it "should redirect to the book list page" do
      response.should redirect_to books_path
    end

It fails saying there's no redirect. So to make it pass, I add a redirect to the controller code.

  def create
    @book = Book.new(params[:book])
    if @book.save
      redirect_to books_path
    end
  end

Now my controller specs pass. Cucumber, I'm coming for you!

Oh, you again.

Last time, we got through the first 3 lines of the feature and failed on line 4 (When I press "Create"). When I run it this time, it gets through the same 3 lines and then fails in the same place again, saying that no action responded to index. I add index to the routes.

  map.resources :books,  nly => [:new, :create, :index]

I re-run the feature and get the same error message. WTF, cucumber?! It turns out that rails' implementation of REST uses the same path helper for create and index, so the path helper for index already exists, even though the method does not. A little strange, I know. But we need an index method, so it's back to rspec.

rspec: For the first time, for the last time...

I write a spec for the index method.

  describe "#index" do
    it "should be successful" do
      get :index
      response.should be_success
    end
  end

I still get no action responded to index. So l add the method in BooksController, empty to start.

  def index
  end

Specs pass, back to cucumber!

How can I miss you if you won't go away?

Cucumber tells me there's no template for index. So I create an empty one, and re-run. This run, for the first time, I pass line 4 (yaaaaay) but then it fails on line 5 (Then I should be on the book list page) because it can't figure out what I mean by "the book list page." That goes in the cucumber path helper.

    when /the book list page/
      books_path

OMG five out of six steps pass! Now cucumber says it can't find "War & Peace" on the page, so let's make the index view list the existing books. Back to rspec...

Don't go away mad...just go away.

I add the following it block to the spec for index.

    it "should assign a list of existing books" do
      Book.create!(:name => "Endymion", :description => "weird")
      get :index
      assigns(:books).should_not be_nil
      assigns(:books).length.should == 1
    end

It fails because I'm not creating @books in the controller, so I fix that.

  def index
    @books = Book.all
  end

Now the specs pass - back to cucumber.

We really have to stop seeing each other like this.

Cucumber still says it can't find War & Peace, because I haven't added printing out the books to the index view. I'll fix that.

<%- @books.each do |book| -%>
    <%= h book.name %>
    <%= h book.description %>
<%- end -%>

Re-run cucumber and ... ta-da! The feature passes! I've done everything I need to call the story done. I have the minimum amount of code I need, because all the code I wrote was driven by the feature. Story: delivered!