My journey from Rspec feature tests to Rails system tests

Goals:

• Achieve reliable headless testing with system tests that utilize javascript
• Learn the new Rails system test setup

Quite often I find myself spending hours debugging(and sometimes finally deleting) a flaky feature test. Finally I decided to stop doing that, and after watching the Rails Conf talk about system testing, decided it was time to act.

Since my first goal is to always keep it simple, I didn't jump right away into the Rails system tests. Due to the rest of my testing suite being written in Rspec, I first tried to go headless with that setup.

Rspec failure(s)

Trying headless chrome with capybara

spec/rails_helper.rb

Capybara.register_driver :headless_chrome do |app|
  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
    chromeOptions: { args: %w(headless disable-gpu) }
  )

  Capybara::Selenium::Driver.new app,
    browser: :chrome,
    desired_capabilities: capabilities
end

Capybara.javascript_driver = :headless_chrome

Result:

• Selenium::WebDriver::Error::unknownError...basically it gave me the 'is not clickable at point' error. I tried working around these by doing simple fixes to capybara finders, digging more for answers, and in the end(about 30 minutes), couldn't get it resolved. So I moved on to the next attempt...

Trying capybara webkit

Result:

• Well this didn't go very far as it needed a Qt dependency, and when I tried to brew install that, the compiler barfed due to, what seems to be, Xcode upgrade dependency. Due to not having admin privileges on my mac, I took the easy way out and abandoned webkit for now.

Trying poltergeist

spec/rails_helper.rb

require 'capybara/poltergeist'

Capybara.javascript_driver = :poltergeist

Modifying some tests for known 'click_on' issues:

Before:

click_on 'Create Engine'

After:

find("input[type=submit][value='Create Engine']").trigger('click')

Result:

• Success! ...or so I thought. All features passed. However, when ran with entire test suite, they randomly failed. So I tried fixing them with different swap-outs for 'have_content' checks and such. Those then passed, ran again, they failed and the others(that I didn't fix) passed. At this point I was thoroughly frustrated, because either I run my test suite for features separately from other tests, or I need to fix this...
• Note: I assume this has to due with the database cleanup and different AR threads and such that I've read about. I did have my database cleaner setup to truncate on js tests, but when ran in headless mode with the rest of my test suite, it was flaky. Also, before going down this headless path, they worked fine.

Being fed up with flaky tests and 3 failed attempts at remedying this in Rspec, I decided to now move on and try the Rails System Tests...

Rails System Tests:

Setup:

Prerequisites:

• Rails 5.1
• Poltergeist
• Mocha
• Puma
• Some other gems I like: FactoryGirl/Capybara(of course), standard test setup

test/test_helper.rb

• Simple setup really, only added a few helpers and such:

require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'
require 'mocha/test_unit' # for stubs
require 'minitest/spec' # for let/it/etc

# not sure about this, but rails kinda does it here...feels sloppy, but I hate
# typing require in all my test files
Dir[Rails.root.join("test/support/**/*.rb")}.each { |f| require f }

class ActiveSupport::TestCase
  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
  fixtures :all

  # Add more helper methods to be used by all tests here...
  extend MiniTest::Spec::DSL
  include WaitForAjax # hack for remote js calls that waits for ajax to be done
end

test/application_system_test_case.rb

• This file comes with Rails 5.1, but I switched it to use poltergeist:

require 'test_helper'
require 'capybara/poltergeist'

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :poltergeist
end

test/system/app/my_map_test.rb

• This is just one of the test files, but good enough to show the setup
• I like to abstract away a lot of the standard Capybara DOM traversal steps, so those will be in the next file.

require 'application_system_test_case'

class MyMapTest < ApplicationSystemTestCase
  class StandardType < MyMapTest
    let(:subject) { MyMapForm.new('standard_type') }

    test 'page has different types' do
      subject.visit_page

      assert has_content?('first type')
      assert has_content?('second type')
      assert has_content?('third type')
    end

    test 'page has address link' do
      subject.visit_page
      assert has_link?('Address Import')
    end
  end
end

Note: this was originally written in Rspec, but the conversion was simple...

• s/describe/class, s/it/test, s/expect..to/assert, s/have_content/has_content?..etc

test/support/forms/my_map_form.rb

• This is my file to hold all the repeatable/dirty details about traversing the DOM

class MyMapForm
  include Capybara::DSL
  include FactoryGirl::Syntax::Methods
  include Rails.application.routes.url_helpers

  def initialize(type)
    # using factory here of course and converting to find the factory
    @item = create(type.to_sym)
    # one more...
    create(type.to_sym)
    @app = item.app
  end

  def visit_page
    # don't care about authenticating here
    # in rspec, this uses allow_any_instance_of...
    ApplicationController.any_instance.stubs(:require_long).returns(true)
    ApplicationController.any_instance.stubs(:can_edit?).returns(true)

    visit app_path(app)
    find('a', test: 'My Map').click # in non headless, click_on 'My Map' works
    self
  end

  private 

  attr_reader :item, :app
end

Conclusions:

Pros:

• It seems to work so far...and let's be honest, that is the most important thing.
• Default functionality of a screenshot in my terminal on failure. Basic, but nice.
• Seems to solve the multiple db connection issue, and therefore reducing the flakiness of these gui driven system tests
• Runs faster than non headless implementation
• I suppose it makes running tests in CI env easier, as we can skip some of the browser setup?
• Gives the team some exposure to Rails test framework (this can also be seen as a con below)
• By using a different test suite for system tests, it gives a clear line of delineation in configs and troubleshooting, etc.
• Is part of Rails core, so I have full confidence in the future support and improvements

Cons:

• Adds technical debt to the team since we do everything else in Rspec
• Documentation/examples for Rspec is miles ahead from what I've been able to find for Rails System tests (hence some of the reason for this writeup)

Final Thoughts:

While this journey into Rails System tests feels good today, it is quite possible I will look back in a month or so and wonder...what was I thinking using this and multiple test suites... But then again, don't we always look back and have those thoughts about our development decisions?

Of note, I also see that rspec-rails, is planning on supporting this soon as well, I think, as seen by this issue. If that comes to fruition, it will be worth a look, and definitely worth consolidating back into one test suite...we'll see.

Comments

[Ben Halpern]

@maestromac thoughts on this?

[Mac Siri]

This is very interesting. Here are my thoughts on those drivers

• Poltergeist: worked great but doesn't officially support es6.
• Webkit: Installing Qt was a major pain. I eventually got it installed. I thought it gives the best performance but it was passing js tests that should've failed. If I had given it more time I'm sure it would work well.
• chromehelper: This is what we stuck with, but not the headless version because it felt slower (could be just me). Things worked well out of the box and it pleases me to be testing on the most popular web-driver.

This is the first time I heard of Rails system test. I would have to upgrade to Rails 5 first in order to use it 😅. Thank you so much for sharing!!

[Doug Stull]

good to know about es6! Before this feature, I had just finished converting to using webpacker gem and converted a few js files into es6. I don't think I had any system tests around the js classes, so perhaps I'll create one and see how poltergeist does :)