Integration & Unit Tests with TDD in Rails

Recent articles from Progress Updates:

• The only daily stand up meeting template you need
• The only two questions you need to ask in your daily standup scrum
• The only daily standup meeting agenda you will ever need
• How to Create a Weekly Report With Templates and Examples
• How to automate your end of week status reports

-

The advantage of using outside-in TDD is that it flows naturally from the business requirements, to the user actions, and then to the internals of the application - and we don't have to think too far ahead while implementing this. We just have to worry about the next thing in the test, and it will drive you to implement all the details, and then deliver the feature.

The challenge with TDD is that you have to be really familiar with Rails and testing in particular. Because right in the beginning you have to write the high-level integration tests which requires several technologies to work well together. And when the test fails, you have to know exactly what needs to be done to pass the test.

Implementing a new feature via TDD

I have a Shopify app that is live on the app store, and for marketing purposes, I wanted to build a free feature for the app. This post looks into how I went about adding the new feature using Test Driven Development.

This is the sample flow for the new feature:

1. A user visits the landing page.
2. He then enters the main keyword and the Shopify product page URL on the form
3. When he submits the form, the details are saved in the database
4. A background-job is started in parallel (Lets call it ScrapePageWorker)
5. The user is taken to the results page

Unit tests are useful for testing business logic in models, and controller tests are useful for testing controllers in isolation - making sure they receive the right inputs and that they render the right response. The above 5 step flow seems ideal for an integration test as we are not dealing with just one controller. Instead we are going all over the application - starting with the landing page, talking to the database, starting a BG job and being redirected to another page.

Integration Tests

I am using Rspec for testing, and I start by creating a new file boostctr/spec/features/free_marketing_tool_spec.rb

Lets start with the first step in sample work flow: A user visits the landing page.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
    end
  end
end

In the next two steps, he enters the details into the form and submits it. I am using Capybara for simulation real-world interactions - to get the virtual user to fill in the details in the form.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
+     fill_in "Enter focus keyword", with: "modular sofa"
+     fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
+     click_button "Check Product Page SEO"
    end
  end
end

Upto now, the user has visited the page, filled in the details and clicked the submit button. At this stage a few things should happen simultaneously - 1) The details are saved to the database, 2) the user is redirected to another page, and 3) a background job starts to scrape the product page to get all the required data.

Let's start with the HTTP side by making sure that the user is redirected to the show page.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
      fill_in "Enter focus keyword", with: "modular sofa"
      fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
      click_button "Check Product Page SEO"
+     expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
    end
  end
end

Next, we'll check the database to make sure that the form details have been saved into a table called Audits, and that the information stored is correct.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
      fill_in "Enter focus keyword", with: "modular sofa"
      fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
      click_button "Check Product Page SEO"
      expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
+     expect(Audit.first.focus_keyword).to eq('modular sofa')
+     expect(Audit.first.url).to eq('https://hauslondon.com/products/mags-modular-sofa-by-hay')
    end
  end
end

We also need to make sure that the Sidekiq background job runs properly. For this, we'll call the inline mode.

require 'spec_helper'
  feature "free marketing tool" do
    scenario "first time" do 
      visit audits_path
      expect(page).to have_content("Shopify Product Page SEO Checker")
      fill_in "Enter focus keyword", with: "modular sofa"
      fill_in "Enter page URL", with: "https://hauslondon.com/products/mags-modular-sofa-by-hay"
      click_button "Check Product Page SEO"
      expect(page).to have_content("Thank you! We are checking if the product page is SEO optimised.")
      expect(Audit.first.focus_keyword).to eq('modular sofa')
      expect(Audit.first.url).to eq('https://hauslondon.com/products/mags-modular-sofa-by-hay')
+     Sidekiq::Testing.inline! do
+       MarketingTool::ScrapePageWorker.perform_async(Audit.first.id, Audit.first.focus_keyword, Audit.first.url)
+     end
    end
  end
end

Background Workers

Calling Sidekiq inline makes the job run immediately instead of enqueuing it. For this to work, we have to create a Sidekiq background worker, and I do this by creating a new file boostctr/workers/marketing_tool/scrape_page_worker.rb

class MarketingTool::ScrapePageWorker
  include Sidekiq::Worker
  sidekiq_options :queue => 'default', :retry => false

  def perform(audit_id, keyword, url)
    PageScraper.new(audit_id, keyword, url).start_scraping
  end
end

Service Objects

Scraping a page does not really fit into a model or a controller. Also, I feel that I will be using this page scraping functionality in other parts of my app. Therefore, I can use Service Objects for this scenario. Service Objects are Plain Old Ruby Objects(PORO) that are used to execute one single thing, and I can create a new file for this on boostctr/services/page_scraper.rb

For actually scraping and reading the contents of the product page URL I am using the HTTP gem and Nokigiri.

class PageScraper
  attr_reader :audit_id, :keyword, :url

  def initialize(audit_id, keyword, url)
    @audit_id = audit_id
    @keyword = keyword
    @url = url
  end

  def start_scraping
    begin
      response = HTTP.get(url)
    rescue => e
      raise e
    else
      if response != nil 
        code = response.code
        if code && code.to_s.start_with?('2')
          doc = Nokogiri::HTML(response.to_s)
          MarketingTool::CheckTitleTagWorker.perform_async(audit_id, doc.title, keyword)
          MarketingTool::CheckUrlWorker.perform_async(audit_id, get_handle_from_url(url), keyword, keyword)
          MarketingTool::CheckHeaderWorker.perform_async(audit_id, doc.search('h1').map(&:text), keyword)
          MarketingTool::CheckMetaDescriptionTagWorker.perform_async(audit_id, doc.at("meta[name='description']")['content'], keyword) if doc.at("meta[name='description']")
          MarketingTool::CheckKeywordsFirst100Worker.perform_async(audit_id, get_all_image_tags(doc), keyword)
          ...
        end
      end
    end
  end

  def get_handle_from_url(url)
    url.sub(/(\/)+$/,'').match(/\/([^\/]+)\/?$/)[1].gsub("-", " ")
  end

  def get_all_image_tags(doc)
    image_tags = []
    doc.css('img').each do |img_node|
      img_attributes = img_node.attributes.values # list of image attributes
      img_attributes.each do |attr|
        image_tags << [attr.value]
      end
    end
    image_tags
  end
end

The service object above scrapes the page, and gets all the important SEO details of the page. Each SEO test is sent to a different worker. For example, checking if the the page title has a number in it is done by the CheckTitleHasNumberWorker. By separating each task into a different worker, we are ensuring that each worker is small and does only one thing.

class MarketingTool::CheckTitleHasNumberWorker
  include Sidekiq::Worker
  sidekiq_options :queue => 'default'

  def perform(audit_id, title, keyword)
    audit = Audit.find(audit_id)
    title_tag_has_numbers = title.downcase =~ /\d/ ? true : false
    audit.update(title_tag_has_numbers: title_tag_has_numbers)
  end
end

Unit Tests

This completes the whole 5 step user story we wrote in the beginning. But we also need to add Unit tests, to make the Audit model bulletproof - i.e. to make sure that all validations pass before it is saved to the database. I need to test a few validations:

• In a Shopify store, all product URLs have to be under the /products folder
• The User has to enter both the URL and the focus keyword

For creating a unit test, I add a new file boostctr/spec/models/audit_spec.rb

require 'spec_helper'

describe Audit do 
  it "URL must be a Shopofy product page" do 
    correct_audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/products/sofa')
    correct_audit.save
    expect(Audit.count).to eq(1)
    wrong_audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/sofa')
    wrong_audit.save
    expect(Audit.count).to eq(1)
  end
  it "saves itself" do 
    audit = Audit.new(focus_keyword: 'sofas', url: 'exmaple.com/products/sofa')
    audit.save
    expect(Audit.first).to eq(audit)
  end
  it "keyword must not be empty" do 
    audit = Audit.new(url: 'exmaple.com/products/sofa')
    audit.save
    expect(Audit.count).to eq(0)
  end
  it "URL must not be empty" do 
    audit = Audit.new(focus_keyword: 'sofas')
    audit.save
    expect(Audit.count).to eq(0)
  end
end

Conclusion

This post looked at how we can do the following

• Using Test Driven Development to build a new feature
• Creating integration tests to test the flow across multiple controllers
• Adding unit tests to test business logic and validations
• Adding background workers to run two or more tasks simultaneously
• And adding a Service Object for logic that does not fit into a model or a controller.