Ruby on Rails System testing (E2E testing) using Natural Language with OpenAI API

Background

End-to-end (E2E) testing can be challenging to maintain.

For instance, even a simple login screen can have various variations. As a service evolves, there often arises a need to upgrade a straightforward login screen to a more advanced one with enhanced functionality.

As shown in the figure below, what appears to humans as the same "screen for entering ID and password" can be vastly different for machines. It is not easy for a system to identify both interfaces as login screens and test them with the same script.

What Is the Problem?

E2E testing is not a one-and-done task; it needs to adapt to changes in the user interface (UI). Otherwise, the cost of updating the tests to keep up with UI changes can outweigh the benefits of ensuring the functionality specifications through testing.

In current common practices for E2E testing, UI elements such as buttons and text boxes are selected using CSS selectors or ARIA (Accessibility). These elements are then interacted with, such as clicking. However, even for the same login screen, relying solely on mechanical selection via CSS selectors makes it difficult to accommodate UI changes.

Additionally, when humans conduct tests, they attempt to recover from operational errors along the way. On the other hand, mechanical testing lacks such adaptability. For instance, if login information is mistakenly omitted and a submission results in an error, the automated test engine may still try to proceed with operations as if the login were successful. This is a common cause of what is known as a "flaky test."

Writing Ruby on Rails System Tests in Natural Language!

With the remarkable advancements in generative AI, it is now conceivable to devise a method that allows tests to be written in natural language and executed directly. Inspired by this idea, I have developed a prototype and published it as a Rubygem.

https://github.com/YusukeIwaki/charai

before do
  Capybara.current_driver    = :charai
  Capybara.javascript_driver = :charai

  page.driver.additional_instruction = <<~MARKDOWN
  * Our website consists of 2-pain layout. The core function of this web service is helping users to find a good job offer.
  * On the left-pain, we can filter job offers by text search queries and conditions by checkboxes. On the center-pain, job offers are shown. It contains 30 items, and the second item can be a sponsored advertisement item.
  * The center-pain is scrollable while the left-pain is not. So please keep it mind that we have to put our mouse cursor to the center-pain for scrolling.
  MARKDOWN
end

it 'should work' do
  page.driver << <<~MARKDOWN
  * Browse to the job offer list view. Then input "Ruby on Rails".
  * After inputing the search query, submit it and check if the job offers shown in the center-pain is really related to Ruby on Rails.
  * If most of the search result is irrelevant to Ruby on Rails, please mark this testcase as "Assertion failure".
  MARKDOWN
end

Directions

Traditionally, natural language commands are converted into CSS selectors for execution. While this approach can adapt UI changes, it does not resolve the issue of recovering from operational errors.

To overcome this limitation, this library employs a unique approach: the AI autonomously observes the screen and generates Ruby code for performing the required actions. The generated Ruby code is executed, and the resulting screenshots are feedback into the AI for analysis. Based on this analysis, the next steps are determined and executed. This process essentially models how humans perform manual testing.

By adopting this method, even if an click opens an unintended page, the AI can analyze the screenshot of the unexpected page, issue instructions to return to the previous page, and continue testing from there. This approach closely mimics the way humans recover from errors during testing, ensuring robustness and adaptability.

How to implement it?

To implement this approach in Ruby on Rails, creating a custom Capybara driver is the simplest method.

There is plenty of information available via a quick Google search on how to use a Capybara driver. However, guidance on creating a Capybara driver is scarce. You would need to study the Capybara source code or refer to the implementation of existing libraries like Cuprite, and then replicate their techniques to build your own driver.

Traditional Capybara drivers offer a wide range of methods under the Capybara DSL, such as click, find, and set. However, the driver we aim to create simplifies this dramatically by providing only three methods:

• visit Method: This is used to navigate to the initial web page.
• Precondition Method: This method communicates the preconditions for the test.
• Test Content Method: This method conveys the actual test instructions to be executed.

By limiting the functionality to these three essential methods, this new driver focuses on allowing AI-driven natural language instructions to handle dynamic testing processes. This approach aligns closely with the goal of creating adaptive and resilient E2E tests.

More detailed instruction to implement the driver is introduced in this movie and deck (Sorry in Japanese!!)

Does it really work??

At this moment, it works... to some extent. The current accuracy is still low, and it is not yet production-ready. However, this approach holds the potential to significantly transform the development experience of writing and running system tests in Rails.

The prototype showcases how leveraging AI in this manner could revolutionize E2E testing by making it more adaptable and human-like. While there is much room for improvement, the foundation is promising, and further refinements could lead to practical, impactful solutions in the future.

So please try it, and give me your feedback! :)

YusukeIwaki / charai

Charai(Chat + Ruby + AI) driver for Capybara. Prototype impl for Kaigi on Rails 2024 presentation.

Charai

Chat + Ruby + AI = Charai

Setup

Add gem 'charai' into your project's Gemfile, and then bundle install

Also, this gem requires Firefox Developer Edition to be installed on the location below:

• /Applications/Firefox Developer Edition.app (macOS)
• /usr/bin/firefox-devedition (Linux)

Configuration

Configure your Capybara driver like below.

Capybara.register_driver :charai do |app|
  Charai::Driver.new(app, openai_configuration: config)
end

Capybara.register_driver :charai_headless do |app|
  Charai::Driver.new(app, openai_configuration: config, headless: true)
end

Please note that this driver required OpenAI service.

OpenAI

config = Charai::OpenaiConfiguration.new(
  model: 'gpt-4o',
  api_key: 'sk-xxxxxxxxxxx'
)

Azure OpenAI (Recommended)

config = Charai::AzureOpenaiConfiguration.new(
  endpoint_url: 'https://YOUR-APP.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-05-01-preview',
  api_key: 'aabbcc00112233445566'
)

Usage

Since this driver works with the OpenAI service, we can easily describe E2E test like below :)

before do
  Capybara.current_driver    = :charai
  Capybara.javascript_driver = :charai
  page.driver.additional_instruction = <<~MARKDOWN
  * このページは、3ペイン構造です。ユーザが仕事を探すためのページです。
  * 左ペインには仕事の絞り込みができるフィルター、中央ペインが仕事（求人）の一覧で、30件ずつ表示されます。
  * 左ペインにマウスを置いてスクロールしても、中央ペインはスクロールされません。一覧をスクロールしたいときには、中央ペインの座標を確認し、その中央にマウスを置いてスクロールしてください。
  * 右ペインは、広告エリアです。検索条件に応じた広告が表示されます。
  MARKDOWN
end

it 'should work' do
  page.driver << <<~MARKDOWN
  *

…

View on GitHub