Test Automation Toolbox

Characteristics of a Good Test Automation Framework

It’s been a while since I’ve written new blog posts about test automation. Some of the older posts don’t really reflect how I write test automation frameworks for acceptance testing currently so it’s time to write about the way I’ve done it in my the latest projects.

The key aspects of a test framework to me are:

• maintainability
• ability to add new tests quickly easily

The next sections explain these aspects in more detail.

Maintainability

Test automation work is quite labour intensive and it is important to be able to make changes with as little effort as possible. Websites and their requirements are subject to change and having to update many tests when an ID or the class of an HTML element changes is not a good investment of your time.

Another factor in being able to maintain a test framework is readability. The code should follow best practices for the language in question, i.e. for Ruby follow ::RuboCop:: recommendations with possible exceptions as defined by the organisation. Class, method and variable names also go a long way in making the code more understandable. The method names also need to accurately describe what the method does. Having inaccurate method names is a bug waiting to happen as a fellow tester uses the method expecting specific behaviour.

Extensibility

The second item in the list was the ability to add new tests quickly. The main factor for this is a good test framework which provides support for the cases that need to be tested and allows for defining specific values when needed. One example related to API testing is to enable to user to write tests with default values for headers for the happy path and override the defaults in a test case where a header should be missing.

A big step towards solving both of these areas is to build a test API around the website or an API that is being tested. I’ve used a similar approach in several of my past projects and found it to be a very maintainable way of writing tests. This is a good and tried pattern and will work for many organisations and with different test/BDD frameworks.

Having a framework that allows for easily adding new tests when a problem is discovered due to a missing test or when new functionality is added enables the team to progress quickly without delays due to adding of tests. Testing should not be a bottleneck in the end of a waterfall type of process but go with the hopefully agile flow of the team’s other work.

Modelling the product under test

When testing websites, most people nowadays use the page object pattern to model the HTML of the application. This reduces the need to directly call low level API’s like Selenium WebDriver or even Capybara when implementing the tests. It also isolates the effect of changes in HTML to the tests. If the ID of an element changes, it will only need to be changed in one place.

The most effective way of reducing the need to directly call WebDriver methods is to use a Gem implementing the page object pattern. The one that I’ve used and recommend is SitePrism. When using SitePrism, there is rarely a need to use the next level of abstraction, i.e. Capybara directly. In my experience, finding HTML elements in test steps leads to brittle and hard to maintain tests.

Application Model

While page object pattern is something that is mentioned often related to website testing, the same concept can be applied to API testing. Each service can be encapsulated in a test API making HTTP calls to the API easier and repeatable. One doesn’t need to define the URL or the headers in a test unless the test is about calling the API with wrong or missing header values.

As with any code, test code should be structured in a logical way. A website can easily have tens of pages and building a page object file for each one leads to lots of files that need to organised in a good way. What I typically do is to combine the page objects and other code used for testing a site to a site object library. In addition to having methods that have directly to do with taking actions on an element, like clicking a link, one can implement higher level actions, for example, login, which combines several steps.

Reuse of the Application Model

An additional benefit of encapsulating the site/API in a test API is that the actions can easily be reused for another purpose. If there is a need to monitor the application the groundwork has already been laid and the same test API can be used to construct calls in a monitoring system, usually with very little code. When the test API is build into a Gem, other people in the organisation can import it into their projects and they do not need to figure out the intricacies of your API or the ID’s of HTML elements on pages when the application is a website.

Another advantage is that should your organisation want to switch to a different test framework, the logic for interacting with the application can be reused.

Controlling the Browser

For test automation work using Ruby, Capybara is the most commonly used library for controlling the browser. which is a given in most projects. The github repo has good documentation on using Capybara and there’s also a book available on Amazon, Application Testing with Capybara.

Capybara takes care of most of the mundane things that are needed when controlling the browser. It allows for finding elements on the page as well as taking actions on the elements, for example:

• clicking on a link
• ticking a checkbox
• selecting an item from a select list

with methods named appropriately, i.e. click, check and select.

As with other tools that automate the browser, the process consists usually of two steps:

• finding an element
• taking an action with the element

The default way of finding elements with Capybara is using the CSS selectors, which works really well. For the cases where the front-end developers aren’t paying attention to semantic HTML, there is also support for XPath.

Page Object Model

Most test automation projects at present use the page object model. In my opinion, the best library for implementing the page object pattern is SitePrism. The gem works seamlessly with Capybara and does what is expected from a page object model gem.

It allows for defining pages, their URL’s and also elements located on each page. SitePrism generates a set of useful methods for accessing the elements and for waiting for the element to appear / disappear. Elements can be located on the page with the typical finder methods, i.e. ID, CSS, tag name, XPath etc.

SitePrism also has a concept called a section, which is a section of a page that can be included in several pages. One could use sections for defining the page header or a footer, a login dialog or other common parts of websites. Sections can be used in a similar way to elements and have similar methods available as elements. Sections can also contain other section or elements for building hierarchies of elements. Once a section is defined, it can be anchored onto a page by setting the root element of the section.

Also, there is nothing stopping you from defining other methods in the page models. While SitePrism methods are a good starting point for test automation, there is often a need have specific methods for complicated operations that are not only about waiting / acting on a single element. One could implement higher level actions using several elements or methods that require more login in the page model.

One cool thing about using SitePrism is how easy it is to combine the usual Capybara DSL to the actions. For example

element :top_menu, '#top-menu'

generates (among other methods) top_menu

Capybara DSL can then be used by chaining it to the SitePrism element, i.e.

top_menu.find('a.active')

This statement would find the first active link (assuming the development team uses the common pattern of setting the CSS class to active for an active link). This specific element could be more easily found directly by defining the SitePrism elements as

element :active_link, '#top-menu a.active'

but either way of accessing the element might be preferable depending on the use case.

Summary

As a summary, building a test framework with the tools mentioned in this post, i.e. Ruby, SitePrism, RSpec / Cucumber is a good starting point for most projects. I have a simple example repo available on Github, which uses these concepts although it might not be up to date with the site as it was implemented 6 months ago.