Slaying the Hydra: Parallel Execution of Test Automation

The Great Constraint

“How long does it take to run the regression?”

“Why does the regression take so long?”

These questions represent the a large constraint with test automation execution. If the suite doesn’t provide feedback in an appropriate time frame it impacts both the decision-making ability of our stakeholders and our ability to maintain the quality of the tests.

To put things more simply, single threaded execution of automated tests are often too slow to meet the business needs of the application under test.

Modifications can be made to increase the speed of the suite by shaving down the run time of individual scenarios and/or removing unnecessary scenarios. Ultimately, we end up in the same place where the regression is just simply too slow.

Thomas has a great blog post about the common failure points of automation implementation. I would strongly suggest reading this as it is a good starting point to understanding automation challenges and provides a foundation for where we are going.

The Real Question

The real questions posed back to your team should be “What is the appropriate execution time of the regression?”

The answer “as fast as possible” is not acceptable. Increased speed means increased resources and planning that will cost the team both time and money. Getting an accurate answer to this question becomes the basis of our research on the cost of the solution.

Dependencies

For the sake of argument let’s say you have a specific execution time for the feedback loop. If the current infrastructure does not support a feedback loop that short, the team should consider:

Are the individual test scenarios robust and independent enough to handle being executed in parallel?

If the answer here is no for any reason this work should be included as part of the effort. In an ideal world a test scenario should execute completely independent of other scenarios, meaning it should not impact or be impacted by other scenarios (commonly called “atomic” tests).

Does the team have the drive to provide time and resources to this effort?

The resources could be everything from additional physical or virtual machines to time with other developers/team members to help build the solution. If the team is not able to free up team members to work on this solution then it’s a wasted effort. Additionally, ensure that there are motivated capable individuals on the team that can contribute.

Past Solutions

I’ve experienced the speed of the regression impacting the teams I have supported in my career. The solutions below are ones that I have implemented in the past that I would not recommend:

In Cucumber, tagging is a process done at the feature or scenario level to group scenarios into locatable and executable sections. This process is helpful for smoke tests, regressions or functional areas that can the be executed or excluded at run-time. I would not recommend splitting regression for parallel execution utilizing static tags because tagging should be used to signify the logical groups a test belongs within and nothing more.

An extension of the above would be running different logical groups at different times. For example: running checkout scenarios on Tuesday and search scenarios on Wednesday. The feedback loop for the regression is now multiple days and doesn’t provide rapid feedback which we expect.

Requirements

So far, I have told you what I believe to be the most common constraint in test automation feedback lops, some questions I would ask your team, and some things I would not recommend doing. In this section I am going to go full ten commandments style and lay down the requirements of what we want from our tool.

Our tool should be able to:

• Execute on multiple workstations in parallel in order to increase the efficiency of running the scenarios.
• Utilize a CI/CD tool to allow for orchestration of the process.
• Report back the status of the regression in a meaningful and consumable way to our stakeholders.
• Allow for easy modification where/when required.  

Going forward

With this information in mind the following course is going to be taken as a series of blog posts in order to serve as a guide in fulfilling these requirements:

Part 1 – Orchestration overview and setting a clean slate – In this section the practical implementation of the orchestration component will be discussed along with the importance of insuring a clean slate.

Part 2 – Run-time state and splitting up the execution – Discussion of what should happen during and immediately before the tests begin running.  

Part 3 – Consolidation of information and reporting – How to collect test result information and report it to the stakeholders.

Part 4 – Modifications and next steps – What potential changes could occur and what are the next steps from here.

Series Navigation

Slaying the Hydra: Orchestration Overview and Setting a Clean Slate >>