Slaying the Hydra: Run-Time State and Splitting Up the Execution

In this third post of the blog series on parallel test execution, I explain how to execute distributed parallel test automation. The previous entry can be found here.

As discussed previously, The running stage (see below) within the pipeline context is set to execute three builds of the test_runner freestyle job in parallel. Each build is receiving the following parameters:

• browser – either equal to ‘ie’ or ‘chrome’
• total_number_of_builds – equal to ‘3’
• build_number – equal to ‘1’, ‘2’ or ‘3’

Freestyle Job Overview

In the following sections, I explain what freestyle components need utilized when constructing the test_runner job in Jenkins.

Parameters

As seen from the image above, parameters are being passed from the pipeline job into the freestyle job. We will update the freestyle job to be parameterized. This selection is made when configuring the Jenkins job (see below).

Next the freestyle job is configured with these parameter names:

• browser –  the value received from the pipeline parameter value.
• total_number_of_builds –  the value received from the pipeline parameter value.
• build_number – the value received from the pipeline parameter value.
• workspace_location – to show a different way of doing things, we can see from the image above that I did not pass a value for workspace location in the pipeline. When I configured the parameter (below), I set a default value in the freestyle job. This default value will be linked to the workspace_location parameter now unless I otherwise specify.

Node Selection

In this section we restrict where this build can execute to only machines associated with the @local tag only. This setting is located in the Manage Jenkins > Manage Nodes section of Jenkins. It provides us the ability to ensure we are not utilizing nodes that are otherwise utilized or not configured to run the cucumber tests in the steps below.

Version Control

In the Source Code Management section, we specify what testing suite to retrieve via version control and utilize for this effort, which will pull the suite down within the workspace. The “clean before checkout” additional behavior (Jenkins functionality) will remove any files in the workspace that are not in the Git repo before pulling the suite down. This allows for a clean slate for every execution.

Splitting Code

class Splitter
  def total_builds
    ENV['total_number_of_builds'].to_i
  end

  def build_number
    ENV['build_number'].to_i
  end

  def main_run
    scenarios = feature_iterator
    splits = job_splitter(scenarios)
    assignment = job_assigner(splits)
    feature_mod_iterator(assignment, 'features', true)
  end

  def feature_mod_iterator(split_assignment, current_location = 'features', assign = true)
    array = []
    split_assignment.each do |value|
      mod_value = value.gsub('@regression', '@split_builds')
      regex = /#{value}$/
      files = return_all_files(current_location, '*', 'feature')
      files.each do |file|
        output = File.open(file, 'r', &:read)
        modified = output.gsub(regex, mod_value)
        if assign
          File.open(file, 'w+') { |f| f.print(modified) }
        else
          array.push(modified)
        end
      end
    end
    array
  end

  def feature_iterator(current_location = 'features')
    files = return_all_files(current_location, '*', 'feature')
    array = []
    files.each do |file|
      array.push(return_all_gherkin_scenarios(file))
    end
    array.flatten
  end

  def return_all_gherkin_scenarios(file)
    output = File.open(file, 'r', &:read)
    output.scan(/(@regression.*\n. (Scenario:|Scenario Outline:)?.*)/).map { |value| value[0] }
  end

  def return_all_files(current_location, filter = '*', file_type = '*')
    Dir.glob("#{current_location}/**/#{filter}.#{file_type}")
  end

  def job_splitter(scenarios)
    split = scenarios.length.to_i / total_builds.to_i

    container = []
    total_builds.times { container.push([]) }
    mod_scenarios = scenarios.clone

    total_builds.times do |index|
      container[index].push(mod_scenarios[0..(split - 1)])
      container[index].flatten!

      (0..(split - 1)).to_a.length.times do
        mod_scenarios.delete_at(0)
      end
    end

    mod_scenarios.each_with_index do |value, index|
      container[index].push(value)
    end
    container
  end

  def job_assigner(scenarios)
    scenarios[(build_number.to_i - 1)]
  end
end

one = Splitter.new
one.main_run

At a high level, the code block above is creating an array of arrays that split up the regression tests evenly between the number of executors. The build_number value is utilized to access the corresponding index value of the array. All of the tests in that location are re-tagged from @regression to @split_builds locally on the workspace that houses the Ruby/Cucumber code pulled down from version control.

You would have to change the @regression tag to whatever you are utilizing to tag your tests as regression on your team.

The cool thing is that this will run on each of the three workspaces and re-tag a unique subset of tests. Because the total_builds value is the same for all the jobs kicked off, it will create the same nested array structure on every workspace. The difference between workspaces comes about because of the build_number parameter that chooses which subset of tests to re-tag.

Running the Split Code

We should house the code above within our testing framework in version control.  Within the Build section of Jenkins we then create a windows batch command. Next we set the environment variables that the code utilizes total_builds and build_number as being equal to the parameters set within the freestyle job. We can now run the ruby command passing the path to the .rb file that houses the code within the workspace (in reference to the code above).

Running the Tests

We set up another windows batch command to set environment variables for browser and or_tags, and in this instance, we kick off the tests utilizing a rake task. Cucumber Rake is a useful tool, but we could just as easily run a Cucumber command.

The important thing is that we are passing what will be the tag modified locally on each workspace(split_builds) to run only the tests assigned to that workspace. Additionally, we passed the browser variable set within the pipeline and passed to the freestyle job.

Storing Results

In our last batch command, we are extracting the json test results file and storing it on the workspace_location as a json file named with the build_number value (either 1, 2, or 3). This workspace location is the same as what we utilized in the clearing stage and what will be utilized in the consolidation stage.  

Review and Next Steps

To review, in this post, we figured out how to build the freestyle job that is responsible for splitting, executing, and storing the results of our tests.

In the next post, we discuss how to consolidate the information from the freestyle job builds into a concise cucumber report.

Series Navigation