From the Pipeline v20.0

This entry is part 20 of 34 in the series From the Pipeline

The following will be a regular feature where we share articles, podcasts, and webinars of interest from the web.

Cucumber Reports

The creators of Cucumber have released a free, cloud-based service for sharing execution reports for scenarios. Both the Java and Ruby implementations of Cucumber have the report functionality built-in with simple commands or environment variables. If enabled, the console output links to an online execution report. The results will be available for a 24-hour period before being automatically deleted. In the future, the execution reports can be linked to a GitHub repo and will no longer be scheduled for auto-deletion if claimed.

Separating Automation Tooling from Automation Strategy

“When people do not have good luck with automation, it is hardly ever because of the tool being used, but almost always because of the wrong automation strategy, wrong expectations, and wrong adoption of automation. Automation tools only answer the “how” of automation, while having an automation strategy gives answers to who, where, when, what, and why. Here’s why it’s so important to have a test automation strategy.”

Ten More Commandments on Automation

Paul Grizzaffi provides an excellent overview of the common pitfalls organizations run into with test automation. His advice brings the same concepts we would expect from development code and applies is to test automation code because we should treat them the same way. Check to see if any of your test automation code breaks a commandment!

The Technical Debt Quadrant

The article a breakdown of the four types of technical debt as originally described by Martin Fowler. Sometimes we make deliberate mistakes either to simply push the product to production or because we don’t properly consider design. Other times our own inexperience with a technology means we make mistakes. How we react to that accrued technical debt determines if we are reckless or prudent about identifying the problem to help ensure we don’t repeat the mistakes again.

Webinar: Add Static Code Analysis to Your CI/CD Pipelines

Perforce recently hosted a webinar on static code analysis in CI/CD pipelines. Plenty of excellent lessons to learn about adding quality to deployment pipelines. “With the amount of software being installed into devices across all industries, it has become essential that the embedded code is safe and secure, reliable, and high quality. However, ensuring that the embedded code meets these standards and is delivered in a timely manner can be a daunting and time-consuming challenge. For that reason, it is essential that developers pair static code analysis with efficient software development practices, such as CI/CD pipelines. View this webinar and learn how to add static analysis to your DevOps process.”

From the Pipeline v19.0

This entry is part 19 of 34 in the series From the Pipeline

The following will be a regular feature where we share articles, podcasts, and webinars of interest from the web.

Mozilla: The Greatest Tech Company Left Behind

A sad story about layoffs at Mozilla with cuts happening in the developer tools division. The article contains a brief history of Mozilla, including their contributions to web. Mozilla actually pulls most of it’s revenue from Google, a competitor to the browser, to make the search engine the default for the browser. Probably one of their biggest failures is not having a presence in the mobile web market, which is dominated by Chrome and Safari.

Introducing BDD to Your Team: How Does it Affect Your Role as a Tester?

Bas Djikstra provides a roadmap for introducing your team to BDD. The first fundamental mindset change is involving testers from the start of development of every new feature. Bas recommends involving testers in refinements sessions upfront. Specifications should be discussed upfront with the business partners and work with the developers should be collaborative during actual feature development. The last piece of BDD is automating those specifications to align with end-user behavior.

How Continuous Testing is Done in DevOps

“DevOps does speed up your processes and make them more efficient, but companies must focus on quality as well as speed. QA should not live outside the DevOps environment; it should be a fundamental part. If your DevOps ambitions have started with only the development and operations teams, it’s not too late to loop in testing. You must integrate QA into the lifecycle in order to truly achieve DevOps benefits.”

Breakpoint Highlights: Test Automation

BrowserStack’s first virtual conference recorded all their sessions and have made them available to everyone. It’s a good alternative to the face-to-face events many of us have missed since COVID. At least with these virtual events we can watch them at our leisure and not have to worry about choosing between two talks at the same slot. Details: “We hosted BrowserStack’s first-ever virtual summit, Breakpoint 2020 last week. We saw over 10,000 registrations and 2500+ attendees from 155 countries, 18 speakers, and 2 cats! Speakers from Twitter, Trivago, Selenium, Robot Framework, and more talked about agile testing, quality at speed, decentralized testing, release reconstruction, and testing best practices.”

A Guide from Key DevOps Experts

Eran Kinsbruner of Perforce is releasing another anthology book written by a group of DevOps professionals on Artificial Intelligence, Machine Learning, and Robotic Process Automation. The book covers: “The fundamentals of AI and ML in software development and testing, including the basics for testing AI-based applications, classifications of AI/ML, and defects tied to AI/ML. Practical advice and recommendations for using AI/ML-based solutions within software development activities, like visual AI test automation, AI in test management, testing conversational AI apps, RPA benefits, and API testing. More advanced and future-focused angles of AI and ML with projections and unique use cases, including AI and ML in logs observability, AIOps, how to maintain AI/ML test automation, and test impact analysis with AI.”

Book Club: Behavior Driven Development – Discovery (Chapter 5)

This entry is part 5 of 5 in the series BDD Discovery

The following is a chapter summary for “BDD Books – Discovery” by Gaspar Nagy and Seb Rose for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Background on BDD Discovery

DISCOVERY

“Explore behavior using examples

Written by the creator of SpecFlow and the author of The Cucumber for Java Book, this book provides inside information on how to get the most out of the discovery phase of Behaviour Driven Development (BDD). This practical guide demonstrates good collaboration techniques, illustrated by concrete examples.

This book is written for everyone involved in the specification and delivery of software (including product owners, business analysts, developers, and testers). The book starts by explaining the reasons BDD exists in the first place and describes techniques for getting the most out of collaboration between business and delivery team members.

This is the first in the BDD Books series that will guide you through the entire development process, including specific technical practices needed to successfully drive development using collaboratively-authored specifications and living documentation.”

BDD Discovery

Chapter 5

In this chapter, the author’s share their experiences of how to get business representatives fully involved in the BDD approach and what kind of challenges they might face.

“Collaboration means that team members with different roles work together to deliver working software. A team will include both the business (product owners, project sponsors, business analysts, etc.) and technical personnel (such as developers, testers, UX experts and operation staff).”

Chapter 5 – How to Get Business Involved

5.1 Learn From Your Peers

The key benefits typically attributed to BDD are:

  • Reduced cycle time
  • Reduced rework/rejections
  • Reduced number of production issues
  • Keeping the implementation costs of new features under control
Cucumber is one option for practicing BDD

5.2 The One Where Business is Not Involved

Challenges in getting the business to adopt BDD include:

  • Have a default resistance against new methodologies
  • See this approach as an overhead
  • Be afraid of learning and infrastructure costs
  • Have formal or legal distance from the team (e.g. the business Representative is a part of the customer, not the vendor)
  • See BDD as a testing or a development technique

BDD should not be a developer-only or tester-only activity. Without business, consider the following options: (1) the situation is a temporary step towards full collaboration, (2) envision the scenarios as business examples with the business explaining the story, (3) focus the detailed requirement discussions as examples, and (4) present the results & methods to the business regularly.

“Without the business involved, unless the team is very business-focused and disciplined, the scenarios become technical, data-driven and dry. This removes the possibility that the scenarios will provide a constructive feedback loop between the business and delivery team – so the scenarios become an overhead.”

Chapter 5 – The One Where Business is Not Involved

5.3 BDD is for Solving Problems

BDD has no value unless it is helping solve problems, improve development efficiency, and produce better results. In order to get business engaged, pain points (or opportunities) must be found that exist in the current process.

Product Owner is Overloaded

BDD can help reduce the Product Owner’s workload. BDD improves communication efficiency by focusing on early, direct communication with the business, ensuring fewer misunderstandings and less rework.

BDD product owner activities:

  • Participate in requirement workshops (e.g. Example Mapping)
  • Review the scenarios
  • Give feedback about the implemented application

Comparison to classic product owner activities:

  • Writing specifications
  • Presenting the stories to the team
  • Discussing ad-hoc questions during implementation
  • Collecting deviations from the expected result in the Sprint review
  • Managing and prioritizing requirement-related bugs

Production Issues are Common

Production Issues are not only costly to fix, but can result in customer dissatisfaction and mistrust. BDD helps prevent production issues being introduced during requirement analysis and detailed specification.

The resulting reduction in production issues is the most easily measurable benefit of applying the BDD approach.

Challenge to Get the Product to a Deliverable State

There is and underestimated gap between implementing a feature and bringing it to a releasable state. Side tasks include: creating deployment packages, updating user documentation or verifying performance requirements.

To ensure that the application can be reliably released at short notice, the team should follow the continuous delivery (CD) approach. Continuous Delivery requires a team to work on the product in short cycles, where each cycle contains all the necessary steps to make the application deliverable. A high level of automation is required to perform the necessary verification and packaging steps.

  • NOTE: The process that automatically builds, tests and packages each product increment as a potential deliverable is called continuous delivery. The process that also automatically releases each deliverable to end users is called continuous deployment.

To move towards continuous delivery, stories should be sliced up into very small chunks. Sometimes teams adopt an approach called horizontal slicing, where tasks are defined along the different layers of the application. Vertical slicing is a functional breakdown where each scenario represents a portion of the expected behavior and makes sense on its own.

Customers are Not Engaged with the Product

Customers who do not like or do not use the product often indicate a development process that does not focus on value. BDD encourages the team to focus on realistic examples of application behavior, so the product evolves according to the users’ viewpoint.

The collaboration techniques used by BDD can help UX expert involvement earlier.

Deadlines are Often Missed

Deadlines are set based upon an estimate of how long to deliver the set of required features (the scope), which can be missed. The key to the successful management of deadlines is to allow more flexibility in the scope: “Try to implement as much value as we can for the given deadline.”

BDD will not tell you how to cut the scope, but it does provide a framework for enabling functional cuts, even within a user story. Scenarios provide a functional breakdown of the story, and allow the reduction of scope by identifying and deferring lower priority behaviors.

Difficult to Get an Overview of Progress

Having a good overview of progress is essential to good project management. Delivered scenarios are an excellent indicator of progress.

The traditional approach of calculating an arbitrary percentage of work completed is not as effective.

Introduction of New Features Causes Side-Effects

The success of a great new feature in the application can easily be overshadowed by a bug that gets introduced at the same time. As the code-base grows the chance of a change producing unwanted side-effects increases.

Adopting BDD encourages good automated test coverage, where automated tests and scenarios are tightly connected to the requirements.

Once a change causes a failing test, it is easy to diagnose whether this problem is related to contradictory requirements or defects in the code. Automation test coverage acts as a safety net.

Not Being Able to Quickly React to Market Changes

In order match competitors in a changing market, the product development model must keep the cost of implementing new features low, while maintaining the quality of the application.

Experiments are valuable at the code-level and specification-level. The behavior framework for the product can be used by the product owner or the business analyst to investigate how the system behaves with different parameters or different user workflows.

Bug versus Feature Debate

Developers and testers build up their own solutions independently: the developers write the application code and the testers write the test cases. The redundancy might result in conflicts at the point when the two parallel worlds meet: when the test cases are used to test the application.

The rework and frustration can be avoided by reducing developers and testers to work on understanding the details of the requirements together.

Changes Regularly Requested at Sprint or Feature Reviews

The iterative development concept of agile is based on the fact that feedback for a feature occurs when business stakeholders actually see it demonstrated. If changes are discovered regularly during the review, then the details in the requirements are not being described properly.

BDD supports the detailed requirement analysis in many ways:

  • By focusing on examples, ambiguities can be recognized and cleared up
  • Since scenarios document the specification in a common language, misunderstandings are less likely to occur.

Product Owner Never Reads the Scenarios

A common situation is a product owner that never reads the scenarios. Scenarios specify how the application should behave. Confirm the Product Owner has access to the living documentation, whether it be stored in source control or a project tracking tool.

Integrate the publishing process into the continuous integration pipeline and make the result easily accessible (by adding a link to the project dashboard, for example). Improve how scenarios are tagged. Take time to regularly reflect on how the living documentation is being used by the business and what changes might make it more useful.

5.4 What We Just Learned

BDD depends on collaboration. Getting the business involved is key to success. The authors presented some tips based on experience to adopting BDD within organizations. There are two follow-up books to BDD Discovery: Formulation and Automation.

Outside Resources for Further Reading

Books:

  • Adzic, Gojko. Specification by Example: How Successful Teams Deliver the Right Software. Shelter Island, NY: Manning, 2011.
  • Nagy, Gáspár, and Seb Rose. The BDD Books: Formulation. In preparation. http://bddbooks.com/formulation.
  • Nagy, Gáspár, and Seb Rose. The BDD Books: Automation with SpecFlow. In preparation. http://bddbooks.com/specflow.
  • Adzic, Gojko. Bridging the Communication Gap: Specification by Example and Agile Acceptance Testing. London: Neuri, 2009.
  • Wynne, Matt, and Aslak Hellesøy. The Cucumber Book: Behaviour-driven Development for Testers and Developers. Dallas, TX: Pragmatic Bookshelf, 2012.
  • Freeman, Steve, and Nat Pryce. Growing Object-oriented Software Guided by Tests. S.l.: Addison-Wesley Professional, 2009.
  • Crispin, Lisa, and Janet Gregory. Agile Testing: A Practical Guide for Testers and Agile Teams. Upper Saddle River, NJ: Addison-Wesley, 2009.
  • Hendrickson, Elisabeth, Ward Cunningham, and Jacquelyn Carter. Explore It! Reduce Risk and Increase Confidence with Exploratory Testing. Dallas: Pragmatic Bookshelf, 2013.
  • Kniberg, Henrik, Mattias Skarin, Mary Poppendieck, and David Anderson. Kanban and Scrum: Making the Most of Both. United States: InfoQ, 2010. Print/Web. https://www.infoq.com/minibooks/kanban-scrum-minibook.

Publications:

  • J.-C. Chen and S.-J. Huang, “An empirical analysis of the impact of software development problem factors on software maintainability,” Journal of Systems and Software, vol. 82, pp. 981-992 June 2009. Dallas: Pragmatic Bookshelf, 2013.

Online:

Book Glossary

Behavior Driven Development (BDD) is an agile approach to delivering software.

Continuous Delivery requires a team to work on the product in short cycles, where each cycle contains all the necessary steps to make the application deliverable. A high level of automation is required to perform the necessary verification and packaging steps.

Continuous Deployment is the process that automatically releases each deliverable to end users.

Example:

  • Context: describes the state of the system before the action takes place.
  • Action: the event that causes the behavior to take place. It might be some action by a user, the system, a scheduled job, or any other stimulus that can cause the system to react.
  • Outcome: description of the state of the system after the user or system behavior has taken place. It should contain enough detail to measure the behavior against expectations.

Horizontal Slicing are tasks or user stories defined along the different layers of the application.

Living Documentation is a form of documentation that represents the current state of the application, which is updated in real-time.

Kanban optimizes end-to-end delivery (“lead time”) by visualizing the workflow and limiting work in progress (WIP) to be able to detect and fix bottlenecks more easily.

Persona in user-centered design and marketing is a fictional character created to represent a user type that might use a site, brand, or product in a similar way.

Requirement Workshop is an activity where the team meets regularly, the purpose is to explore a user story, the scope of the story is discussed, the story is illustrated with examples, and questions are documented that no one in the workshop can answer.

Structured Conversation is a facilitated exchange of ideas that conform to a predefined form.

Test-Driven Development (TDD) helps speed up the feedback loop by demanding that teams write automated tests before they write the code.

Vertical Slicing is a functional breakdown where each scenario represents a portion of the expected behavior and makes sense on its own.

Book Club: Behavior Driven Development – Discovery (Chapter 4)

This entry is part 4 of 5 in the series BDD Discovery

The following is a chapter summary for “BDD Books – Discovery” by Gaspar Nagy and Seb Rose for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Background on BDD Discovery

DISCOVERY

“Explore behavior using examples

Written by the creator of SpecFlow and the author of The Cucumber for Java Book, this book provides inside information on how to get the most out of the discovery phase of Behaviour Driven Development (BDD). This practical guide demonstrates good collaboration techniques, illustrated by concrete examples.

This book is written for everyone involved in the specification and delivery of software (including product owners, business analysts, developers, and testers). The book starts by explaining the reasons BDD exists in the first place and describes techniques for getting the most out of collaboration between business and delivery team members.

This is the first in the BDD Books series that will guide you through the entire development process, including specific technical practices needed to successfully drive development using collaboratively-authored specifications and living documentation.”

BDD Discovery

Chapter 4

The prior chapters emphasized the importance of collaboration for BDD to be successful. This chapter demonstrates how BDD can be integrated into the Software Development Lifecycle (SDLC) by describing the approach in more detail. The chapter will answer the following questions:

  • “Who should write the scenarios?”
  • “Who do we need to participate in the requirement workshops?”
  • “Should the testers or the developers automate the scenarios?”

4.1 The BDD Approach

Figure 21 is a simplified approach of BDD. In practice, BDD is more complex and requires several connected activities to be successful.

For each step in the process, please refer to Figure 22.

#1 Pick a User Story

  • What: Requirements elicitation and prioritization.
  • When: Before the requirement workshop – preferably at least a day in advance.
  • Who: Product owner, business analyst, customer, key stakeholders
  • Outcome: A candidate user story, scoped with relevant business rules (acceptance criteria). There should be enough detail to allow the development team to explore the scope of the story using concrete examples.

#2 Requirement Workshop

  • What: Discuss the user story by focusing on rules and examples.
  • When: Regularly throughout the project – we recommend running short workshops several times a week.
  • Who: All roles must be represented at this meeting (at least the three amigos), but multiple representatives of each role can attend. Each role representative has their own responsibility and focus.
  • Outcome: The candidate story is refined, and is frequently decomposed into smaller stories, each accompanied by a set of rules and examples. The examples should be captured in as light a format as possible, and should not be written using Given/When/Then.

#3 Formulate

  • What: Formulate the examples into scenarios.
  • When: Before the implementation of the story starts. Best done as the first task of development, but sometimes done as a separate workshop where all scenarios scheduled for the iteration are formulated.
  • Who: In the beginning, when the language and style we use to phrase the scenarios is still being established, it is recommended to do it with the entire team. Later, it can be efficiently done by a pair: a developer (or someone who is responsible for the automation) and a tester (or someone who is responsible for quality) as long as their output is actively reviewed by a business representative.
  • Outcome: Scenarios added to source control. The language of the scenarios is business readable and consistent.

#4 Review

  • What: Review the scenarios to ensure they correctly describe expected business behavior.
  • When: Whenever a scenario is written or modified.
  • Who: Product owner (and maybe business analyst, customer, key stakeholders).
  • Outcome: Confidence that the development team have correctly understood the requirements of the business; the behavior is expressed using appropriate terminology.

#5 Automate

  • What: Automate the scenarios by writing test automation code.
  • When: Automate scenarios before starting the implementation, following a test driven approach. This way the implementation can be “driven” by the scenarios, so the application will be designed for testability and the development team will have greater focus on real outcomes.
  • Who: Developers or test automation experts. When doing pair programming, pairing a developer with a tester works well.
  • Outcome: Set of (failing) automated scenarios that can be executed both locally and in continuous integration/delivery/deployment environments.

#6 Implement

  • What: Implement the application code to make the automated scenarios pass. Completed through TDD cycle of Red/Green/Refactor.
  • When: Implementation starts as soon as the first scenario has been automated, so the implementation is being driven by a failing scenario.
  • Who: Developers.
  • Outcome: A working application that behaves as specified. This can be verified automatically.

#7 Supplementary Test

  • What: Perform manual and other forms of testing, described in test strategy document (e.g., scripted, manual testing, exploratory testing, performance, and security).
  • When: As soon as enough is known about the story to begin testing. Scenarios provide a functional breakdown of the user story, so each scenario itself contributes a meaningful part of the application’s behavior. Test preparation can start even earlier.
  • Who: Testers – other team members can help with some aspects of testing, but these activities are usually coordinated by testers.
  • Outcome: High quality working application; the story is done.

#8 Release

  • What: Produce a feature that can be delivered to production. A released product should be used to gather feedback from the users, which can provide input to future development cycles.
  • When: At any time that all tests are passing for a feature.
  • Who: The development team is responsible for producing the releasable features, but DevOps is usually responsible for delivering the finished feature to production.
  • Outcome: An “done” release package.

4.2 BDD in Scrum

In Scrum, work is organized into sprints that are 2-4 week iterations. There are regular workshops that focus on the customer’s requirements throughout each Sprint (backlog refinement, backlog grooming sprint planning preparation, sprint planning) where user stories are prepared.

At the Sprint Planning meeting, the team chooses a few stories to implement in the upcoming sprint.

Once the Sprint is done, the team presents their results in a Sprint Review/Demo and potential process improvements are discussed in the Sprint Retrospective.

Requirement Workshops

In BDD, the team will explore and discover details of stories as part of refinement activities If the Product Owner is available, then they should organize short Example Mapping sessions regularly to replace backlog refinement meetings. Examples will help focus the work on outcomes.

Formulate

Establishing a shared understanding is important, but making the entire team discuss the best way to express a scenario in business language is not efficient.

The best time to formulate the details of scenarios is outside the requirement meetings. Formulation is usually undertaken by a pair, often a developer and a tester.

In some projects, the scenarios have to be formally approved by the product owner or someone else. In these cases, the formulation has to be prioritized and planned in a way that the team does not get blocked.

How scenarios help with the decomposition into tasks:

“In Scrum, stories are usually broken down into tasks. Tasks are work units that represent the design decisions of the team and they are also used to track the daily progress of the team.”

Chapter 4 – BDD in Scrum

Tasks are owned by the development team The tasks are often technical and do not have to be understandable by the Product Owner. However, examples and scenarios should describe the functional breakdown of the story (business readable).

Tasks are aligned to scenarios, which results in several benefits:

  • The team will focus on the expected behavior during the implementation.
  • Feedback about the story implementation earlier.
  • Manual testing can be started before the story is fully implemented.
  • The functional progress provides better transparency and hence increases the trust between the development team and the product owner.

4.3 BDD in Lean/Kanban

Kanban optimizes end-to-end delivery (“lead time”) by visualizing the workflow and limiting work in progress (WIP) to be able to detect and fix bottlenecks more easily.

The activities of the BDD approach are realized in user stories, which are the typical work unit tracked on Kanban boards. Example Mapping sessions can be used to feed the input of the development pipeline whenever capacity allows.

For automation implementation, an additional column would be added to the Kanban board. The tasks formulate, automate, implement, review and supplementary test can be seen as a sub-workflow.

4.4 BDD in Distributed Teams

BDD in distributed teams is carried out in the same way as for co-located teams. Good audio-visual facilities and collaborative online tools are essential. Providing information about the scenario implementation status can lead to improved cooperation between developers, testers, and product owners.

Requirement Workshops with Distributed Teams

Modifications are necessary over the use of index cards in Example Mapping. •A remote requirement workshop will still maintain the same standards:

  • Focus on rules and examples when discussing the story
  • Nominate a facilitator who keeps the meeting going
  • Collect results in a format that all team members can understand, so that They can challenge them if necessary
  • Capture questions that block the discussion and make them visible to everyone
  • Provide an easy way to have an overview of what we have discussed already
  • Produce shared notes

There is no online implementation of Example Mapping; however, Google Sheets can be used as an active workspace during the remote requirement workshop.

4.5 BDD in Fixed Scope Projects

“Projects with fixed price, fixed scope, fixed “everything” are definitely not ideal for agile software development, especially when we’re building highly complex, sophisticated and usable applications in a rapidly changing business environment.”

Chapter 4 – BDD in Fixed Scope Projects

Typical fixed scope projects are provided with a detailed specification document that is supposed to describe all the necessary details to provide a cost estimate.

A large portion of work is about understanding the domain, trying to provide solutions for the requirements that satisfy the customer, stay within the scope of the original, contracted specification document, and stay within budget. BDD can help to provide a frame for domain discovery and for documenting the detailed requirement decisions the team makes.

Use Example Mapping to prepare the individual features for development. When a feature is ready for development, the specification document should be used to build an Example Map with all the rules and illustrative examples. The static specification document can be transformed into a continuously validated living document.

4.6 BDD in Regulated Environments

Regulated Environments focus on the following areas:

  • Completeness and correctness of the specification
  • Coverage provided by the tests and the testing strategy in general
  • Evidence that the testing has been performed for a particular version

BDD applies to a regulated environment in the following ways:

  • The scenarios are selected initially to be illustrative examples of the specifications. The aim is to ensure consistency and shared understanding across the team.
  • The set of scenarios can be extended by further test cases using the same format, to reach the coverage prescribed by the testing strategy.
  • The scenarios are business readable and they bring together the specification and the tests. This duality ensures a higher consistency and linkage between these areas.
  • The scenarios are strictly versioned, together with the application code.
  • BDD tools typically execute scenarios directly and produce a report of the execution result.

Running automated scenarios in this environment will document the expected behavior, describe the tests that verify the behavior, and provide evidence of execution.

4.7 What We Just Learned

BDD is a set of practices that, when combined, can improve the quality of a team’s development process.

BDD integrates with various common development methodologies and team configurations.

With appropriate technology support, BDD can work well for distributed teams, acting as an aid to collaboration.

From the Pipeline v18.0

This entry is part 18 of 34 in the series From the Pipeline

The following will be a regular feature where we share articles, podcasts, and webinars of interest from the web.

Jenkins Documentation Progress

Jenkins joined the Continuous Delivery Foundation in 2019 as one of the first open source projects. Documentation changes are proposed via pull requests to the Jenkins.io GitHub repository. They show an improvement between the proposal of a pull request and a response to the pull request. Jenkins also has managed to merge 85% of pull requests within a 24-hour window. Another recent positive change is the support for plugin documentation stored inside the GitHub repositories. They know have blog posts, documentation, webinars, conversion tools, and progress measurements to assist plugin maintainers and other contributors as they migrate documentation from the Jenkins wiki to the plugin GitHub repositories.

Scientists rename human genes to stop Microsoft Excel from misreading them as dates

27 human genes have been renamed because Microsoft Excel kept misreading their symbols as dates. Excel is commonly used in sciences to store and examine data, with many instances of data becoming corrupted by Excel’s auto-formatting. A study from 2016 showed that nearly one-fifth of all published papers had been affected by Excel errors. The HUGO Gene Nomenclature Committee has recently published new guidelines for gene naming that include symbols that affect data handling and retrieval. This story is an excellent example of software design decisions having unintended consequences for users. In this instance, the users changed their behavior to adapt to the software because the issue was so widespread it was easier to change naming genes than to expect Microsoft to update Excel.

SelectorsHub: The Next Gen XPath, CSS Selectors Tool

Selenium locators are a core part of how the open-source tool finds & interacts with web elements. Selecting a dynamic locator or generic locator often results in flaky tests. Two of the most popular locators are CSS and XPath. The SelectorsHub tool is a browser extension that auto-suggests all the possible combinations of attributes, text, siblings, etc to build selectors. The tool also supports those pesky iFrames and SVG elements. Overall a pretty cool feature released by LambdaTest!

Improving Test Data Collection and Management

“There is much published about the data we generate to assess product quality. There is less discussion about the data testers generate for our own use that can help us improve our work—and even less is said about recommended practices for data collection. Test data collection, management, and use all call for upfront planning and ongoing maintenance. Here’s how testers can improve these practices.”

CukenFest 2020 On-Demand

Many tech events went from in-person to virtual this year. Cukenfest was no different. After joining the SmartBear team, the folks over at Cucumber now have access to more resources to spread the word about BDD. This year CukenFest is online, with all the recordings of the sessions available for viewing. Check it out!

Book Club: Behavior Driven Development – Discovery (Chapter 3)

This entry is part 3 of 5 in the series BDD Discovery

The following is a chapter summary for “BDD Books – Discovery” by Gaspar Nagy and Seb Rose for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Background on BDD Discovery

DISCOVERY

“Explore behavior using examples

Written by the creator of SpecFlow and the author of The Cucumber for Java Book, this book provides inside information on how to get the most out of the discovery phase of Behaviour Driven Development (BDD). This practical guide demonstrates good collaboration techniques, illustrated by concrete examples.

This book is written for everyone involved in the specification and delivery of software (including product owners, business analysts, developers, and testers). The book starts by explaining the reasons BDD exists in the first place and describes techniques for getting the most out of collaboration between business and delivery team members.

This is the first in the BDD Books series that will guide you through the entire development process, including specific technical practices needed to successfully drive development using collaboratively-authored specifications and living documentation.”

BDD Discovery

Chapter 3

This chapter answers the questions: “Are examples really enough to specify a feature? How many examples do we need to specify a feature?”

3.1 How Hard is Concrete?

Outcome: description of the state of the system after the user or system behavior has taken place. It should contain enough detail to measure the behavior against expectations.

Action: the event that causes the behavior to take place. It might be some action by a user, the system, a scheduled job, or any other stimulus that can cause the system to react.

Context: describes the state of the system before the action takes place.

Anatomy of an Example

3.2 Is All That Concrete Essential?

Rules are generic expressions of how the system ought to behave – they each cover lots of possible situations. An example expresses a single situation that illustrates an application of a rule. The purpose of the example is to clarify a rule; each example must be specific and precise.

Each example illustrates a single rule, and should only mention concrete data that is directly related to the behavior being illustrated.

Getting the right level of detail is not the primary concern – inessential detail should be removed later when examples are formatted as scenarios.

3.3 How Many Examples Do We Need?

All possible states cannot be reasonably captured in Examples. A balance must be achieved between Examples and Business Rules.

The team determined that the delivery address can’t be changed once an order has moved beyond the “waiting for pickup” state. The team resolved the misunderstanding by using examples. Please refer to Figure 17. A happy middle ground must be reached in describing enough examples to deliver the feature and writing too many features for a given business rule that preclude efficient delivery of the software.

3.4 Why Stop Now?

“Catching all implementation mistakes will be important, but during the discovery phase our focus is on the requirements: we would like to prevent bugs from ever happening.”

Chapter 3 – Why Stop Now?

The Examples demonstrate the development team understands what they are being asked to do and that the business understands what they’re asking for.

“When we start considering the exhaustive exploration of all possible combinations, we have moved away from understanding the requirements into the realm of software testing. When the examples start to address concerns that the product owner is not interested in, it’s time for the facilitator to bring the discussion back on track.”

Chapter 3 – Why Stop Now?

Good coverage for test cases is still important, however this is an activity outside the requirement workshop. The intent of the workshop is engage the product owner and the team.

3.5 Rules vs. Examples

Are examples alone sufficient to specify the functionality of an application? It is not always possible to “reverse engineer” the rules from the examples. Both rules and examples are needed to document the expected behavior of the system.

“The rules provide the concise, abstract description, and the examples provide precise, concrete illustrations of them.”

Chapter 3 – Rules vs. Examples

Why “Specification by Example”? The emphasis is on the use of examples to support a specification by making it harder for the rules to be misinterpreted.

Rules and examples are not the only way to specify the behavior of software. Other tools complement them, such as: definitions, model diagrams, formulas, and glossaries.

3.6 My Example Illustrates Multiple Rules!

Try to come up with Examples that illustrate a single rule. Focusing on a rule and utilizing it are two different things. Consider whether the example utilizes the rules, focuses on the rule, and/or illustrates the rule.

If an example illustrates multiple rules, then split the example into several that each focus on a single rule.

3.7 The Bigger Picture

Short, focused examples illustrate the behavior of a single rule, but the overview of the whole system behavior is missed.

Leverage wireframes, page-flows, box-and-arrow diagrams, and other tools to illustrate the application behavior.

3.8 What We Just Learned

The requirement workshop is an excellent place discuss the understanding of the requirements. Creating concrete examples that illustrate the rules are challenging.

Examples on their own are not sufficient; rules should be documented as well. An Example is comprised of context, action, and outcome.

Examples should illustrate a single rule and contain only data essential to understand the behavior of the rule.

Book Club: Behavior Driven Development – Discovery (Chapter 2)

This entry is part 2 of 5 in the series BDD Discovery

The following is a chapter summary for “BDD Books – Discovery” by Gaspar Nagy and Seb Rose for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Background on BDD Discovery

DISCOVERY

“Explore behavior using examples

Written by the creator of SpecFlow and the author of The Cucumber for Java Book, this book provides inside information on how to get the most out of the discovery phase of Behaviour Driven Development (BDD). This practical guide demonstrates good collaboration techniques, illustrated by concrete examples.

This book is written for everyone involved in the specification and delivery of software (including product owners, business analysts, developers, and testers). The book starts by explaining the reasons BDD exists in the first place and describes techniques for getting the most out of collaboration between business and delivery team members.

This is the first in the BDD Books series that will guide you through the entire development process, including specific technical practices needed to successfully drive development using collaboratively-authored specifications and living documentation.”

BDD Discovery

Chapter 2

“In this chapter, we are going to peer into the daily work of a software product team to learn more about how they use structured conversations to help them discover what the expected behavior of the next feature should be. We’ll start by describing one of their requirement workshops. This will introduce concepts that you’re not familiar with, but don’t worry, all your questions will be answered later in the chapter.”

Chapter 2 – Structured Conversation

2.1 Where is My Pizza?

The sample used in the book is a Pizza delivery service, with an app that will track the real-time location of order(s). This example will be used throughout the book as the team develops features to modify the delivery address of an order after the order has been submitted.

2.2 A Useful Meeting

The section introduces the idea of a “Requirement Workshop”:

“The team meets regularly (usually several times a week) to discuss the work that they’ll be undertaking in the next sprint or two. The purpose of this meeting is to explore a story, understand it’s scope and illustrate it unambiguously with concrete examples. While they’re doing this, they may discover new details about the story. They may also ask questions that no one at the meeting is able to answer right away.

What matters most in this meeting is to bring diverse perspectives together, so that they can learn about what needs to be done and work together more effectively. In other organizations, similar meetings have been variously called three amigos meeting, discovery workshop, specification workshop, story refinement, product backlog refinement and backlog grooming – as always, the name is less important than the purpose.”

Chapter 2 – A Useful Meeting

Requirement Workshop

  • The team meets regularly
  • The purpose is to explore a user story
  • The scope of the story is discussed
  • The story is illustrated with examples
  • Questions are documented that no one in the workshop can answer

The team will consider “Change Delivery Address”. The feature will implement the following:

  • The system will need to be able to confirm whether it’s possible to change the delivery address
  • The system will need to check that the new address is not too far from the current one.
  • Determine the state of the order

Throughout the book, the authors demonstrate a technique called Example Mapping to help facilitate requirements workshops. Example Mapping is a way to conduct a structured conversation.

2.3 Collecting Examples

Persona in user-centered design and marketing is a fictional character created to represent a user type that might use a site, brand, or product in a similar way.

An example is written on a green card. Please refers to Figures 7 & 8.

Business Rules are written on blue cards. Only valid address is accepted. Estimated time of arrival is updated or new address should be within restaurant’s delivery range.

Problems or Questions are written on red cards. Please refer to Figure 12.

The team discusses other business rules such as valid address is accepted, estimated time of arrival is updated, and new address is within restaurant’s delivery range. For each rule, one or more examples are used to illustrate those rules. The workshop finishes in a time-box of 30 minutes with the completed Example Map shown in Figure 14. Further details on the content of an Example Map is discussed in Section 2.5.

2.4 Deliberate Discovery

“Discovery that happens while you’re developing software can be thought of as ‘accidental discovery’ – it may upset your schedule, or even derail or interrupt your roadmap entirely. The discovery that happens during a requirement workshop is ‘deliberate discovery.’”

Chapter 2 – Deliberate Discovery

The understanding of the user story is deliberately explored with concrete examples. Otherwise, learning happens accidentally during development of a story. Examples are used both to illustrate what is known and question assumptions.

2.5 Example Mapping in a Nutshell

Example Mapping is a simple, low-tech method for making conversations short and powerfully productive.”

Matt Wynne

Examples are captured on green cards – illustrate concrete behavior of the system.

Rules are written on blue cards – these are logical groupings of the examples usually focusing on a particular condition. Commonly referred to as acceptance criteria (AC), business rules, or simply requirements.

Questions or assumptions are captured on red cards – any topic that would block the discussion.

User Stories are written on yellow cards – start by discussing a single user story, but as we are digging into the details, we often decide to split the story into smaller stories and postpone some of them.

The User Story is placed on the top row and the rules are arranged in a row underneath. Examples pertaining to a given rule are placed underneath that rule. The red cards are placed to the side of the Example Map. For each session, a facilitator should schedule the meeting and keeps the session active by ensuring discussion is captured on the cards and everyone agrees with the wording.

2.6 How to Establish Structured Conversations

Structured Conversation is a facilitated exchange of ideas that conform to a predefined form.

A structured conversation exhibits the following properties:

  • Collaborative – all attendees participate actively
  • Diverse Perspectives – all primary areas of a team are represented
  • Short – regular workshops in a time-box so the feedback loop is quick
  • Progressive Focus – capture the progress of the workshop in real-time
  • Consensus – agreed concrete examples measure the workshop’s success

Each property in the structured conversation is elaborated on further below.

Collaborative

  • Conversations should involve the entire team
  • Establish a culture where everyone can participate

Diverse Perspectives

  • Requirement workshops should have representation from different perspectives (development, test, business, UX, etc.)
  • The business representatives focus on the fulfillment of the business goals
  • The developers explore the technical implications of the feature
  • The testers challenge the feasibility of testing the feature and help identify special edge cases

Short

  • Requirement Workshops should be no longer than 30 minutes
  • Long meetings are exhausting and expensive.
  • Short meetings can be scheduled more frequently
  • Frequent meetings can vary the attendees to improve shared ownership and can also reduce the impact of unanswerable questions

Progressive Focus

  • The workshop has a focus on what will be discussed but not discovered
  • Understanding becomes progressively more complete
  • Meeting should have a stated purpose
  • Understanding should be captured as it develops
  • Be able to quickly grasp the state of the discussion
  • Stop discussions that aren’t going anywhere

Consensus

  • The output is correct
  • The feature is sufficiently understood
  • There is no hidden or private knowledge
  • Who is responsible for answering each remaining question

2.7 What We Just Learned

“Software development is a learning process. The more we can learn about the problem, the easier solving it becomes. This process can be made more effective by having several team members (with different perspectives) analyzing the requirements together before they start developing the software. These collaborative requirement workshops are most productive if they are kept short and run regularly throughout the project – often several times a week.”

Chapter 2 – What We Just Learned

In the next Chapter of the series, we’ll learn how to write concrete Examples.

Book Club: Behavior Driven Development – Discovery (Chapter 1)

This entry is part 1 of 5 in the series BDD Discovery

The following is a chapter summary for “BDD Books – Discovery” by Gaspar Nagy and Seb Rose for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Background on BDD Discovery

DISCOVERY

“Explore behavior using examples

Written by the creator of SpecFlow and the author of The Cucumber for Java Book, this book provides inside information on how to get the most out of the discovery phase of Behaviour Driven Development (BDD). This practical guide demonstrates good collaboration techniques, illustrated by concrete examples.

This book is written for everyone involved in the specification and delivery of software (including product owners, business analysts, developers, and testers). The book starts by explaining the reasons BDD exists in the first place and describes techniques for getting the most out of collaboration between business and delivery team members.

This is the first in the BDD Books series that will guide you through the entire development process, including specific technical practices needed to successfully drive development using collaboratively-authored specifications and living documentation.”

BDD Discovery

Chapter 1

Behavior Driven Development (BDD) is an agile approach to delivering software.

BDD is about collaboration and domain discovery.

JUnit requires developers to use the word “test” in their method names, which emphasizes testing over documentation, design, and definition of the expected behavior.

Dan North originally designed BDD as a set of practical rules for naming and structuring tests to preserve a connection to the requirements. The business-readable format was designed so the business representatives could confirm the expected behavior.

1.1 The Missing Link

The purpose of software development is to deliver solutions to business problems.

A continuing challenge is to verify that the software actually satisfies the requirements.

Waterfall methods have slow feedback cycles, which allows projects to go off track. In response, the industry began to experiment with methodologies like XP and frameworks like Scrum.

Please refer to Figure 1

BDD helps maintain a connection between the requirements and the software – and as such acts as a bridge. The bridge is made out of examples. Every test is an example. Each example is an expression of a required system behavior.

“If you have sufficient examples you define the behavior of the system – you have documented the requirements. Business people remain engaged, because the examples are expressed in business language. Fast feedback is preserved, because the examples are automated.”

Chapter 1 – The Missing Link

Chris Matts is originally credited with Given, When, Then keyword format for examples written as Scenarios. There are several free tools for writing BDD Scenarios: Specflow and Cucumber.

1.2 How Does BDD Work?

“Examples (and their formalized representation – scenarios) play a critically important role in BDD. To understand how BDD works, let’s have a look at the way that these scenarios are created and how they drive the development process.”

Chapter 1 – How does BDD Work?

BDD is used when the details of user stories are discussed, in the form of examples. Focusing on examples makes the business rules clear. As a general guideline, each business rule should be illustrated by one or more examples.

“Examples also enable us to explore our understanding of a rule. Exploration often leads to the discovery of complexities and assumptions that otherwise would not be found until much later in the development process.”

Chapter 1 – How Does BDD Work?

Examples:

  • Examples can take various forms: input-output data pairs, sketches of the user interface, bulleted lists of different steps of a user workflow or even an Excel workbook illustrating a calculation or a report.
  • All examples describe a behavior as a combination of context, action, and outcome.

Some examples are formulated into scenarios when a user story is ready. BDD tools turn these scenarios into executable tests before the related behavior has been implemented in the application itself.

Test-Driven Development (TDD) helps speed up the feedback loop by demanding that teams write automated tests before they write the code. A test is written first, which initially fails. Next, the application feature is implemented so that the test passes. After code cleanup (refactoring), the next test can be written.

BDD drives development in a similar fashion to TDD, however the scenarios are described from the perspective of the user. Begin by writing a scenario that should fail. Next, write the automation & application code until the scenario passes. After code cleanup (refactoring), the next scenario can be written. BDD does not replace TDD, but rather is built into the red-green-refactor cycle.

Please refer to Figures 3 and 4

BDD Feedback Benefits:

  • Implementation correctness for developers
  • Overall solution for the product owner
  • Implemented behavior to help business analysts understand existing functionality
  • Provides a signal for manual / exploratory testers that a feature is ready for testing
  • Safety net for developers, identifying unwanted side effects of changes
  • Detailed documentation of application behavior for any support team
  • Defines a domain language that is understood by everyone

1.3 What About Testing?

BDD does not replace classic testing or testing skills. BDD does not define how testing should be performed, but rather provides guidelines about the Agile Testing Process.

The focus of the Agile Testing Process is to help ensure defects are never added to the codebase in the first place. Testers are involved in project requirement discussions, to help prevent bugs.

A significant proportion of defects are rooted in problems that arise from misunderstood requirements (see in Outside Resources for J.-C. Chen and S.-J. Huang, 2009). The primary type of verification is done through exploratory testing (see in Outside Resources for E. Hendrickson, 2013).

1.4 A Language That is Understood by Everyone

Every project is a voyage of discovery. A language must be established in which the customer can explain the problem(s) in detail and the development team can explain the solution. Teams should write scenarios using this language. If scenarios drive the implementation, then the solution will be close to the business domain.

1.5 Living Documentation

Living Documentation is a form of documentation that represents the current state of the application, which is updated in real-time. Scenarios make up the living documentation, which should be understandable by everyone. Scenarios should be written in domain-specific terms to describe the behavior of the application.

1.6 What is BDD, Then?

BDD is an agile approach that consists of three practices that have to be addressed in order:

  • “The first practice is discovery, a structured, collaborative activity that uses concrete examples to uncover the ambiguities and misunderstandings that traditionally derail software projects.”
  • “The second practice is formulation, a creative process that turns the concrete examples produced during discovery into business-readable scenarios.”
  • “The third, and final, practice is automation where code is written that turns the scenarios into tests.”
  • The above practices must be adopted in order to gain the expected benefits.

The benefits of automation:

  • “When the tests pass, the development team can be confident they have delivered what the business have asked for.”
  • “The tests give the development team a safety net when the time comes to modify the code.”
  • “The tests form living documentation of how the system behaves, readable by the business, guaranteed to be up-to-date.”

Please refer to Figure 5

1.7 What We Just Learned

BDD was created to address the challenges associated with misunderstandings in a project team. Examples make an excellent bridge between business requirements and technical specifications.

Scenarios written in a ubiquitous language, act as both documentation and tests. As documentation, they close the gap between business and development. As tests, they demonstrate the solution being implemented aligns with business requirements. The Scenarios provide documentation that describe the actual application functionality.

The focus of BDD is collaboration and the examples are turned into test cases. The purpose is to inform all involved that development is moving in the correct direction.

From the Pipeline v17.0

This entry is part 17 of 34 in the series From the Pipeline

The following will be a regular feature where we share articles, podcasts, and webinars of interest from the web.

The Top 5 Considerations for Creating a Successful Cloud-Based Pipeline

The article posted by Even Glazer covers the top five considerations for creating and running an automated, cloud-based pipeline. He advises anyone looking to implement a pipeline to consider the constraints of your organization, particularly around data security policies. The top five considerations area: (1) Consider Your Business Needs First, (2) Develop Your Cloud-Based Pipeline as Part of Your Apps, (3) It’s All About Continuous Improvement, (4) Enable Self Service Features, and (5) Track Your Pipeline, Microservices and Compliance Policies.

Patterns of Distributed Systems

Another mammoth post that can be turned into a book chapter on Martin Fowler’s site. This time, guest author Unmesh Joshi takes us through a set of patterns he observed in mainstream open source distributed systems. Several of these patterns are works in progress but the article itself is well worth a read.

“Distributed systems provide a particular challenge to program. They often require us to have multiple copies of data, which need to keep synchronized. Yet we cannot rely on processing nodes working reliably, and network delays can easily lead to inconsistencies. Despite this, many organizations rely on a range of core distributed software handling data storage, messaging, system management, and compute capability. These systems face common problems which they solve with similar solutions. This article recognizes and develops these solutions as patterns, with which we can build up an understanding of how to better understand, communicate and teach distributed system design.”

Code Coverage Best Practices

Another great post from the Google Testing blog about code coverage. They openly question whether or not code coverage alone reduces defects and a high % of coverage being responsible for higher quality in test coverage. Chasing a specific number does not mean the application under test is of good quality. Instead, it’s important to use a risk-based approach to testing and ensuring that all deployments are gated by code coverage.

“We have spent several decades driving software testing initiatives in various very large software companies. One of the areas that we have consistently advocated for is the use of code coverage data to assess risk and identify gaps in testing. However, the value of code coverage is a highly debated subject with strong opinions, and a surprisingly polarizing topic. Every time code coverage is mentioned in any large group of people, seemingly endless arguments ensue. These tend to lead the conversation away from any productive progress, as people securely bunker in their respective camps. The purpose of this document is to give you tools to steer people on all ends of the spectrum to find common ground so that you can move forward and use coverage information pragmatically. We put forth best practices in the domain of code coverage to work effectively with code health.”

Searchiiiiiing, Seek And Locate…But Only With Appropriate Attributes for Automation

Another great post by everyone’s favorite metal-loving test automation architect Paul Grizzaffi. In his latest post, Paul discusses one of the big drawback of UI-based automation: attributes used for locating elements. Sometimes the only access we have to an application is via the UI, so we must interact with elements that can be typed into, pressed or click, and need their values to be inspected. Sometimes third-party tools (such as Salesforce or SiteCore or Oracle Cloud) have dynamically built elements that make attributes difficult to nail down. We can instead use “data” attributes to add arbitrary attributes to an HTML element to help automation developers locate elements and make UI-based automation a little less brittle.

WebDriverIO for Javascript Automation Testing

Joe Colantonio walks us through the various implementation of WebDriverIO for automation testing. WbDriverIO is a javascript-based testing tool that uses Selenium. People typically use WebDriverIO if they don’t want to build their own framework from scratch and they want some additional features not provided in vanilla Selenium. The article provides a list of these features along with several podcast interviews on the subject for those looking to learn more.

Slaying the Leviathan: Containerized Execution of Test Automation-part 1

This entry is part 1 of 2 in the series Slaying the Leviathan

Introduction

In this three-part series on test automation, I will explain how to utilize Docker, a Ruby/Cucumber web framework, and Jenkins to run parallel test automation within Docker containers.

All of the code shown in this series is accessible on GitHub here. This repository contains all Docker and Test Automation components necessary to follow along and complete the same steps on your machine. If you plan on following along, you will need to install Docker, JenkinsGit, and Ruby

After you’ve installed the necessary prerequisites, go ahead and pull down the suite from GitHub. Navigate to the directory in the terminal where you want this framework to reside and run the command: ‘git clone https://github.com/jerren1122/docker_web_repo.git

If the command executes correctly, you should now have the docker_web_repo pulled down onto your local machine. In order for the later functionality to work I would ensure that the root folder of the framework has the name sample_cucumber.

Framework Introduction

Before we start with Docker, it’s important to understand how the framework functions when running locally. Completing the following steps provides a level of insight into the steps/components necessary for the framework to run.

  • Step one has completed as we have cloned the repo from GitHub.
  • Navigate to your terminal and type in ruby -v, which returns the version number of Ruby installed on your machine. If it doesn’t, the Path environment variable needs to be updated to include the location of the Ruby executable. I am running Ruby 2.6.6, my path to the Ruby executable would be C:\Ruby26\bin.
  • In the terminal, navigate to the directory where you pulled down the docker_web_repo, type in ‘gem install bundler‘, allowing you to utilize bundler’s ability to install all the gems listed in the Gemfile of the repo.
  • Now input ‘bundle install‘ which pulls all the gems listed in the Gemfile down from https://rubygems.org.
  • Navigate here and download the ChromeDriver of the same version number as Chrome on your machine.
  • Once the ChromeDriver has been downloaded, place it in the bin directory of the version of Ruby you installed.
  • This suite utilizes Rake tasks to run, for more information about Rake tasks click here. For this blog post, all you have to do is set an environment variable and run the Rake task. First, input set or_tags=regressioninto your terminal and then type rake features:default. You will should see the output that the regression is running. No browser will visibly open because the suite is running in headless Chrome.

We now have a checklist of the environmental setup to be completed for this web repository to execute automated tests. 

Additional Functionality

Another powerful facet of this framework is the code in lib/utilities/dynamic_tags.rb. This code allowed the tests in the suite to be split into sections and ran in parallel. All this code required at runtime is for environment variables set for total_number_of_builds and build_number

This code worked by replacing a tag (by default @regression) with the tag @split_builds for a subset of the tests corresponding to the build_number provided as an environment variable.

Docker 101

The best overview is provided by the folks at Docker here. For our purposes, it’s important to understand the following.

At a high level, Docker allows for multiple processes to run on a single host. Each processes have a unique Host name, IP address and File System. Shared between them are the Hosts Operating System, CPU, and Memory. This is very similar to what a Virtual Machine is; however, one difference is that Virtual Machines have their own operating system.

Additionally, Docker offers methods of storing and deploying these processes. All of the setup necessary to run a process is stored as a Docker image. The Docker image is a snapshot of what you want the environment to look like for the process to run successfully.

The Docker image is constructed, utilizing a set of steps, housed within a Dockerfile.

Lastly, when we deploy a running instance of these images, which we call a “container” in the Docker world.

Conclusion and Next Steps

In this first blogpost of a three-part series, we executed the framework locally on our machine and became familiar with some of the components in the framework. Additionally, we have explored what Docker is, conceptually.

In the next blogpost, we dive into the actual implementation of Docker and how it interacts with the testing framework.