ADA validation is the oft-forgotten child in QA Automation conversations. As Quality Assurance professionals we focus on functional, performance, and security testing but miss the value and importance of accessibility validations. Any site that is customer-facing has an obligation to comply to ADA standards. Therefore, it’s important for us to make accessibility an up-front concern in testing.
For a little background, the Americans with Disabilities Act was signed into law in 1990. The law legally obligates the prevention of discrimination in all areas of public life. There are five sections of the law, but for web applications only the regulations within Title 3 – Public Accommodations apply.
The frequency and severity of the lawsuits related to ADA Title 3 are rising year over year, as evidenced by the chart above. Most of the lawsuits require the company to set aside resources to become ADA compliant.
The commonly accepted standard to gauge ADA compliance is the WCAG (Web Content Accessibility Guideline). The guidelines are provided by the W3C (World Wide Web Consortium). They are broken down into four sections.
Perceivable – Information and user interface components must be presentable to users in ways they can perceive.
Operable – User interface components and navigation must be operable.
Understandable – Information and the operation of user interface must be understandable.
Robust – Content must be robust enough that it can be interpreted by a wide variety of user agents, including assistive technologies.
Each guideline has multiple subsections and multiple success criteria to determine compliance; the criteria are judged on A, AA, and AAA standards with A being the lowest compliance level and AAA being the highest. Additional information can be found HERE.
There are multiple options for ADA compliance testing that fall into a few categories: (1) In house QA utilizing tools like JAWS to go through the web applications flow utilizing tools designed for disabled individuals; (2) Companies that will complete scans and/or manual validations as a service; and, (3) Static tools built as add-on’s into browsers such as Axe and Wave. I personally have no problem with any approach that nets results but would like to provide a fourth option to the list.
Pa11y is an accessibility testing tool utilized to determine a websites’ WCAG Compliance. The tool can scan based on A, AA, and AAA standards and can be executed directly against a web page or an HTML file.
To jump into the weeds a bit, Pa11y utilizes Puppeteer which is a node library that provides an API to interact with headless Chrome. When Pa11y runs Puppeteer it’s creating a headless Chrome browser, navigating to the web page or opening the HTML file in the browser and then the page is scanned against an WCAG compliance rule-set.
The next logical question is “what rule-set is utilized to determine WCAG compliance?” Pa11y by default utilizes HTML Code Sniffer, which is a client side script that scans HTML source code and detects violations of a defined coding standard.
Pa11y will return the following for each violation found: a description of the error, the WCAG guideline violated, the path to the HTML element, and the content of the HTML element. Pa11y by default will output to the command line but with some configuration modifications can export either CSV or JSON.
Additionally, Pa11y has the ability to run the Axe rule-set against the HTML spun up in the browser. This can provide a good level set if your developers are utilizing Axe as well.
So now that we have covered Pa11y, the next step will be discussing the ways in which we can implement Pa11y to run automatically.
The first way is built into Pa11y: we can implement the actions functionality that allows the browser to navigate through the web-page utilizing CSS Selectors via Puppeteer.
The second way is to utilize an existing test automation framework to complete the following steps:
- Utilize your test framework to navigate to the desired scanned page, scrape, and save the HTML to disk.
- From the command line pass the HTML file to Pa11y
The first option is beneficial if you’re building from the ground up as no additional framework is needed. The second option is beneficial if you have a built automation framework and you want to utilize the existing navigation to all the various pages that require validation.
Whichever choice you make the scripts should be built into a Continuous Integration (CI) job utilizing your tool of choice (Jenkins, Bamboo, etc). In addition to providing a way to continuously execute your scripts the CI tool will provide a location for storage of the scans to prove compliance if a lawsuit requires proof of compliance effort.
One important note: Automated scans with Pa11y do not replace the need for manual validation as there are WCAG requirements that cannot be validated via an automated scanning tool.
In summary, every web development team should be validating WCAG compliance as part of its software development life cycle. Also, WCAG compliance should be included in teams definition of done for a given card. Lastly, to maximize success for an application under test the results should be transparent and utilize your preexisting automation framework if possible to do the heavy lifting.
