In the European Union, the number of citizens with some form of disability is rising rapidly. An article published by the European Parliament states that in 2020, this number is estimated to reach 120 million in 2020. [1] Therefore, it is evident that making web applications more accessible becomes increasingly more important.
One way to test accessibility is through the Web Content Accessibility Guidelines (WCAG), a set of guidelines developed by the World Wide Web Consortium (W3C). These guidelines were developed with the goal of providing a shared accessibility standard for content on the web. [2] The WCAG guidelines can therefore not only be used for developing accessible applications, but also for assessing these applications.
Tooling for Automated Accessibility Assessments
In many projects, continuous integration is used in combination with linters or code analyzers, which automatically test code on a set of criteria, such as requiring explicit access modifiers on methods. A similar approach can be used to assess accessibility. The W3C provides a list of tools that can be used for this purpose. [3]
A bit of background: during my final semester at the Hanzehogeschool in Groningen, the Netherlands, I did my graduation project at the RDW (the Dutch vehicle authority). The project involved finding a way to automate accessibility assessment based on WCAG version 2.1. The tool I ended up using was Axe Core, developed by Dequeue Labs.
Let us consider a tool like Pa11y. During the research phase of my graduation project, this was one of the tools I tested. The tool works really well. Given a URL, Pa11y assesses accessibility and returns the results. For simple web applications, such as this website, this works really well. No authentication to worry about. Little to no user feedback.
Now, let us consider a situation in which a tool like Pa11y would be less helpful. Consider a large forum with over 1,000,000 members and twice that many messages. Each member belongs to one or more member groups. Each member group has a set of permissions. Can you see how this could become a problem? If we decide to use crawling, the time it would take to run a ‘simple’ accessibility assessment would be enormous. Each page would have to be scanned. This will take time.
Yet, we should also consider that, in a hypothetical scenario in which we actually run an accessibility assessment on the forum described above, that the assessment will be incomplete. Guests do not have access to the administration panel. Members that do have access to the administration panel, will not be able to see the login page. Then how can we get a complete assessment?
An Alternative Approach
Rather than running recursive accessibility assessments on a URL, an alternative approach would be to integrate them into automated tests. Whenever the UI changes, an accessibility assessment would be run. As long as there is a unit test for a component, it can be tested for accessibility. Assessments are no longer limited by component visibility, rather by the availability of tests.
One of the reasons I ended up using Axe Core in my final product was that it provided a way to run accessibility tests without the need to provide a URL. [4] Adding accessibility assessment calls to automated tests can even be done automatically, which I did in my final product.
Advantages and Limitations of Using Tests (for A11Y)
A major advantage of this approach is that any component can be assessed, regardless of visibility level. Assessments only need to be run as often as new UI elements appear. Therefore, in case of a large forum like we described before, we would only need to scan a component that displays messages in a topic once.
However, using automated tests to perform accessibility assessments relies, by definition, on the existence of automated tests. If a UI element is not exposed by any tests, then it cannot be tested for accessibility. It could be argued that any sufficiently large project should have sufficient automated tests, but in some projects this simply might not be an option. In these cases, it would be more practical to run assessments through URLs.
Limitations of Automated Assessments
It must be noted that the complete automation of accessibility assessments is unrealistic. For example, guideline 2.4.2 describes that all web pages should have a descriptive title. [5] However, automated tools most likely will not recognize that ‘About us’ is not a descriptive name for a download page.
Hence, automated tools should therefore be used in conjunction with manual testing.
Conclusions
With online accessibility becoming an increasingly bigger concern, measures must be taken to ensure that web applications are accessible. One way to accomplish this is by using automated tools, which can be integrated in build processes.
Using a tool like Axe Core, we can use existing tests for automated accessibility assessment. This allows assessment of each component that is exposed by at least one test. Therefore, even those components restricted to users with certain permissions can be assessed. The availability of sufficient tests is crucial for this approach.
Although automated testing cannot realistically fully replace manual testing, development time could be shortened when automated testing is used in conjunction with manual testing.
Footnotes
[2] https://www.w3.org/TR/WCAG21/#background-on-wcag-2
[3] https://www.w3.org/WAI/ER/tools