Automatic Accessibility Testing Tools - Possibilities and Limits
So far, automatic testing tools have played a minor role in digital accessibility. But what exactly can these tools do, and where do they reach their limits?
Dangers and problems of automation
First, let me clear up a misunderstanding. It's not about people or automation. It's about using automation and, for example, AI where it works very well, using it critically where it is still flawed, and not using it where human power is needed. With the approaches we have used so far in accessibility, we are making progress at a snail's pace.
There is the risk that customers and service providers or other parties will focus solely on the tool or even optimize for it. As a consultant, you should always warn against this. However, the resistance of many consultants is probably based more on the fear of losing their jobs due to automation. I currently consider the risk to be very low. No tool can replace experience and context analysis.
However, it does make sense to make sure that you are green in the testing tools: Many mass warning letters use Wave, for example, to create lawsuits. With the European Accessibility Act, this scam will also come to the EU, and the warning letters are already sharpening their digital knives. As unpleasant as non-accessible websites are, mass warning letters are a few steps below. This is not about the legitimate complaints of individuals or interest groups.
Automatic testing tools are the beginning. They do not replace know-how or a thorough manual check. But they are just as necessary as know-how and manual testing.
Axe as a basis
Most automatic testing tools are based on the free axe library, not to be confused with the axe-core tool offered by Deque. The respective providers can of course add extensions, but axe is often the basis.
Logical and context errors
Automatic testing tools are perfect for finding logical errors. Logical errors here mean errors in the code. A logical error is missing labels or events that are only based on a mouse click. A logical error is also when attributes are set when using ARIA that do not belong to the ARIA element used. It is not a logical error if I set incorrect ARIA roles because the software cannot recognize whether the link or button role would be correct for the specific element. Humans are rather bad at finding logical errors.
We call such problems context errors, i.e. errors that can only be found in context. Is the alternative text, heading, label or ARIA correct or not in this context? Currently only a human can recognize that. Humans are far superior to machines in this regard.
automatic evaluation tools can quickly identify logical accessibility problems. These include missing alt text for images, insufficient color contrast, unlabeled form elements and missing ARIA attributes.
The tools offer a quick way to perform an initial analysis of a website. This allows developers and designers to receive immediate feedback and quickly fix basic problems.
Automatic tools can be used regularly to continuously check websites for accessibility, known as monitoring. This is particularly useful for ensuring that new content and updates do not affect accessibility.
Most automatic testing tools provide detailed reports and recommendations to help developers understand and fix the problems found.
Areas of application for automatic testing tools
For the web, automatic testing tools are available for all areas from conception to design, development to QA and editing. All common tools such as Figma, development environments and content management systems can be expanded accordingly.
A bookmarklet is also an automatic testing tool in a certain sense, as it makes aspects visible that would otherwise have to be laboriously searched for in the source code or with a screen reader.
What automatic testing tools can check
It is said that around 35 percent of the WCAG criteria can be checked automatically. A study by Deque states that approximately 57 percent of WCAG issues can be found. 78 percent of the issues were based on just five criteria. The following criteria are responsible for 80 percent of all issues:
3.1.1 Language of Page 4.1.1 Parsing 1.4.3 Contrast (Minimum) 2.4.1 Bypass Blocks 1.1.1 Non-Text Content 4.1.2 Name, Role, Value 1.3.1 Info and Relationships Source
It must be said that bypass blocks and language of parts are pretty superfluous and contrasts are one of the weak points of the automatic testing tools. Many false positives and false negatives are produced for contrasts. 4.1.1 is obsolete due to WCAG 2.2. 1.3.1 and 4.1.2 are among the most complex WCAG criteria, so it is not surprising that most issues occur here. Deque, as a leading provider of such tools, naturally has an interest in presenting the checking tools in a very positive light.
Remember the statement made above: Logical errors such as a missing label or an incorrectly referenced ARIA ID are easy to find. However, the machine does not yet recognize if the element is actually an accordion and an incorrect ARIA role has been set.
Automatic tools can reliably check whether images on a web page have alt text. Alt text is important to ensure that screen reader users can understand the content of images.
These tools can analyze the color contrast between text and background to ensure that they meet accessibility standards.
automatic evaluation tools can check the structure of HTML code to ensure that it complies with accessibility guidelines. This includes checking heading hierarchies, list structures and other semantic elements.
Form elements such as input fields, dropdown menus and buttons must be correctly labelled so that they can be recognised and interpreted by screen readers. Automatic tools can reliably identify missing or incorrect labels.
ARIA attributes help to improve the accessibility of dynamic web content. Automatic tools can check whether ARIA attributes are used correctly and whether they comply with the guidelines.
Such tools should always help to correct the error. This used to be critical because the errors were displayed very cryptically. However, some tools such as Siteimprove or Silktide offer detailed explanations and help. The next step would of course be to automatically fix errors where possible or to make suggestions for correct code that can be adopted with little effort.
What automated testing tools cannot do so far
Automated tools cannot evaluate the quality and usability of complex interactions. They cannot judge whether a website is intuitive and easy to navigate for screen reader users, or whether interactive elements such as dropdown menus and modal dialogs work correctly.
Automated testing tools can produce both false positives and false negatives. This means that they may identify problems that are not problems, or miss real problems. This can lead to an incorrect assessment of accessibility.
Automated tools have difficulty analyzing content in context. They cannot judge whether a video without subtitles is acceptable for the specific context of the website, or whether alternative access routes are offered. Contextual content requires human judgment and understanding. In one specific case, ARIA markup was used for an accordion, but the correct markup would have been a tooltip. An automated check as it stands today could only check whether the markup is formally correct, but not whether it makes sense in the context.
What's in the future
Automatic accessibility checking tools have already made great progress, but there are still many opportunities for further development. Here are some potential developments that could be expected in the near future:
By using advanced artificial intelligence (AI) and machine learning, automated checking tools could be able to detect and assess more complex accessibility issues. This could include, for example, detecting patterns and context in content to enable more accurate assessments.
Future tools could use advanced semantic analysis to better understand the context and meaning of content. This could help them identify and assess not only technical barriers, but also content and language barriers.
By analyzing user behavior and interaction patterns, automated tools could better understand how real users interact with a web page. This could help them identify problems that cannot be discovered by purely technical analysis.
Advances in simulating user interactions could enable automated tools to perform more realistic tests. This could include simulating screen reader users, users with motor impairments, or users using alternative input methods.
Future testing tools could integrate more seamlessly into existing development environments and workflows. This could be achieved through enhanced APIs, plugins for common development environments, and better collaboration tools to make accessibility testing an integral part of the development process.
In addition to identifying issues, future tools could also provide automated fixes and concrete suggestions for action. This could help developers fix accessibility issues faster and more efficiently.
Advances in personalization could allow testing tools to tailor tests to users' specific needs and abilities. This could be achieved by creating user profiles and adapting testing methods to different user groups.
Future tools could provide real-time monitoring and reporting to ensure that websites and applications remain continuously accessible. This could be achieved by integrating monitoring tools and dashboards that provide developers and designers with instant feedback.