Comparison of important Accessibility Testing Methods

In this article, I would like to compare the different ways in which digital accessibility can be tested. The main focus is on when which method can have advantages or disadvantages. This is about structured and methodical testing, not just clicking through.

I distinguish between four types of testing: automatic testing, expert testing, testing by disabled laypeople and testing by non-disabled laypeople.

Automatic methods:

Automatic accessibility tests use specialized tools to check websites and digital content for compliance with accessibility standards. These tools analyze the source code of websites and identify common accessibility problems, such as missing alt text for images, incorrect heading structures, low color contrast and problems with keyboard navigation. Automated testing can be used as a standalone testing process or as part of a more comprehensive audit that also includes manual testing.

The typical workflow of an automated audit includes the following steps:

  1. Choosing the tool: There are several tools, such as Axe, WAVE, Lighthouse and others, that can be used for automated accessibility testing.
  2. Performing the analysis: The tool scans the source code and structure of the website or application and reports accessibility issues it detects.
  3. Reporting: The tool generates a report listing the founded issue, often with advice on how to fix the issues.
  4. Review and fix: Developers review the reported issues and fix them before re-testing them.

Benefits

  • Efficiency and speed: Automated audits can quickly analyze a large number of pages or applications, which is particularly beneficial for large websites.
  • Cost-effective: Since they do not require human testers, automatic tests can be more cost-effective than manual tests.
  • Consistency: Automatic tools offer consistent assessment because they always work according to the same rules and human errors are minimized.
  • Immediate feedback: Developers and designers receive immediate feedback on the problems and can integrate this into their development process.
  • Problems can be found during the design and development phase without any in-depth know-how and can be partially resolved automatically or with software support.

    • Disadvantages

    1. Limited scope: Automatic tools cannot test all aspects of accessibility, especially complex interactions, user experiences or semantic meanings. Many problems, such as the actual meaning of alt text or the usability of forms, can only be identified through manual testing.
    2. False positives and false negatives: Automated tools can produce both false positives (reporting problems that are not an issue) and false negatives (missing real problems).
    3. Lack of context assessment: Tools cannot fully understand the context in which elements are used. For example, they could meet technical requirements but not assess whether the solution is actually accessible for users with disabilities.
    4. Tool dependency and concentration: It happens that managers only rely on the tools or even optimize for them. In the USA, websites are often optimized for WebAIM Wave, because it is used by many mass warning letters to collect money in dubious ways.
    5. Interpreting the results requires some effort in any case, even though tools such as Silktide or Accessibility Insights now provide good help.

    Most - if not all - automatic testing tools use the free axe library, an open library of test rules. Some tools allow you to add your own rules. Individual providers can also add additional functions.

    Through machine learning and other methods, it is expected that the tools will soon be able to check significantly more than the 35 percent of WCAG problems that are common today.

    In summary, automatic accessibility checks are a valuable addition, but they do not replace manual tests by trained testers. Automatic testing tools are a must for design and development. They can also be used to monitor extensive websites.

    Expert Evaluations

    Expert tests are evaluations that are carried out by experienced people using a valid procedure.

    The typical process of such an expert test includes the following steps:

    1. Initial meeting and requirements analysis: The experts clarify the specific requirements of the project and the target groups. They discuss the content to be tested and prioritize critical areas.
    2. Manual review: The experts manually check the website or application for accessibility issues. They pay attention to usability for different user groups, including people with visual, hearing, mobility and cognitive impairments.
    3. Use of assistive technologies: The experts test the website with various assistive technologies such as screen readers, keyboard-only navigation, Braille displays or magnification software to ensure that these aids work correctly.
    4. WCAG assessment: Using the WCAG guidelines, the experts assess whether the website meets accessibility requirements and identify areas that need improvement.
    5. Preparation of a report: The experts document their findings, describe the problems found and make concrete recommendations for eliminating the barriers.
    6. Feedback and follow-up: After implementing the recommended improvements, the experts review the corrections again and provide further feedback if necessary.

    Advantages

    1. In-depth analysis: Experts can identify complex and subtle Identify accessibility issues missed by automated tools. They assess not only the technical aspects but also the actual usability for people with disabilities.
    2. Context assessment: Experts can understand the context and semantic meaning of content, allowing them to conduct an informed accessibility assessment. They can also assess whether the user experience is intuitive and easy.
    3. Tailored recommendations: Experts can provide specific, actionable recommendations based on the individual needs and context of the website or application.
    4. Testing with assistive technologies: Experts use real assistive technologies to ensure that content is accessible in real-world usage scenarios.
    5. Edge case consideration: Experts are able to consider atypical or complex scenarios that are outside of the usual test cases but may be critical for some user groups.

    Disadvantages

    1. Cost-intensive: Manual testing by experts is usually more expensive than automated testing as it requires more time and specialized knowledge.
    2. Time-consuming: The process is more time-intensive, especially when a comprehensive test of a large website or application is required.
    3. Subjectivity: Although experts are highly qualified, there is always a certain degree of subjectivity that can lead to different interpretations of accessibility requirements.
    4. Dependence on individual expertise: The quality of the review may depend on the skills and experience of the individual expert. Different experts may come to different conclusions.
    5. Less efficient for large websites: For very large websites or applications, the manual testing process can be impractical because it is difficult to check every single page or function.

    In summary, manual reviews by experts provide in-depth analysis and provide valuable insights that automatic tools cannot provide. They are particularly important for the final assurance of accessibility, but should ideally be supplemented by automatic reviews to ensure a comprehensive and efficient assessment. They are particularly useful for testing finished templates or components, but less suitable for testing large numbers of screens.

    Testing by disabled laypeople

    Accessibility testing by people with disabilities who are not accessibility specialists is an approach where real users with various disabilities test digital content, websites or applications. These users are usually not professional testers, but represent the target audience of the content and use it in everyday life. They interact with the website or application using their usual assistive technology or other tools and provide feedback on their experience.

    The process of such a test typically includes the following steps:

    1. Recruitment of testers: People with various disabilities are recruited, including people with visual, hearing, mobility or cognitive impairments. These users reflect the diversity of the potential user base.
    2. Provision of tasks: Testers are given specific tasks to perform on the website or application. These tasks can include everyday interactions such as navigating the website, filling out a form, or finding information.
    3. Observation and documentation: As testers perform tasks, they are observed and their interactions are documented. This can be done through screen recording, video recording, or on-site observation.
    4. Collecting feedback: After testing is complete, users provide feedback on their experience. This includes both positive and negative aspects, as well as suggestions for improvement.
    5. Analysis and implementation: Developers and designers analyze the feedback, identify barriers and difficulties, and implement improvements accordingly.

    Benefits

    1. Real user feedback: This type of testing provides authentic feedback from the people who actually use the content. They identify barriers that occur in the real world and provide insights into practical usability.
    2. Real usage scenarios: Since the testers are not accessibility specialists, they use the website or application as they would in everyday life. This allows for a realistic assessment of accessibility.
    3. Identifying unforeseen problems: Users with disabilities can identify problems that are overlooked by experts or automated tools, as they may encounter unexpected difficulties that only arise in real-world use.
    4. Strengthening user-centeredness: By involving disabled people in the testing process, development becomes more focused on the needs and experiences of the target group.
    5. Direct feedback on assistive technologies: Since users use their own assistive technologies, the feedback can directly help improve the compatibility and usability of these technologies.

    Disadvantages

    1. No systematic expertise: Since testers are not accessibility specialists, they may not be able to identify all potential barriers or understand why certain problems occur. Their feedback may therefore be subjective and less structured.
    2. Limited scope: Tests are often limited to specific scenarios that are relevant to the users in question. This can result in other important accessibility aspects being overlooked.
    3. Incomplete coverage: Because these users only test specific tasks or areas, the entire website or application may not be tested for accessibility.
    4. Variability in experience: Test results can vary greatly depending on the individual skills, experience and technologies used by the test subjects.
    5. Lack of depth: These tests usually offer less technical depth and detailed analysis than tests conducted by accessibility experts.
    6. Time-consuming: Recruiting and conducting the tests and evaluating them can be very time-consuming, depending on the test design.

    In summary, tests conducted by disabled people who are not accessibility specialists provide valuable insights into actual usage and user experience. They are particularly useful for identifying real barriers and improving usability. However, these tests should ideally be combined with automated testing and expert assessments to ensure a comprehensive assessment and improvement of accessibility. Such tests are ideal for evaluating paths such as an eLearning path or a shopping experience.

    Layperson Tess

    Layperson accessibility testing refers to the review of digital content by people without specific accessibility expertise. These laypeople typically conduct tests using clear instructions or checklists that cover basic accessibility aspects. These tests can be conducted by internal employees of a company or by volunteers who are not accessibility experts.

    The typical flow of such a test includes the following steps:

    1. Providing instructions and checklists: Laypeople are given detailed instructions to guide them through the testing process. These instructions are often based on simplified versions of the Web Content Accessibility Guidelines (WCAG) and cover basic aspects such as contrast ratios, alternative text for images, and keyboard navigation.
    2. Performing the test: Laypeople test the website or application using the instructions provided. They review specific elements, such as the language, fonts, and size of the website, and then review the content in a format supported by the website. For example, whether images have alt text, whether all functions can be operated with the keyboard, or whether the font size is adjustable.
    3. Documentation of results: Testers document their observations and note problems they have found. This documentation can be in the form of reports, checklists, or simple feedback forms.
    4. Feedback and discussion: Results are collected and forwarded to the developers or accessibility team. In some cases, testers can also participate in a feedback discussion to explain their experiences in more detail.
    5. Revision and follow-up: Adjustments are made based on the layperson's feedback. A retest can then be carried out to check the improvements.

    Advantages

    1. Easy access: This type of testing can be carried out easily and quickly, as no in-depth technical knowledge is required. Anyone with basic computer skills can take these tests.
    2. Wide applicability: These tests can be applied to a wide range of content and are particularly useful for companies that do not have in-house accessibility experts.
    3. Promote awareness: Conducting such tests can increase awareness of accessibility within a company or organization and promote a culture of inclusion.
    4. Cost-effective: Because they do not require specialized professionals, these tests are often less expensive and require fewer resources than expert or user testing.
    5. Rapid implementation: Because the tests are based on clear instructions, they can be conducted quickly, which is particularly useful during regular reviews or early stages of development.

    Disadvantages

    1. Limited depth: Because the testers are not experts, they can only identify obvious barriers. Complex or subtle accessibility issues are often missed.
    2. Lack of technical expertise: Testers may not be able to understand technical details or the causes of problems, which can limit the quality of feedback.
    3. Inconsistent results: Because tests are conducted by people with different backgrounds and understanding, results may vary and be inconsistent.
    4. Superficial testing: These tests tend to focus on basic aspects and may miss deeper issues that would require more comprehensive analysis.
    5. Reliance on instructions: The quality of tests depends heavily on the quality of the instructions provided. Unclear or incomplete instructions can lead to ineffective testing.

    In summary, non-expert testing provides an accessible and cost-effective way to identify basic accessibility issues. It is particularly useful in situations where specialized experts are not available. However, these tests should be considered a complementary measure and ideally complemented by expert assessments and testing by users with disabilities to ensure a comprehensive and reliable accessibility assessment.

    Conclusion:

    The different accessibility testing methods - automated testing, expert testing, non-specialized testing by disabled people and non-expert testing - all have their specific strengths and weaknesses. Their usefulness varies depending on the development phase of a digital project, and a combination of these approaches usually leads to the best results.

    1. Automated testing is great for the early stages of development and for regular automated checks throughout the project. It is efficient, cost-effective and can quickly check a large number of pages for common accessibility issues. However, they are limited to technical aspects and cannot capture more complex problems.
    2. Expert testing is most useful in the late development phase or after significant milestones. Experts provide in-depth analysis and tailored recommendations that ensure that subtle and context-specific barriers are also identified. However, these tests are time-consuming and costly, which is why they should be used specifically to finalize the quality of accessibility.
    3. Testing by disabled people without specialization is particularly valuable in the testing and validation phase and in user acceptance testing. They provide authentic feedback on the actual usability and help to identify unforeseen problems that may arise in real usage scenarios. Their feedback is crucial to making the application truly user-centered, but it is less structured and may lack technical depth.
    4. Testing by laypeople is most useful in the early development phase and in internal quality checks. They are a cost-effective way to identify basic accessibility issues and raise awareness of accessibility within the team. These tests are good for initial testing or in situations where quick, simple checks are required, but they should be used as a complementary tool and not as the sole testing strategy.

    In summary, the most effective strategy for ensuring accessibility is to integrate all four methods at different times in the development process and Evaluation after the publication of the project. Automated testing and lay testing can be used regularly and early on to identify obvious problems. Expert and user testing by disabled people should be used specifically to ensure the depth and quality of accessibility and to validate the application from the end user's perspective.

    It is important to have a strategy for all methods. Clicking around randomly, as is often the case with methods 1, 3 and four, does not provide any relevant insights.

    More on Testing & Evaluation