How to Conduct an Accessibility Test

This post is about the procedure for conducting an accessibility test. Naturally, auditors know how to evaluate accessibility. However, it can also be useful for the client to understand the process in order to be better prepared for it.

Tips for the Client

As a client, it is useful to consider in advance which components and views should be tested. We love such clients. Since we come from the outside, we do not know the internal logic of the system; we can only judge from an external perspective.

Please plan for sufficient lead time. Most clients underestimate the testing effort, the number of issues, and the time required to fix them. As a rule of thumb:

  • If you have never performed an accessibility test before, and
  • If you have many systems or if these systems are complex,

you should expect many problems and a significant amount of time for remediation. Unfortunately, the accessibility skills of one's own project participants are often overestimated, or many people misjudge their own abilities.

My insider tip: Have the component library tested first. Many fundamental issues only need to be fixed once within the component and are then resolved in all applications that use that component. The test is then not redundant, but it can be carried out much faster, which reduces overall costs.

WCAG EM as a Guideline

Anyone wishing to conduct an accessibility test can follow the WCAG-EM (Web Content Accessibility Guidelines Evaluation Methodology). This evaluation methodology is a guide provided by the World Wide Web Consortium, specifically the Web Accessibility Initiative, to test accessibility systematically and methodically.

The methodology is publicly accessible. There is currently a published version as well as a draft for version 2.0. The current version primarily refers to websites. Version 2.0 will additionally include other formats such as apps and PDFs. However, the basic procedure remains largely the same.

For those not deeply familiar with the WCAG, it is important to note: The WCAG are normative. This means they are binding if you refer to them. In contrast, most WAI documents are informative. They serve as guidance but are not mandatory.

Gaining an Overview

The first step of an audit is an initial inspection of the test object. For a website or app, this means going through all relevant pages or screens once to get an overall impression.

This inspection is particularly useful for identifying early on whether a full audit even makes sense. For example, with PDFs, it can be quickly determined whether they contain structured text. If there are no tags, no headings, no paragraphs, and no tables correctly marked up, the document is not accessible. In such cases, the audit can be terminated early.

Such a quick check is also useful for drafts. For websites, different versions often exist, for example, in staging environments. It happens that outdated or incomplete versions are submitted for testing. In such cases, one should point out early on that basic requirements are not met and a full audit offers no added value.

This approach saves time and costs and is more transparent toward the customer than conducting a full audit despite obvious deficiencies.

During the inspection, one should also analyze which components and states are present. This includes, for example, whether elements are expandable or have different states. This helps to better estimate the scope and complexity of the audit.

Since websites and apps often contain a vast number of pages, a full manual audit is usually not possible. Therefore, exemplary pages or components are selected. The goal is to identify and document the most important and representative parts.

The methodology also provides for defining a conformance level (A, AA, or AAA). In practice, this step is often less relevant, as Level AA is targeted in most cases.

A further step is determining the standard to be applied. This stepis often already defined by legal requirements, such as the EU directive 2016-2102 or the European Accessibility Act.

Defining the Scope of the Audit

A central point is the definition of the test object (the scope). It is important not only to determine what will be tested but also to explicitly state what will be excluded.

This demarcation should be done in consultation with the client. One example is web views within apps that display external content. If this content is not within the operator's responsibility, it can be excluded from the test.

Legacy systems also play a role. Many websites consist of several systems, such as a new section and an older area. In such cases, it must be clearly defined which parts are part of the audit and which are not.

The selection of the test object is typically a collaborative process: the auditor suggests relevant pages or components, and the client confirms or supplements this selection.

For classic websites, at least the following page types should be considered:

  • The homepage,
  • A simple content page,
  • A complex content page (e.g., with tables or interactive elements),
  • A form page (e.g., a contact form).

The homepage is particularly important as it often features a unique structure and specific components.

For applications with clearly defined processes, such as online shops, complete workflows must be tested. A typical process includes product search, adding items to the shopping cart, logging in, entering address and payment data, and completing the order.

According to WCAG, a process is only considered compliant if all steps of the process are compliant. Therefore, the entire workflow must be audited.

This significantly increases the testing effort but is necessary to make a valid statement about accessibility.

Conducting the Audit

Once the test object has been defined and the client is involved, the actual audit begins.

This step is complex and heavily dependent on the individual approach. Some auditors work through each page systematically according to all WCAG criteria. Others group test steps and check multiple pages in parallel, for example, across different browser tabs.

Regardless of the method, a clear and consistent structure is crucial. The audit should always follow a fixed schema.

For each criterion, it is first checked whether it is applicable at all. There are criteria that are not relevant in many cases, such as requirements regarding blinking or flickering. Other criteria are almost always relevant as soon as content is present. These include, among others:

  • Correct use of headings,
  • Structured text markup,
  • Keyboard accessibility.

Subsequently, the applicable criteria are systematically tested for all selected pages or components.

A major reason for the duration of such audits is the low degree of automation. Automated tests often generate misinterpretations (false positives and false negatives). Therefore, automated results must always be verified manually.

All identified problems must be documented. Good documentation includes:

  • A precise and understandable description of the problem,
  • A screenshot or a relevant code snippet,
  • An assignment to a specific criterion (e.g., WCAG or EN 301 549),
  • A recommendation for remediation.

The error description must be specific. Descriptions that are too general are not actionable for developers.

The assignment to an EN criterion is important. It ensures that the issue is an actual accessibility problem and not a subjective assessment.

Recommendations for remediation should be provided but do not need to be exhaustive. Describing one possible solution is sufficient. The concrete implementation remains the responsibility of the development team.

It is also important to describe the expected behavior. In other words: How should the function work correctly? This information is often more helpful than a technical solution, especially when different technologies or frameworks are used.

Since auditors are generally not specialists in every framework, it is not their job to provide specific implementations for every system. Their task is to formulate accessibility requirements and the desired behavior in an understandable way.

Another crucial aspect is the documentation of the testing environment. This includes:

  • Tools used (e.g., browser tools, bookmarklets, accessibility toolbars),
  • Browser and version,
  • If applicable, screen readers (e.g., TalkBack) including the version,
  • In certain cases, the operating system.

This information is necessary to ensure that test results are reproducible. Browsers and screen readers can exhibit different behaviors or bugs.

A good audit report must be understandable for the client. It should clearly answer:

  • What was tested?
  • What is the specific problem?
  • What is the expected behavior?

After completing the audit, the results should be discussed with the client. Due to the volume of information, it is sensible to focus on the most important findings.

Additionally, it can be helpful to support the client in creating tickets or to work directly within their ticket system. Direct communication with the development team is recommended, as questions often arise during implementation.

The audit report should be provided in a format that is easy to process further, such as a CSV file or via a tool that integrates directly into ticket systems. Direct access to the client's ticket system can significantly simplify the process.

Additional services, such as the creation of user stories, can be offered but are not necessarily part of the standard scope of an audit report.

In practice, several testing cycles are usually required. Typically, at least two rounds are needed to identify and resolve the most significant issues.

The effort depends heavily on the complexity of the system. Systems with a long history and many legacy components are much more difficult to make accessible than modern applications. Since they contain more errors—which is almost always the case. Documenting these issues is also significantly more time-consuming.

class="mozTocH2" id="mozTocId519699">More on Testing & Evaluation