The WebAIM Study - are 96 per cent of the websites not accessible

WebAim published a new analysis in March 2023. With high numbers such as 50 million errors found on one million start pages, attention is guaranteed. My criticisms remain, however.

96 percent of the most used websites are not accessible - the news is currently making the rounds on Twitter and relevant accessibility channels. Great stuff, especially if you're just reading headlines. Personally, I do not find the WebAIM study to be meaningful for a number of reasons, or incorrect conclusions are drawn. I would like to explain the reasons in this post.

A short explanation: In professional circles we do not speak of accessibility, but of conformance. Conformance means that a certain standard has been met, for example WCAG 2.1 at level AA. Since the term "accessible" is not firmly defined for websites, this expedient is always necessary.

Article Content

Summary of my Critic

Since the article has become quite long, I would like to summarize the main criticism at the beginning:

As a rule, the found/alleged errors should not lead to restricted usability of the websites by disabled persons. Each subset of websites is poorly usable or unusable by a subset of disabled persons, but WebAim's analysis tells us nothing new in this regard. Usability by disabled persons and compliance with accessibility rules are not always congruent. When some experts claim, based on the WebAIM analysis, that 96 percent of the websites cannot be used at all by disabled persons, this can only be described as nonsense. It is extremely rare for an entire website to be completely unusable, but individual parts such as the login, the cookie message and similar things are often unusable.

WebAIM does not weight how serious the errors are. 1 or 1000 errors, according to WebAIM it is equally relevant. The communication of the effects is not designed for information, but for maximum effect.

If I read a study that claims almost 100 percent of vendors break rules, I would conclude that the rules are unworkable. Is that what WebAIM wants to say that it is not possible to be compliant with WCAG rules? If not, what is the gain in knowledge?

Automated tools are of limited value. You can make statements about a large amount of data that may not be relevant to the individual object. I can say, for example, that persons in Germany are on average 1.80314 meters tall and weigh 71.235124 kg, which may be true on average, but not for any individual. The websites probably have errors, but the WebAIM study cannot prove how many there are and whether they are relevant for use.

The converse conclusion is also wrong: Because WebAIM found no problems on 4 percent of the websites checked, these pages do not have to be accessible or usable for disabled persons, as Knowbility claims on Twitter. As a rule of thumb, around 35 percent of the problems can be found automatically. At most, WebAIM's study shows that these 4 percent tested automatically (probably with WebAIM's Wave) and ironed out these errors. However, they could still contain masses of errors that cannot be detected automatically. This clearly shows how little meaningful the WebAIM score is.

Methodology

The one million websites were automatically checked using WebAIM's WAVE tool. Much more can be said about the methodology. It is simply not possible to qualitatively analyze such a large number of websites in a reasonable amount of time.

But this is also where the first problem begins: The tool examines both WCAG criteria according to A and AA. But even in the US, most operators are not committed to accessibility and generally only aim for an A, if at all. There is no point in testing organizations for AA that do not aspire to AA, for example because they do not feel bound by the Contrast requirements.

Automated tools are of limited to no help

While WebAim Wave may still be one of the better tools, the consensus is that these tools can find maybe 30 to 40 percent of accessibility bugs. The tools are still rather lacking in my opinion, I have access to Siteimprove and Silktide and they both give a lot of errors that are irrelevant. There are many false-positive results, ie errors are claimed that do not stand up to manual testing.

There are things that can be measured automatically like the presence of certain HTML elements, ARIA attributes, labels, alt text and some contrasts. But the list of things that they cannot evaluate is longer. This includes the meaningfulness of alternative texts, the sensible use of ARIA, the correct marking of texts or form elements.

In short: Whether Wave displays errors or not is completely irrelevant. A lazy but smart developer runs the tool over it, irons out the bugs, and gets their site compliant without improving an iota of accessibility.

On the contrary, the tool sets false incentives, namely the optimization for automated testing tools. Why tedious manual testing when WebAIM gives you the go-ahead with one click?

As WebAIM itself notes, websites are becoming more and more complex. However, I assume that many websites, especially from Anglo-American countries, have the issue of accessibility on their radar. That means they take care of alternative texts or meaningful link descriptions. However, it is sometimes not possible to take these factors into account for externally integrated content.

A large part of the errors are likely to be due to such integrated content: This is, for example, social media content or advertising. If you go to WebAIM, you should probably omit such content as you can't make it accessible. This is more likely to deter persons from accessibility. Something else applies to integrated libraries such as generators for infographics, here, of course, attention should be paid to accessibility. However, WebAIM Wave does not check this separately. It would make sense to separate real website content and content from external sources such as advertising networks, which would allow a more realistic evaluation. I don't know whether this is always technically possible, but the results are simply not meaningful because you don't know

Let's take a closer look at the errors (the numbers refer to an older WebAIM study):

  • 86 percent with errors in contrast: As noted above, no AA criterion
  • 66 percent images with missing alternative texts: This is probably about externally integrated content over which one has no great influence, the same applies to links without text.
  • 53 percent with missing form labels: Annoying indeed, but that can only be judged in context. For example, when it comes to the search field and there is only one field, this error is not that bad.
  • 28 percent missing document language - completely irrelevant, since most users of the websites are likely to be native speakers. The language attribute is pretty much the biggest crap the accessibility experts have ever come up with.

No page is without flaws

The one million most-visited websites are probably maintained by larger teams. It can always happen that individual editors make mistakes: be it the faulty integration of a widget, the wrong nesting of headings or forgetting the alternative text. He who is flawless cast the first stone on WebAIM.

This means that a single mistake by an editor can result in the website failing the WCAG. You can find it useful, but it is not relevant in practice.

So 96 percent of all websites have errors, it should be closer to 100 percent. Anyone who has ever evaluated websites knows that you can find errors if you search for them specifically.

In the end, however, it is not about technical perfection, but about the fact that persons with disabilities can use the website. The WebAIM study actually says nothing about this.

Nobody claims that all websites are perfectly accessible. But the claim that 98 percent of websites cannot be used by disabled persons is just nonsense. WebAIM does not state this explicitly, but suggests it through the overall presentation of the communication. Sheri BYRNE-Haber writes “98% of websites are completely inaccessible.” on page 33 in her eBook “Giving a damn about accessibility.

To be clear, it's good to have this amount of data. Making the raw data available for research would be even better. The nonsense consists in the conclusions WebAIM suggests and others draw from it.

The problem is that if a single bug is found, a website is already non-compliant. Errors are not weighted. So there is no difference between an image description is missing somewhere and the contrast of the entire page is missing, both are errors, except that one usually plays no role and the other has a huge impact. In this sense, a tiny error in the code has the same meaning as a cookie message that cannot be hidden with the keyboard. The former doesn't matter in practice, the latter prevents a number of persons from using the site. That can't be a reasonable benchmark.

Motivate or demotivate

A customer tried to persuade me to mention the study in one of the training sessions. I refused for the above reasons. But also because I think the signal is fatal. The study can show that others do not do it better than you do and then motivate you to do more.

In my opinion, however, it has a demotivating effect. Doesn't it say that WCAG 2.1 AA is essentially unworkable? And that with websites, some of which may have a six-figure budget? If giants like Amazon or the New York Times don't manage to make their websites accessible, how will the local self-help association succeed? In my opinion, such studies encourage fatalism because they suggest that there is little progress.

In fact, the only benefit I see is that a large amount of data is generated here. This allows comparisons to be made and developments to be identified over time.

From my point of view, comparing the websites with each other makes no sense, websites are complex or less complex, it would not make sense to compare a simple media site with an online shop.

It seems to be new that WebAim forms categories, but maybe that slipped through my mind. Under Site Categories you can track different industries, their average error rate and development.

In fact, the other statistics are much more interesting: What is the relationship between the system/framework used and the error rate? Do websites with ads have more errors than those without ads?

The raw data from the analysis would be interesting for researchers, but WebAim does not seem to want to make it accessible.

What is the purpose of this study?

Basically, I appreciate the colleagues from WebAIM. I am all the more amazed that they are publishing such a study. What I write here is, so to speak, the small 1 by 1 of accessibility and of course also known to those responsible.

Basically, I only have two explanations: Either they actually believe in the quality of their tool so much that they simply ignore the points mentioned above. Or - I suspect - the study is a pure PR gimmick. That's pretty handy for a quick message: "96 percent of all websites exclude disabled persons". Can be packed wonderfully in a headline. The fact that this study was published by WebAIM says little about the attitude of WebAIM specialists: It would not be the first time that marketing has done something different from the specialist department. Even digital accessibility is not a welfare, but a business like any other.

This has little to do with reality. At least most of the text-heavy offers can be used well, even if they have minor shortcomings in terms of accessibility. Any given website may not be usable by a subset of disabled persons, but that has relatively little to do with the WCAG score.

And I'm also not sure if they did a service to accessibility. It amazes me that WebAIM thinks they need this kind of PR. Even well-known accessibility specialists disseminate the study uncritically - which does not mean that it makes sense. I can only assume that this is about self-marketing or that they are not able to assess the quality of such studies. Or – that's my guess – they haven't read the study at all. The problem with such analyzes is often that only headlines or summaries are read. The accessibility professionals share the results because they can use it to prove their right to exist.

Read more