The Fails of the German BITV Test

The BITV test is not a useful tool for measuring accessibility and is of limited value as a seal of quality. Certifications for the accessibility of websites are just as pointless. I explain the reasons in this post.

Note: My criticism is generally aimed at such procedures, BITV-Test or WAKA are only used as examples here, the problem generally lies in the structure of the test procedures. Those who follow the relevant guidelines and allow feedback from disabled people do not need certification or the BITV test. Often enough, the results are used as a killer argument against the criticism of those affected.

Article Content

Snapshot

A BITV test can only be a snapshot. At time X, a test was performed.

And in the meantime, the world has turned three times, features like a cookie message have been added, content has been embedded or removed, navigation has been optimized... Of course you can't run a BITV test for every change. But strictly speaking, at least the component that was changed would have to be rechecked.

Narrow focus on blindness and low vision

A large part of the accessibility scene revolves around the topic of blindness. Their requirements are plastic and essentially easy to implement. Whether it will actually be done is another question.

But what about the physically disabled, autistic, mentally disabled and so on? Their requirements are hardly recorded with the BITV test.

Pointless criteria

Unfortunately, the BITV test contains some criteria that lead to further barriers:

Test step 3.1.2a

Words and paragraphs in other languages marked

If there are words and sections of text in another language within a page, they must be marked up using the lang attribute.
Source

This relates to WCAG criterion 3.1.2

Success Criterion 3.1.2 Language of Parts The human language of each passage or phrase in the content can be programmatically determined except for proper names, technical terms, words of indeterminate language, and words or phrases that have become part of the vernacular of the immediately surrounding text.
source

This is a grossly negligent overstepping of the WCAG criterion. Any blind speech output user will tell you that the marking of individual terms as foreign-language extreme is annoying and superfluous and can even limit understanding.

This is actually highly relevant today: if a WCAG 2.1 success criterion is not passed, the website is not considered compliant, i.e. not accessible according to this rating. A website cannot be considered compliant if the name Jane appears somewhere and has not been marked as English.

What BIK thinks remains their secret.

Technical accessibility is not everything

The BITV test essentially checks the technical substructure of a website. That too makes sense within limits. However, the needs of other disabilities are not covered here either. You don't use the source code but the visual web interface. And if this is not optimized for the user, this is a greater barrier - incidentally also for the blind and visually impaired. Overall, the BITV test completely ignores usability and information architecture issues.

Page-based test methods are not up to date

A test procedure that considers individual pages in isolation no longer makes sense in times when entire applications are running in the browser. Of course you should look at key templates, but the core concern of the user or the key process of the website is more important: be it the contact form, an ordering process or a comment function. Everything where the user has to interact with the website is important. Yes, you should look at the most important templates on the website, but also play through the most important processes. However, the BITV test is page-based.

The BITV test does not do the users justice

While users mainly use smartphones and maybe even tablets, the BITV test is conducted with desktop versions of the website. However, different problems can arise on the mobile version than on the desktop version. The BITV test therefore does not meet current requirements. This is also reflected in the fact that outdated browsers are to be used for the test:

For example, keyboard usability is only checked with the two most common browsers, Internet Explorer and Firefox.
Source

Internet Explorer may still be widespread in government agencies, but it is definitely not among the blind, and even less so among other disabled people.

BITV test invites developers to cheat

A lazy but smart developer simply optimizes the website so that it gets 100 points on the BITV test. This is fairly easy since the procedure is openly documented.

If the contractor knows that a BITV test is pending at the end, he will bet from the outset that as many points as possible will be achieved. This can, but does not have to, contribute to the accessibility of the website.

Stamp it and you're done

The findings from the BITV test or from certifications are often used as a manslaughter argument when a disabled person complains about the website: We did the BITV test after all. What else should we do?

Such procedures can - also because of the costs - only be snapshots. After all, they cannot be repeated every time the page changes. But what use is a BITV test or certification of a site if the measured condition no longer exists.

Closed Society

The responsible group of testers appears as a closed society. The test procedure has existed for more than ten years, but there are only a handful of testers and test agencies, although the issue of accessibility has become enormously important as a result of the 2102 guideline.

The testers are mostly over 50, male and non-disabled - so anything but diverse. To be honest, it's a bit strange that they claim to be committed to accessibility, but are themselves as diverse as an old men's club.

So you artificially reduce the testers and drive up the test costs.

Slightly disabled testers

However, my point of criticism is that there are hardly any disabled people with experience in assistive technologies who carry out these tests. So far I have only met people without disabilities who are responsible for such tests. This is partly for good reason, but users of assistive technology inevitably notice more things than those without a disability. While some tests cannot be performed by the blind or severely visually impaired, some of the steps are suitable for people with disabilities. Which, by the way, could also open up a small source of income. Personally, I have not yet met a BITV tester with a disability.

Is the BITV test meaningless

The BITV test makes sense - in certain situations.

  1. It makes sense during development, because the developers can use it to carry out quality assurance and find serious errors.
  2. It makes sense as a self-test: You can check the status of your website or whether the web agency has generally worked properly.
  3. And it is an opportunity to deal systematically with the BITV guidelines. They are not really suitable for downloading

Bad merging

The connection between the lead organization behind the testing process and the BITV testing website can rightly be described as unfortunate. The company earns its money by conducting such tests, among other things. In any case, this is a clear conflict of interest.

It would have been desirable for the test procedure to be managed by an independent, non-profit-making body. The focus is all too much on enticing website visitors to commission a BITV test. It is a little strange that the company on the page on which the BITV test is discussed not only advertises their services, but can also request a BITV test directly from this company.

Alternatives to the BITV test

In addition to the possibility of carrying out the BITV test yourself, I see two other alternatives:

  • One possibility is an expert test: it can be carried out by a digital accessibility expert. We will typically use a mix of standard testing and heuristic analysis here. It doesn't come out as nice as the BITV test. But the results are much more relevant for practical accessibility.
  • The second and better option is user testing. This can be done with your own employees or during ongoing operations. In the latter case, invite all users to tell you about barriers and problems on the site. You then need to set aside resources to address those reports and make any improvements. Under EU Directive 2102, you would be required to provide accessibility feedback anyway. But such offers can be designed in different ways, according to the motto "Yes, you're welcome" or "Don't worry, we don't care".

Websites are often so complex that user testing is the most insightful, albeit laborious, process. If the resources for such testing are not available, you should set up a feedback mechanism to collect feedback from people with disabilities. Such a feedback mechanism is also provided by the Directive EU 2016-2102, so it has to be established anyway. Furthermore, you can persuade your employees with a disability to take part in such tests, of course as far as possible.

So we have an agency doing a decent job that has adhered to WCAG/BITV guidelines and we have user feedback. That is perfectly sufficient. So put the money that you earmarked for the BITV test into the feedback mechanism, the maintenance and the Quality Assurance.

.

Are the DIAS GmbH and IAAP anti-disabled?

The BITV test has become the standard tool for measuring standard compliance aka accessibility. At the same time, however, he seems to have fallen out of time. This article is a formal criticism of the BITV test, on the content-related criticism of the BITV test. We now only speak of the "DIAS test" since the name BITV or WCAG test suggests a neutrality that is not there. Since the big testing as part of the current EU Directive 2102 is about to start and the BITV test is currently the only instrument available in German, now is a good time to adapt the procedure. In addition, with the introduction of WCAG 3.0 and the revision of EN 301 549, there is also a great opportunity to comprehensively revise the procedure. Update: The DIAS testing association has shown that they are not interested in an update, more openness, participation of disabled people and transparency. We therefore advise against using the DIAS test according to the current status. It is also interesting that DIAS GmbH or the website for the test procedure would not pass their own test. But they consider themselves competent to advise other institutions on accessibility.

Open discussion

As far as I know, the BITV test was originally developed on behalf of a federal ministry. Today it is further developed by a handful of people. One can speculate a lot about who is involved in the development: definitely DIAS GmbH and some people who are also official testers. I do not know whether disabled people or their self-representatives are involved. But this is where the problem starts: The procedure and its development are not transparent. The process of the Web Accessibility Initiative shows how it can be done differently. Here we know who is involved in the development of the guidelines and other documents. The guidelines are open for discussion. The BITV test, on the other hand, looks like a private project. Mistakes can be made here too. Example of this is the language attribute. This also means that the BITV test lacks any legitimacy. It is not commissioned by any public body, no public body is involved and public discussion is not possible. It is also not an industry norm, in which case the legitimacy would be on a more solid footing. Following the WAI recommendations for developing test procedures is nice, but it doesn't add to the legitimacy of the procedure. It should also be made clear at this point: the laws stipulate that websites and content should be made available in an accessible manner. The standard for this is EN 301 549 and thus the WCAG. The BITV test, on the other hand, is not anchored in any law as a benchmark. Accessibility and a fulfilled BITV test are therefore not synonymous. Currently it's just a project of DIAS GmbH and the people that DIAS allows access to. Why isn't there a GitHub repository for this? Alternatively, DIAS could also communicate that it is their private project, in which case no one would think of considering the procedure as a seal of quality. I also don't think the tandem exam makes sense in its current form. The final BITV test should be carried out by two examiners. But is it likely that two experienced examiners will arrive at very different assessments? It is completely incomprehensible to me why one of the examiners should not have a disability relevant to digital accessibility. It seems as if one wants to exclude disabled people from a lucrative field of activity.

Lack of involvement of disabled people

Nothing about us without us is one of the main slogans of the independent living movement. Unfortunately, I don't see BIK actively involving disabled people. On the contrary, I have the impression that they think they know better than those affected, see the discussion about the language attribute. The fact that one or the other disabled person works in any of the organizations involved does not change the criticism. It doesn't automatically mean that this person is also actively involved.

Obscure Weave

The website for the BITV test is operated by DIAS. This gives the process a flavor, to say the least. On the one hand it should be an open procedure, on the other hand the website is not operated independently. The final difficulty is that the BITV test can be commissioned directly on the same page. That tells me indirectly: If you can't do it anyway, let's do it. And it also carries the stigma of lack of independence.

Criticism of the presentation

It may not seem so important at first glance, but the BITV test website is not attractive either. Looks kinda 2010s flavored. For us, content is paramount, but who would blame designers and developers for finding the layout unattractive? How to convince these people to use this tool or to find accessibility attractive? Also, many test steps have apparently not been proofread. There are some slobs in there that a decent spell checker would have found. Another reason why we need an open discussion.

Summary

Let me summarize the main points of criticism of the BITV test:
  • It is not clear who is involved in its development. It is also not clear whether and to what extent disabled people or their representatives are involved.
  • Regardless of this, the procedure does not even have indirect legitimacy, as it is not openly discussed.
  • The procedure contains errors: see the discussion of the language attribute.
  • Criticism from disabled experts is not taken seriously by the testing association.
Incidentally, my impression is that the number of testers inside is kept artificially low. Somehow I can't imagine that only a handful of agencies and freelancers apply there. Now one can argue that it does not want to be an open procedure, but a protected procedure owned by DIAS. You can accept that, although it originally came about as a result of public funding. However, it should then be communicated in this way and not act as if it were a public and legitimate procedure. In my opinion, DIAS GmbH is not living up to its responsibility. On the one hand, she wanted to establish the BITV test as a neutral instrument. On the other hand, she treats the test as if it were hers. In addition, DIAS GmbH and the associated testing association are resistant to advice and are not prepared to respond to criticism from those affected. Basically, I think DIAS GmbH and the BITV testing association are anti-disabled.

Read more