Should Sighted test with screen readers?
In various manuals a screen reader is recommended for sighted testers. But it is rather doubtful that a sighted person learns to use a screen reader fluently like a blind person. And even they would often fail the ultimate test: Turn off the screen and operate your computer completely blind.
Many sighted persons think that because they can install NVDA or launch VoiceOver, they can value the usability of an application for the blind. Let's took a look on the difficulties of screen reader testing.
- Understanding the screen reader output
- Sighted persons see what blind persons do not perceive
- Blindness of sighted persons
- Optimization for the test screen reader
- Why screen readers and not some other assistive technology?
- More on screen readers
Understanding the screen reader output
The output of a screen reader can be divided into two areas. On the one hand, there is the output of visible text. On the other hand, there is what can be called meta-data. For example, the word "New" is useless without a meta-context. Is it a text, a button, an element with a submenu, a radio button? Is it focused or not, is it activated, activatable or not activatable? In the best case, this information is all read to the blind person in addition to the word "New".
Experienced blind screen reader users perceive this information more or less incidentally. It is comparable to the way sighted persons perceive visual components of a user interface. They recognize the nature of an element by its visual design, its status and whether it can be activated or not. If the interface is well designed, you don't have to think about it.
Since visual design remains inaccessible to a blind person, they rely on additional communication of this information. Somteimes this is called verbosity, because you get a lot of information, is partly not of interest in that moment.
Now it rarely makes sense to describe this in detail. We have to cope with a certain amount of background noise, but above a certain level it is difficult to ignore it. So you need as short and compact a summary as possible.
The speech output, or Braille output, is abstract, and it is abstract for sighted persons as well as for newly blind persons or blind persons without much screen reader experience.
Sighted persons see what blind persons do not perceive
Sighted persons are rarely consistent or patient enough to explore an application via keyboard. Moreover, they cannot discard their instinct to look on the GUI.
Let's take a simple example: In the web version of GMail, the actions are only displayed once you have checked a mail. These buttons appear before the actual position of the screenreader. A sighted person sees this of course, but a blind person does not notice it or would look for it after the mail list.
Blindness of sighted persons
In addition there is also the problem of the developer blindness, as you could call it.
On the one hand, it is often the developers themselves who test. But if they are not test professionals, they are rarely neutral. They know what should happen or what should be the output. The user, on the other hand, will not know this.
Second, persons are often more critical of other persons's projects than they are of their own. A test should always be performed by someone who is neutral or even critical towards the matter itself.
Optimization for the test screen reader
Unfortunately, testers tend to optimize for a particular screen reader because that is what they have available, usually Jaws. This is of little use to NVDA and VoiceOver users.
Often we hear the demand to test even with multiple screen readers. Of course, sighted persons can't hardly cope with one screenreader, then they should try it with several and know and understand the differences. Which ones should it be: Jaws, NVDA, Narrator on Windows, VoiceOver on Mac and iOS, Talkback on Android - that would be the minimum. Basically, you would have to look at several versions, at least for Jaws, since not everyone uses the current version. For iOS, there may also be differences in development between iPad and iPhone.
Last but not least, at least every screen reader would have to be tested once with a Chromium-based browser and Firefox. On the Mac, it would even have to be Firefox, Safari and another Chromium-based browser.
Which sighted person masters all these platforms and screen readers sufficiently to get valid results? And who has the patience to do it at all?
This seems to me to be more occupational therapy than serious testing. why not leave it to the professionals, i.e., trained blind persons who are experienced in using screen readers and could do a much better job?
Why screen readers and not some other assistive technology?
For some reason, blind persons are considered very important, often being the only group mentioned in the context of accessibility. But why? Why not the numerically larger number of the visually or motor disabled? Why not the Cognitive disabled? And why test with screen readers and not, for example, screen magnifier, high-contrast mode, or voice control?
Of course, the blinds are an important group. But we should not ignore all the other persons, which also are dependent on accessibility.
As mentioned above, most sighted persons are not able to operate a screen reader the way a blind person would. To navigate through a website once is no art. But which sighted person can fill out a form blindly. So they look, or worse, test with the mouse. What comes out is suboptimal at best.
For the same reason, I wouldn't trust tests done by the newly blind or those with little screen reader experience. They are still struggling with screen reader quirks and therefore have no appreciation of how good the test objects are.
Using a screen reader can be useful to get a feeling for how blind persons work. However, they are unsuitable for testing by untrained persons.