Augumented/Virtual reality and digital Accessibility

Augmented reality is in itself unspectacular: If you take a photo of a building like Cologne Cathedral, for example, information about this building should be displayed on your cell phone in real time.

On the other hand, the expansion possibilities of this technology are practically unlimited. In conjunction with a GPS receiver, a blind or visually impaired person could find out which building they are standing in front of and whether they are in the right place. GPS or other location data is necessary because a blind person will probably not be able to photograph a building properly. In order to record sufficient visual information, the viewing angle, distance, lighting and image sharpness must ultimately be correct. It would also be conceivable to compare data with Google Streetview or similar services, although this would probably be too complex today due to the computing power required.

A clever combination of different factors and information sources such as GPS, panoramic photography, cell phone location data and Google Streetview should provide enough information to clearly identify a building or a location on a street.

For example, a restaurant's opening times and menu could be viewed online. After all, no one likes to ask about the menu and prices. Blind people have a bad hand here because they simply cannot read the map.

It gets more interesting with simple things like bus and tram timetables. Even sighted people can often not read them because they are dirty or the print is simply too small. You don't want to do the fun of looking for the right timetables on the websites of the providers on the computer or mobile. Blind people are lost here, unfortunately many sighted people cannot read - or understand - the timetable. Here, however, tactile QR codes would also help, which simply had to be photographed with a smartphone.

A really annoying problem is buses, trams or trains that are late. There are no announcements at small stations, so nobody knows what's actually going on. On the other hand, the deaf have the problem that they do not hear announcements and thus miss out on important information.

The connecting of geographic locations and related websites could become one of the most interesting applications in the near future. And also bring a piece of autonomy for many blind people and other groups of disabled people.

Virtual reality as an alternative to a lack of accessibility

Virtual worlds like the Metaverse of the Meta Group are admittedly still a thing of the future. But they offer great opportunities for disabled people.

The problem of digital accessibility has not been solved, but it appears to be solvable with manageable means. The situation is different with two other major areas: housing and mobility.

In general, there is a lack of countless apartments, especially due to the high level of immigration caused by the various influxes of refugees. For immigration into the labor market, additional adequate housing would still be necessary in the metropolises. The issue of accessible housing is not mentioned at all, the market has been wiped out and our various governments are failing to solve the problem. Investments amounting to billions and appropriate skilled workers would be necessary - neither of which is foreseeable.

We find the same tragedy in mobility: there is progress, but too little and too slow. Older and disabled people with mobility problems compete with commuters, students and other people for scarce mobility resources; ancient buses, trains, stations and trains make getting on and off a torment.

At the same time, we long for human closeness - at least most of us. The big disadvantage of the periphery is the limited opportunities to meet friends and use cultural facilities. Remote work is a good and sensible thing, but you want to at least meet your colleagues in person every now and then. Online meetings are good, but not optimal.

Virtual worlds could solve many of these problems. As I wrote elsewhere, remote work is interesting for people with mobility restrictions as long as it is inclusive, meaning you are not disadvantaged compared to people working on site. The location problem could be solved better with VR than with online meetings. You still don't have to do without in-person meetings, but they should take place less often. I remember the fantasy saga “Otherland” by Tad Williams. This story featured three disabled people who were able to partially overcome their disabilities thanks to the VR world that was readily available there.

And of course the whole topic of leisure activities - that too is a challenge thanks to the lack of accessibility. This would also be possible with the corresponding VR. Sure, you may never feel the atmosphere of a live festival through VR wearables, but it's a compromise and better than nothing. People tend to forget that such events, including tickets, travel, accommodation and so on, often cost as much as a short vacation; money, or the lack of it, can also be a barrier.

How accessible are VR/XR today?

And now the main question: can VR/AR be accessible? At least when it comes to people with motor impairments, visual impairments and people with hearing loss/deaf people, there shouldn't be any major problems here. Of course, you would have to make sure that zoom, voice and motion control, subtitles and all the other existing tools can still be used and that the appropriate interfaces are available.

There are special challenges for blind people when virtual worlds are designed primarily visually. I enjoyed playing computer games until the 2000s: Civilization, Monkey Island, Need for Speed and so on brightened up a few afternoons for me. After that, my vision became worse and the games became so visually complex that it was no longer possible. But because I come from the sighted world, I haven't been able to do anything with audio games until today. Computer games are 95 percent visual and maybe 5 percent audio.

If we experience the same thing with VR/AR - I'm afraid - then it won't work for blind people. That means you also need clean acoustics: When I sit at a virtual conference table, the voice of a person speaking has to come from the appropriate direction or from above or below, 3D sound. Classic stereo headphones are not enough. The visual world - well, you can't really make it accessible to blind people, either real or virtual, at least not with technology that is available for the foreseeable future.

The much more complex development of tools for haptic feedback would also be urgently needed. This seems to be a thing of the future at the moment, but if no research is carried out, nothing will happen. And conversely, this could also be interesting for sighted people. It's a shame that in an already visually oriented world, VR is purely visual and a little acoustic, but hardly tactile.

But no one is saying that VR simply has to copy reality. Where you would distribute printed handouts, you can distribute digital documents in VR. A virtual screen can be filled digitally and read out by the voice output and so on.

Usage scenarios for VR/AR and disabled or elderly people

In general, AR/VR can be used, for example, to carry out physiotherapeutic exercises correctly without external instructions. The movement sequences could be monitored and corrected using a suitable camera or sensors.

The combination of VR/AR applications with text or voice-based chat systems such as ChatGPT is also interesting. The big advantage is that users can ask the system questions in natural language and are not just dependent on the system’s output.

In a work context, VR/AR can be used to make decentralized meetings more personal than is possible with video communication today. Meetings with VR can be closer to real meetings than video-only meetings and offer additional accessibility options.

In general, the combination of different developments is exciting: Blind people can do many interesting things with their smartphones, but it is impractical to hold the smartphone in their hand all the time. In addition, many applications such as text search or outdoor object recognition are very resource-intensive and error-prone. The combination of AI and VR could significantly improve both the required capacity and the recognition of text and objects.

Blind people

In everyday life, AR could be used to improve orientation and exploration. Examples are:

Detection of obstacles and their dangers - for example, a moving bicycle is more dangerous or requires different strategies than a parked bicycle. So far, obstacle detection systems have only been able to warn very roughly and not provide any information such as “it’s best to keep to the left”.
Recognition of objects in the environment in real time including text recognition. For example, it is good for blind people to know whether the door of a building or a vehicle such as a train is in front of them or in their field of vision. Text recognition can be helpful for recognizing street names and numbers, information on bus and train displays, and so on.
In the computer area, identifying and describing user interfaces would be interesting. So far it has only been possible for blind people to recognize the arrangement of UI elements on screens on touch screens. A prerequisite is also that GUIs have been developed to be accessible, which is not always the case so far.

For blind people, however, it is important that information is conveyed either auditorily or haptically. Even with residual vision, visual stimuli cannot be processed the way sighted people do. For example, 3D effects do not work if the vision in both eyes is not the same, which is the case for many visually impaired and blind people.

Deaf and hard of hearing people

For the deaf and hard of hearing, recognizing spoken language and sounds is particularly important. Speech can come from conversation partners, for example, but also from announcements. It could be converted into subtitles or sign language in real time. However, sign language will probably not be implemented automatically at the moment.

If the people cannot express themselves verbally, converting sign language into text or as voice output would also be helpful for the conversation partners. The next iOS currently offers the option of adapting artificial voice output to your own voice.

Paraplegics

There are opportunities for paraplegics to explore places that they cannot physically reach or can only reach with difficulty. This includes buildings that are not accessible, but also environments such as mountains or regions that are otherwise not wheelchair accessible, such as moors. The visual and auditory channels can be operated. However, the smell dimension of the respective location is still missing to complete the picture.

The control could be carried out via speech, head or eye movements.

Autism

Autistic people usually have problems with unfamiliar situations or with effects that they find disturbing. The shape of the trigger is very individual; it can be a certain color, a noise like the air conditioning or something else.

Appropriate headsets and glasses could be used in such a way that they filter out or compensate for undesirable effects for the individual so that those affected are no longer triggered. They would then be fully aware of their surroundings, except that the elements that triggered them would disappear.