Stefan from Mindscreen on automating Accessibility Testing of PDF and Websites

This is a transcript of the conversation with Stefan Farnetani regarding the automated testing of PDFs and websites using CAAT. Any inaccuracies are my own.

Domingos: So, welcome to a new podcast episode on digital accessibility. Today, I have another guest with me: Stefan Farnetani from the agency Mindscreen. Thanks a lot, Stefan, for taking the time for this podcast.

Stefan: Yes, thanks for the invitation, Domingos. I’m very happy to be here today.

Domingos: You’re very welcome. Today, we’ll be talking primarily about automation, as Mindscreen offers a solution in that area. We’ll also discuss PDF accessibility, since Mindscreen conducted an automated study on this topic in collaboration with axes4—and, of course, we want to know the results. But as always, let’s start with you introducing yourself to the listeners, Stefan, and perhaps also introducing the company Mindscreen and what it does in the field of accessibility.

About Stefan Farnetani

Stefan: My name is Stefan Farnetani; I’m from Munich and in my late 40s. I founded Mindscreen with a friend back in 1999, in the gap between finishing school and starting university. So, I’ve been working professionally in digital accessibility for over 20 years. Initially, I approached it from the angles of frontend development, design, and UX; over the years, my focus shifted more toward testing and the challenge of embedding accessibility within an organization. What else should I mention? Perhaps for the sake of completeness: I originally studied architecture and worked as an architect for only a few years, but that sometimes explains my quirks and my choice of words.

Domingos: Yes, thank you. Mindscreen is a major player in the German accessibility space. You mentioned that you’ve been involved for a long time, but how did you actually get into accessibility in the first place?

Stefan: Well, I suppose I have to give a two-part answer to that. There’s the personal side and the professional side. On a personal level, I had contact with this topic very early on. For one thing, I’m personally affected: I had an accident when I was two years old and lost my right hand. And yes, I can speak from firsthand experience about what it’s like to operate a computer or build an architectural model with just one hand. It’s not just theory for me. Then there was my uncle, who had Down syndrome. My mother had a lot of siblings—there were nine of them in total—and my uncle was fully integrated into the family. It was a given that he was part of the family; he grew up right alongside us. That meant I learned very early on that inclusion isn’t about special treatment but is simply the natural order of things—provided, of course, that the environment allows for it. That’s my personal perspective, which is why I have very few reservations about the topic.

Then there’s the professional side. Initially, I had nothing to do with accessibility. We started programming back in 1999—HTML, CSS, JavaScript, the old-school stuff. We relied heavily on SelfHTML as a platform for learning HTML back then. I think a lot of people in my generation learned HTML coding through SelfHTML and similar platforms. I’d like to take this opportunity to say a huge thank you to Stefan Münz, who set that up and wrote such excellent content. He also deserves credit for including a chapter on accessibility right from the start. So, the question of "how do I do this right" was already there early on. Once we’d read through it—you start with the first chapters and eventually make your way to the end—we naturally decided that incorporating accessibility was a given. And we’ve stuck with it ever since.

It’s a bit like a virus. Once you get started... Back then, there wasn't as much information available; you’d just see someone with a Braille display at a conference and strike up a conversation. Nowadays, it’s much easier to access all that information, but that "bug"—that infectious urge—remains: thinking, "Oh man, if I tweak this setting here or there, it’ll read out better." It’s truly fascinating and incredibly satisfying. That’s exactly how we got into it.

Domingos: Yes, thank you very much. Then the question would be, of course, what does Mindscreen generally do today? So, what services do you offer?

Stefan: Yes, Mindscreen roughly summarized: You can say that we accompany organizations on their way to digital accessibility. The support can be very different - sometimes it is just a single assignment, sometimes it is support over several years. The latter in particular is our favorite, of course, because we really have the opportunity to take a broader look at how the topic can be anchored within the company. Our services include various consultations, training by role or topic, workshops and maturity assessments of an entire company. We get a few services from friendly agencies, and then of course a lot of testing. So, we test web applications, applications or even ATMs and the like.

Testing is always a bit of a love-hate relationship - I'm sure you can relate to that too. As specialists, we always think: Oh my God, now I'm going to be called in and have to test this. But the child actually already fell into the well, right? We are often called to do this when everything is already too late and everything has already been done. We actually want to move forward. At the same time, testing is our tool; it is one of the most important tools we have to communicate with teams. And that's how I think you have to look at testing a little bit. You shouldn't see it as the “accessibility cops”, but rather as trying to establish a different form of communication. In addition to one-to-one conversations, training courses and lectures where we impart knowledge, we also convey ticket-cast knowledge that can then be distributed asynchronously throughout the company.

Yes, if you think about what testing means, then it is no longer seen as a mandatory task. Now I may have digressed a bit - you see, testing is a topic that I deal with a lot and that we discuss a lot internally: What really helps, what doesn't help? I always say jokingly: No report in the world makes any product accessible. Only when teams sit down and accept and implement what is there does it become a shoe, if you will. And there's always the question: How do I prepare the teams to have as little friction as possible - a braking effect, so to speak - and get things done well? That's always one of our main tasks: empowering teams, empowering individuals to do the right thing. Mmm.

CAAT and automatic testing

Domingos: Yes, thank you very much. Exactly, testing is also a good transition to the next topic, namely CAAT. What exactly is CAAT?

Stefan: Yes, so CAAT... Exactly, maybe let's start at the beginning. CAAT stands for Computer Assisted Accessibility Testing, which is our software, our testing platform for accessibility. There is a certain similarity to the English word for cat, “cat”, and we play with that too – and of course our mascot is also a cat. With the testing software, we have practically combined our two areas of expertise: On the one hand, we know how to communicate with companies and how to do good tests, and on the other hand, we know how to make web applications more accessible. So it kind of made sense that we combine the things.

We originally developed CAAT because we couldn't find a reasonable tool that would allow us to implement our reports more professionally and quickly. We looked around a bit and visited the big companies. Of course they were great products, but they didn't give us the right flexibility that was needed for the European region - they were very focused on Section 508 and the American process. And then we decided that we would make our own tool, initially for ourselves. Then, based on feedback from other experts, we realized that we were really meeting a need and decided to set up the tool professionally.

That is still the most important thing for us today: that we are heavily used by experienced specialists and by organizations that are really serious about implementing accessibility. So really from specialists who are in the business of doing things, who know how to do proper things and what you need to do it. Official monitoring bodies also use us, for example in Germany and Austria, and even blind colleagues use CAAT. It was also important to us from the start to create a truly professional tool for everyone. “All” of course means different impairments, but “all” of course also means different company sizes. So, we didn't want a tool that was only affordable for large companies, but rather a professional tool that was possible with a small budget. If you are a large company and do 400 tests a month, then you pay a little more for the software. But even a small company can afford it. That was very important to us, and so more and more was created around it.

Domingos: Exactly, but that's also a good transition to the topic of automation. I think this is also relatively new. How exactly does the topic of automatically checking websites and PDFs via CAAT work?

Stefan: Yes, without going into too much technical detail - I think the basic idea is understandable to everyone: I would like to enter a domain and then I tell the system: "Please start running." And then there are different types of how you can run across the entire website, the entire domain. Whether you now have free crawling, whether you read the sitemap XML, whether you enter URLs, whether you set restrictions and so on. But the basic idea is always: I just enter the domain, then the system runs over it and I can test 5 pages, 50 pages, 500 or 1000 pages in one fell swoop. That in itself is not a problem. This might take three hours longer if I have a lot of pages.

Yes, and that is basically the idea that is now complementary - i.e. as a pair of opposites to manual testing. There are different use cases. Essentially you can say that you can use it to determine the current state. For example, that I say: Where am I with my project? I run the system once and then I see: Ah, okay, there are the most common errors. I might be able to break down the errors into areas and say, Oh, maybe someone else is responsible for this area because they're performing worse? Based on the types of errors, I can determine: Okay, where did this error come from? Is this perhaps due to technical reasons or is it due to editorial reasons? And I can then react to it. Or just, teams can solve smaller errors themselves because they say: Okay, alternative text - here is the explanation of what I have to do. And you don't have to rush to a specialist straight away. This also enables teams to do a lot of things themselves, so to speak.

Exactly, and then we have the second track, which is continuous monitoring. This means we can repeat tests with the same parameters, determine whether something changes, document it and of course also use it as an early warning system. If I see: “Oh, there were suddenly 300 errors somewhere last week,” I can of course research what happened. Maybe if I'm a large organization like a university: Are there new teams? If they have introduced new software or installed a new plugin in WordPress - and hey presto, accessibility drops, for example. So in that respect there are many options. These are just the basic automatic advantages for now. And then of course you can combine that with manual testing. That's of course the more exciting thing that I can say: I can now combine automatic tests with manual tests and thus create new services.

Domingos: And above all, these errors can then be automatically passed on to ticket systems like Jira.

Stefan: Exactly. You can then export them and have a dashboard that you can share - whether password protected or not. You can then download the PDF things as CSV or Excel, because with the web you usually enter things into a ticket system, whereas with PDFs you tend to follow the individual departments, I'll say. I can't say that I'll turn every PDF into a ticket, they're just different approaches.

And of course we now have various metrics that are simply intended to help you take a good look at things. So, I can look at the types of errors and I can look at the pages - even the pages with the most common errors. I have very in-depth information in the PDFs. So, not only can I see whether a PDF has errors, but like in the PAC - the PDF Accessibility Checker - I can display as much information as possible, right down to the errors, and have them highlighted. We use axesCheck, which is the online version of the PAC, which so far only works on the desktop. We are one of the first software products to use this and have incredibly in-depth information about the PDFs. This extends to the authoring software that was used. So which creator and which producer - I think that's how it's broken down. Then I know, for example: Ah, this was created with Word and also saved as a PDF with Word. Or Word and axesWord were used, or Word and then Acrobat. I can have that displayed, and that can b

e a good starting point for advice, so that I say: Okay, I now have an inventory here and I know: Okay, most of the documents come from InDesign. Then let's investigate how InDesign documents are created in the organization. Are they mainly purchased externally or are there departments for which we need to arrange training or change contracts? So exactly, you can use it very well as a consulting tool. In the past, people did this themselves manually, which was very time-consuming. And now it's just a matter of minutes, in the end you get an overview like this.

Relevance of automated tests

Domingos: Absolutely! You've basically already said it, but maybe we need to emphasize it again: Why is it even important to have automated PDF and website checks?

Stefan: So yes, “important” is perhaps a bit of the wrong question. I can perhaps describe the benefits better: The big advantage of automated tests is scaling. This means that the tests cannot check quite as much as a manual test. I always say: A manual test goes into depth and an automatic test goes into breadth. But this width, this scaling... If I were to say, for example, that I was doing a reduced test manually, then as a human I could maybe manage 20 pages. The automatic test can check - what do I know - 10,000, 100,000 pages in the same time; there is actually no real limit. It's just server performance that you have to pump up.

And yes, that is of course a huge advantage because it frees up my capacity again. The problem we have - I don't even know how many thousands of online shops there are in the entire European Union - is that the number of real specialists is of course limited. That means we can't search through everything manually. The idea now is, of course, that the automatic system does a kind of pre-sorting, a sort of pre-combing, I might say. I always have the image of different filter levels in mind. Manual testing by a specialist is a very large, fine filtering unit. But I tend to use them further back in the process. At the front I place filters and sieves that work like coarse rakes where I can filter out branches and coarse material. These are the automatic tests. If I switch this on beforehand, then this large filter unit - the specialist - is even more efficient because he can then really take care of the real problems and doesn't spend his time on standard questions. I'm happy to explain

how to write alternative texts for the 15th time, but it's not efficient. A machine can now actually do this and can explain it quite well and find the problems reasonably well. That's exactly the benefit, I think.

Of course, you should never forget: there are limits. I've mentioned it a few times now - automated tests can't do everything. I think you can sing a song about that too. Depending on who you ask, this number always fluctuates between 35% and 40%. This improves a bit with AI, but I don't want to get into these AI dreams right now. To be honest: the improvements can really be measured in percentages. Instead of 30%, we might be able to achieve 32%, but that comes at an incredibly high price. When I consider how much server computing power I have to throw at it, the ratio is still somehow not good. In the end, no test can avoid manual testing. This is the only way to formulate truly profound insights.You could also say it the other way around: An automatic test can never tell me whether a page or a product is accessible. All he can always tell me is that certain places are not barrier-free. That doesn't sound like much and is perhaps a bit frustrating, but it is incredibly exciting and it is amazing how much we can derive from this data alone. So as I said: This visualization of structures, error clusters, error types and connections is of course important if we don't just chase after the individual error, but rather fight the cause instead of the symptoms. That we say: "Ah, okay, the following software has been used here and that's why this error keeps coming in. Let's see if we can replace this software." You can do that very well with these, I would say, somewhat “duller” analyses. So we haven't even reached the end of the line yet, as far as patterns and problems in an organization can be seen.

Domingos: I think a lot would be gained if developers simply used the testing tools that already exist in the developer pipeline and, of course, fixed the errors. So use ARIA correctly, use valid HTML or see that a label is missing. These are the most common problems: Contrasts, these are all things that can be found relatively easily automatically. If they would fix this beforehand, I think we would all be grateful.

Stefan: Exactly, and that's this point of relief again. Then you and I can deal with topics that are really tricky now and don't have to answer the standard questions. So like I said, I have nothing against it at all, I really enjoy doing it. We're back at Barcamp this month, which is explicitly aimed at inexperienced people, and I really enjoy introducing newcomers to the topic of accessibility. I like to do that, but sometimes you feel like you're listening to your own record because you say, "Hey, I've told you that before - not just once, but 20 times." And it doesn't matter if I tell like-minded people. But if you then repeat it again in the same organization with the next team and then repeat it over and over again, then you think: yeah, okay, then maybe you need to introduce a tool at some point.

Exactly. So I'm not an advocate who says everything will just be automated. But automated tests obviously have a lot of leverage. And what's important is that the data we get there needs to be interpreted. And yes, of course, CAAT also provides a certain interpretation to perhaps help newcomers and organizations that do this independently. But of course a specialist with several years of experience can currently read this data much better. That's why we always recommend working with specialists. Preferably automated tools, but still a framework contract for a specialist who may come in to advise at longer intervals, so that you still have this expertise to interpret the data and perhaps set the right priorities. Exactly. Me and testing... Now we've gotten a little lost in testing again.

Mass test from authorities PDF

Domingos: Yes, no problem at all. One topic I wanted to address: You also carried out an automated check of accessible PDFs with axes4, I think in the spring - you already mentioned axesCheck. What was the basic insight you gained from it?

Stefan: The basic insight... Yes, maybe very briefly, so that you can properly assess the basic insight: We carried out a relatively large-scale study and analyzed almost 70,000 PDF documents from the public sector. We found and roughly analyzed around 120,000, but we then classified 70,000 as relevant to the study and then tested them all with the PAC or axesCheck. The result was quite sobering: Only 9.5 percent of the tested documents that fell into this analysis period were PAC-compliant. This means that they meet the machine-verifiable requirements of a PDF document in the first place.

Well, now I'm putting it in such a complicated way. This means that we also have to deduct an unknown proportion from the 9.5 percent, because machine-verifiable compliant does not automatically mean barrier-free. This means that the actually accessible documents are less than 9.5 percent, and that is really, really frightening - since the public sector has had to do this for years and we have only evaluated PDFs that were published after the legal deadline. In this respect, this also corresponds somewhat with the statements of the official BFIT association, which also determined around 9 percent.

But we still tested different sites. The BFIT-Bund is only responsible for the federal government; We formed four clusters, tested a total of 63 websites and formed the clusters “supreme federal authorities”, “social insurance providers”, “health insurance companies” and “cities”. The highest federal authorities, as of course the most important organs in Germany, also performed best. I think this is also because they were tested again and again in the previous study by axes4 from 2016 to 2018. At that time, only federal authorities were tested. So there has already been a certain level of awareness. The cities did the worst, which I personally find quite bitter, because the cities are the first port of call for most people. No, hand on heart: How often have you recently been on the side of a federal ministry, such as the Federal Environment Ministry? This is rather unusual. You are on the city side because - what do I know - you are looking for the bulky waste plan, want to see even

ts or have to change your address. You are on the city side, and so far there has been very little large-scale analysis. And that's exactly what we've done now.

It was important to us that we also made a data-driven contribution with this study. There are a lot of studies that are currently being shot out and sometimes ten pages are tested or only the first page of 20 pages. None of these are reliable studies. We really wanted to have a study that would allow us to advance the discourse in our community and see: Where are the problems really, what comes out? What's particularly exciting here is that we also evaluated the authoring software. So we can see what percentage of these almost 70,000 documents were created from Word or from InDesign - where the focus is practically in the public sector. Yes, and then of course also curiosities like photocopiers that generate PDFs or service providers like Canva.Stefan: The basic insight... Yes, maybe very briefly, so that you can properly assess the basic insight: We carried out a relatively large-scale study and analyzed almost 70,000 PDF documents from the public sector. We found and roughly anal

yzed around 120,000, but we then classified 70,000 as relevant to the study and then tested them all with the PAC or axesCheck. The result was quite sobering: Only 9.5 percent of the tested documents that fell into this analysis period were PAC-compliant. This means that they meet the machine-verifiable requirements of a PDF document in the first place.

ts or have to change your address. You are on the city side, and so far there has been very little large-scale analysis. And that's exactly what we've done now.

Follow mindscreen

Domingos: Yes, definitely. The last question: Where is the best place to follow you?

Stefan: That's a good question. So, we are definitely on LinkedIn - either me as Stefan Farnetani, the company Mindscreen or the software CAAT. And then we always publish which events we are at on our website Mindscreen.de. We are relatively busy – me and my wife Annett Farnetani. You can always talk to us and meet us there. Exactly. And otherwise through the usual suspicious channels, but those are the most important ones and I'm always happy.

When I personally just take a quick look: What do I have next? We now have the Barcamp in Tübingen, then we have “City, Country, Administration”, I think that’s what it’s called in Berlin, which is now before the summer break. Then Linz, the ICT forum in Linz. Then we have the summer break, and in the fall I can't quite figure out where I'll be soon. But we're sure we'll definitely have a stand at SightCity in Frankfurt or S-CON in Berlin - together with axes4 and IAAP DACH, as far as I know, and a few others. I'm looking forward to it, it's always a really great event. Exactly, so I'd be happy to do it on site and otherwise on LinkedIn. Yes, I look forward to feedback!

Domingos: Yes, well, thank you very much for this interesting conversation. I think if you are interested in this automation in CAAT or in CAAT in general, then you can simply contact us. You also do demos regularly, if I remember correctly.

Stefan: Exactly, every two weeks, alternating in German and English, and otherwise just write to me. After the demo there is always a free demo account. You can also have the demo account without a presentation. So for those who are already experienced testers, they usually get in quite quickly. I still recommend the demo briefly, but you can have it that way too. Exactly, that's the usual thing. Take a look at the page, there are also the prices, and otherwise write an email. We always respond pretty quickly.

Domingos: Exactly. Yes, very cool, thank you very much for the interview. I'm definitely keeping my fingers crossed that things continue to go well for you.