Summary: Automated web accessibility evaluation tools are hard to trust, understand and only provides feedback on a small amount of factors that influence accessibility. Also, a unified web evaluation methodology should be adopted to provide consistent results across tools.
Introduction
When you start working with web accessibility as a site owner you will typically be exposed to online accessibility evaluation tools recommended by your supplier. These tools typically let you enter a link to a web page after some automated checks are made you get a report of all the errors that were found.
While these tools may be a good way to convince your organization to increase funding for accessibility work you should be careful how you interpret their results. Your website may be good enough already and if you try to fix all reported errors you may be spending money in the wrong thing.
As an example of how difficult these tools may be to trust and understand I have selected some of the more popular ones from the Web Accessibility Initiative list of tools and performed some tests.
Letting accessibility evaluation tools test themselves
Here, each of the tools were pointed at their own starting page. When possible WCAG 1.0 triple A settings were selected.
| Tool | Errors | Warnings |
|---|---|---|
| Deque Worldspace | 6 | 6 |
| WAVE | 0 | 0 |
| Functional Accessibility Evaluator | 0 | 0 |
| AChecker | 0 | 115 |
| Eval Access | 0 | 124 |
| Cynthia says | 1 | 0 |
| TAW | 3 | 0 |
Very few errors as expected. After all, these tools are built by professionals and I would expect them to have checked their own service.
Letting them test each other
So, what do they say about each other? Only one way to find out.
| Tool | World- space |
WAVE | FAE | AChecker | Eval Access | Cynthia says | TAW | Sum |
|---|---|---|---|---|---|---|---|---|
| Worldspace | 6 | 0 | 3 | 4 | 14 | 3 | 23 | 53 |
| WAVE | 43 | 0 | 19 | 11 | 9 | 2 | 7 | 91 |
| FAE | 4 | 0 | 0 | 3 | 2 | 1 | 1 | 11 |
| AChecker | 52 | 0 | 0 | 0 | 14 | 4 | 7 | 77 |
| Eval Access | 13 | 0 | 3 | 6 | 0 | 1 | 2 | 25 |
| Cynthia says | 10 | 0 | 11 | 16 | 13 | 1 | 13 | 64 |
| TAW | 8 | 0 | 0 | 10 | 1 | 1 | 3 | 23 |
It is understandable that people find it hard to make use of web accessibility evaluation tools. How are you supposed to interpret these results? None of the tools are in agreement on any of the tested pages. Similar results would be returned for most pages you evaluate.
Observations
- WAVE didn’t find any accessibility issues in any of the pages. Also, WAVE would display a fun error message if you try to make it check itself by URL (I had to copy and paste the source instead).
- The output from many of the tools are really hard to interpret, especially if you are new in the field of web accessibility. The TAW tool, for example, displays tiny icons all over the page and you have to hover them to see what they mean.
- Worldspace uses nested tables for layout (something that WAVE didn’t complain about).
What would be your advice for a site owner that wants to increase accessibility on his/her website? How can they check if their supplier did the right thing when creating the markup?
(Please leave a comment or send me an email if you find any errors).
Comments are closed.