Pitfalls of Web Accessibility Evaluation Tools

Summary: Automated web accessibility evaluation tools are hard to trust, understand and only provides feedback on a small amount of factors that influence accessibility. Also, a unified web evaluation methodology should be adopted to provide consistent results across tools.

Introduction

When you start working with web accessibility as a site owner you will typically be exposed to online accessibility evaluation tools recommended by your supplier. These tools typically let you enter a link to a web page after some automated checks are made you get a report of all the errors that were found.

While these tools may be a good way to convince your organization to increase funding for accessibility work you should be careful how you interpret their results. Your website may be good enough already and if you try to fix all reported errors you may be spending money in the wrong thing.

As an example of how difficult these tools may be to trust and understand I have selected some of the more popular ones from the Web Accessibility Initiative list of tools and performed some tests.

Letting accessibility evaluation tools test themselves

Here, each of the tools were pointed at their own starting page. When possible WCAG 1.0 triple A settings were selected.

Self test result
Tool Errors Warnings
Deque Worldspace 6 6
WAVE 0 0
Functional Accessibility Evaluator 0 0
AChecker 0 115
Eval Access 0 124
Cynthia says 1 0
TAW 3 0

Very few errors as expected. After all, these tools are built by professionals and I would expect them to have checked their own service.

Letting them test each other

So, what do they say about each other? Only one way to find out.

Cross test result (Application that performed the test in the header row)
Tool World-
space
WAVE FAE AChecker Eval Access Cynthia says TAW Sum
Worldspace 6 0 3 4 14 3 23 53
WAVE 43 0 19 11 9 2 7 91
FAE 4 0 0 3 2 1 1 11
AChecker 52 0 0 0 14 4 7 77
Eval Access 13 0 3 6 0 1 2 25
Cynthia says 10 0 11 16 13 1 13 64
TAW 8 0 0 10 1 1 3 23

It is understandable that people find it hard to make use of web accessibility evaluation tools. How are you supposed to interpret these results? None of the tools are in agreement on any of the tested pages. Similar results would be returned for most pages you evaluate.

Observations

  • WAVE didn’t find any accessibility issues in any of the pages. Also, WAVE would display a fun error message if you try to make it check itself by URL (I had to copy and paste the source instead).
  • The output from many of the tools are really hard to interpret, especially if you are new in the field of web accessibility. The  TAW tool, for example, displays tiny icons all over the page and you have to hover them to see what they mean.
  • Worldspace uses nested tables for layout (something that WAVE didn’t complain about).

What would be your advice for a site owner that wants to increase accessibility on his/her website? How can they check if their supplier did the right thing when creating the markup?

(Please leave a comment or send me an email if you find any errors).

Comments are closed.