Pitfalls of Web Accessibility Evaluation Tools

Summary: Automated web accessibility evaluation tools are hard to trust, understand and only provides feedback on a small amount of factors that influence accessibility. Also, a unified web evaluation methodology should be adopted to provide consistent results across tools.

Introduction

When you start working with web accessibility as a site owner you will typically be exposed to online accessibility evaluation tools recommended by your supplier. These tools typically let you enter a link to a web page after some automated checks are made you get a report of all the errors that were found.

While these tools may be a good way to convince your organization to increase funding for accessibility work you should be careful how you interpret their results. Your website may be good enough already and if you try to fix all reported errors you may be spending money in the wrong thing.

As an example of how difficult these tools may be to trust and understand I have selected some of the more popular ones from the Web Accessibility Initiative list of tools and performed some tests.

Letting accessibility evaluation tools test themselves

Here, each of the tools were pointed at their own starting page. When possible WCAG 1.0 triple A settings were selected.

Self test result
Tool Errors Warnings
Deque Worldspace 6 6
WAVE 0 0
Functional Accessibility Evaluator 0 0
AChecker 0 115
Eval Access 0 124
Cynthia says 1 0
TAW 3 0

Very few errors as expected. After all, these tools are built by professionals and I would expect them to have checked their own service.

Letting them test each other

So, what do they say about each other? Only one way to find out.

Cross test result (Application that performed the test in the header row)
Tool World-
space
WAVE FAE AChecker Eval Access Cynthia says TAW Sum
Worldspace 6 0 3 4 14 3 23 53
WAVE 43 0 19 11 9 2 7 91
FAE 4 0 0 3 2 1 1 11
AChecker 52 0 0 0 14 4 7 77
Eval Access 13 0 3 6 0 1 2 25
Cynthia says 10 0 11 16 13 1 13 64
TAW 8 0 0 10 1 1 3 23

It is understandable that people find it hard to make use of web accessibility evaluation tools. How are you supposed to interpret these results? None of the tools are in agreement on any of the tested pages. Similar results would be returned for most pages you evaluate.

Observations

  • WAVE didn’t find any accessibility issues in any of the pages. Also, WAVE would display a fun error message if you try to make it check itself by URL (I had to copy and paste the source instead).
  • The output from many of the tools are really hard to interpret, especially if you are new in the field of web accessibility. The  TAW tool, for example, displays tiny icons all over the page and you have to hover them to see what they mean.
  • Worldspace uses nested tables for layout (something that WAVE didn’t complain about).

What would be your advice for a site owner that wants to increase accessibility on his/her website? How can they check if their supplier did the right thing when creating the markup?

(Please leave a comment or send me an email if you find any errors).

Comments

  1. Ryan says at 2009-04-23 21:04:

    My boss and I were just talking about tools to add to our arsenal, thanks for doing this comparison. As an aside, I picked one of the tools (ATRC) and tested it with our site–I get an out of memory error. I know our site needs work, but that seems a bit silly. Guess that one fails before it even starts :o).

    I use some other tools, such as Fangs as a Firefox add-on and the WebbIE browser, and I agree that we have to keep the data that we get from all these tools in perspective. My organization has been working on a grant to develop a web-based accessibility workbench to test new accessible technologies (not limited to websites), so there is definitely a lot of work to be done in this area. Our approach is to link new technologies with people with disabilities in order to obtain real word data, but we just got denied the grant, so that idea in on hold for now.

  2. Jared Smith says at 2009-04-23 22:04:

    Very interesting article. The problem isn’t so much a lack of standards for evaluation – it’s simply differences in ways that the feedback from the tools are presented.

    You noted that WAVE indicated 0 errors for all of the tools. That’s because WAVE reserves errors for things that are clearly accessibility issues. I think you’ll find that nearly all of the “errors” reported by the other tools are not really errors at all or are things that make no difference whatsoever to accessibility.

    The vast majority of the “errors” found by other tools on WAVE are because we correctly have links in lists, we used italics in one place for true visual styling, and we (heaven forbid!) don’t have black text on a white background. Oh please!

    Our approach with WAVE is to focus on true accessibility by facilitating human evaluation. We do our best to indicate true inaccessibility where we can, and then give the user lots of inline details about the content so a human can determine how accessible the document truly is.

    If there’s anything wrong with tools, it’s that they force silly or hardline interpretations of what accessibility is on their unknowing users.

  3. Peter Krantz says at 2009-04-23 22:04:

    Jared, thank you for your feedback. I believe that WAVE does the right thing in that it is conservative with what it complains about. I used a similar approach in RAAKT.

    I especially agree with your last paragraph and that is why I believe many newcomers to web accessibility find it really difficult.

  4. Karl Groves says at 2009-04-24 15:04:

    I’d also submit that any article of this kind should caution the reader that reliance upon automated testing alone is an insufficient means toward assessing compliance and that nearly all automated tools suffer from the weakness of checking document source as a string. In the former case, determining compliance should include manual normative testing as well as functional testing. Unfortunately too many users of such tools think automated testing is all they need. In the latter case, today’s more advanced enterprise-class web sites and web based applications increasingly rely on client-side scripting techniques which modify the DOM. Many automated tools simply test the document source as a string, decreasing the accuracy and relevance of their results and potentially missing huge errors.

  5. Tom Babinszki says at 2009-04-30 17:04:

    It always comes down to a personal decision at the end. I like automated tools to tell me what could go wrong with the site over-all, at a glance, but I never replaced manual testing with it. A tool, for example can tell me if an image has alternative text, but it will be a personal decision if that text is sufficient to describe the image. These decisions will slightly differ among testers.

  6. Vincent François says at 2009-05-01 17:05:

    Hi,

    The automated tools are there first to help the manual testing, removing the load of repetitive tasks, and secondly to help peoples to “measure” quickly how far they are from “something correct”.

    Exactly like the HTML code validator.

    But, this useful article – and the comments too – point the difference beetween “true” accessibility issues and “hardline interpretations” direct from the book.

    The rules are there to cover a very large scope of disabilities and as content provider we can’t know in advance wich ones will be encountered with our next visitors.

    If we are not able – for technical or money reasons – to meet every success criteria, who have to be responsible to take the decision : a automated tool, enough smart to be loose, or a human, enough formed to decide?

  7. Richard Fortune says at 2009-06-18 04:06:

    Hi all,
    very interesting article. Its an area that the web-dev industry struggles with as a whole. Treating accessibility as a checklist and assuming technology is the answer to our problems.

    It most certainly isn’t, but when working with code we should certainly be closing any gaps before our content ends up in the public domain. By closing gaps I mean locking your code up so it is valid and then having real users validate your sites flow through real world testing. This is often a lot cheaper than people suspect and reaps results infinitely more relevant.

  8. Raph de Rooij says at 2009-08-19 15:08:

    Maybe it’s a bit late to react, but it remains an interesting subject. It’s also the last item on standards-schmandards.com, so I don’t feel too reserved… ;-)

    I’ve tested the various evaluation tools with our own tool. The outcome of this automated test doesn’t say much about accessibility, but at least shows some room for improvement when it comes to build quality…
    Unfortunately, only three of the examined tools offered the possibility to link to the actual test result.

    The quick test confirms Peter’s conclusion: what lacks is a unified web evaluation methodology that provides consistent results across tools.

    Such a methodology exists for version 1.0 of the Web Content Accessibility Guidelines; for our own tool we used UWEM version 1.1.

    Hopefully, funding will become available for UWEM version 2, based on the far more recent WCAG 2.0. It would not only be very convenient to use this specification during the development of the new version of our test tool, but will also lead to more consistency of test results. But I’m not very confident that UWEM version 2 will be developed any time soon…

  9. Mastodon Labs says at 2009-08-26 18:08:

    i’ve been trying the Firefox Accessibility Extension. Looks great so far. It’s for developer creating accessible web content.

    It would be great to be able to rely totally on automated tools for website evaluation, but it’s not possible with the current state of the tools. At some point, people need to do testing. On some level, tools may never be able to fully do the job. Our world is full of exceptions, uncommon circumstance, and change. I use automated tools as mechanism to find trouble areas, then examining the situation for myself. In the end, it comes down to a judgment call.

  10. Pink Zebra Comforter says at 2009-10-18 17:10:

    Isn’t it amazing what turn of events can take place? Appreciate you letting your readers know about this.

  11. Mitra says at 2009-11-09 20:11:

    Hi,

    I am very in Web accessibility and I am conducting a research on web accessibility of UK public sector website. I have been reading
    Central Office of Information(COI)(2008) Delivering inclusive websites [Internet]. Available from:

    COI recommends that website developers and testers should be aware of the limitations and capabilities of the automated tools.

    However, there is only information on performance of Boby and all the previous academic research was performed using Boby automated tools. I am just wondering how the website developers and testers suppose to know about limitations and capabilities of the automated tools except Boby.

  12. شات صوتي says at 2010-01-29 00:01:

    Isn’t it amazing what turn of events can take place? Appreciate you letting your readers know about this.

  13. kareem says at 2010-07-20 21:07:

    Many large organisations I have worked with deploy enterprise solutions such as Compliance Sheriff (http://www.hisoftware.com/products/CS_Accessibility.html), which offer a much more comprehensive audit of your site that some of the free solutions on the market right now.

    Everyone is correct when saying to integrate automated solutions with manual reviews, because no solution in the world can scan against the full WCAG 2.0 checklist.

    However when managing thousands (if not hundreds of thousands) of pages, it is very sensible to deploy an enterpirse tool that can scan whole sites, sub sections, content etc at one time.

    Most solutions freely available are quite limited in what they can show and do not allow any form of customisation, therefore can present a number of barriers to people wanting to understand what the reports show.

    Drop me an email and I can set up some demos for you to look at an enterprise solution and how this may help you see the real value in using an automated tool as well as maunal reviews etc. (User feedback is critical as well)

    kareem@hisoftware.com

  14. yash says at 2011-04-12 22:04:

    All these tools give different results for the same page. I always prefer manual on automated tools.

  15. ranjit mahanti says at 2011-04-13 17:04:

    Interpretation is key here. Life will become very easy if it can be automated completely but tools are lagging in this front.

Trackbacks

Leave a comment

You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Peter Krantz, peter.krantz@giraffe.gmail.com (remove giraffe).