Here’s an excellent commentary on website usability testing written by Robert Hoekman Jr in the alistapart blog for web designers.
A lot is cautionary. Good important stuff, most of which won’t apply to the small organization engaged in do-it-yourself usability testing on nonprofit (or other) websites.
I’ve engaged in several usability sessions for such groups and never saw any risk of the issues Hoekman mentions. But they’re still well worth reading, because they do point to bear traps out there in the usability testing world, situations where you can easily generate highly unreliable results that can mislead your web development.
The article highlights three key benefits that in simplest forms justify every small-scale usability testing effort:
1) Usability testing has high shock value. Everyone goes into testing fully confident that everything is just peachy, intuitive, easy, etc. A couple of rounds of testing shows the glaring problems that those who developed the site were too close to see.
2) Usability establishes trust with stakeholders. Hey, just doing usability testing elevates your standing in the eyes of boards etc who don’t know much about the web and often doubt your knowledge.
3) Usability testing is, as Hoekman frames it, part of the “triangulation process” that includes org goals, user goals, other metrics, etc, to gain a more complete picture of what you’re doing on the web, and how good a job you’re doing from multiple vantages.
The article also includes a number of tools to identify trouble spots, like “5-second tests” (which I’ve seen work remarkably well) and tests involving click stats.