I hadn’t realized how much A/B testing was being done online by ALL KINDS of organizations and enterprises until seeing this article in a recent Wired Magazine. Read it. Won’t take long.
You probably can’t do this kind of testing yourself. Not enough tech resources. Plus most organizations have lists too small to do A/B testing that generates statistically significant results … a big reason I started this blog, hoping the share tests done by big orgs with small.
The Wired article discusses web organizations that are driven by A/B testing in effective — and sometime radical — ways. When you DO have the resources, you can put up parallel web pages and deliver them randomly to users, and read the results instantly. A lot quick than mail programs, for sure.
Fundraising, for the ’08 Obama campaign and others, can fine-tune aspects of their sites to get more email addresses or donations. Consumer outfits can boost sales. Google used A/B to shape their search mechanisms and they way they deliver results to users.
A couple of lessons that all fundraising organizations should learn about testing:
– The risk is making only tiny improvements. If you’re investing in testing, don’t fine-tune when you can transform.
– Data makes the call. The statistical winner wins. Not the head of the organization. The Wired article notes that “A/B tends to shift the whole operating philosophy — even the power structure — of companies that adopt it.” Decisions start being made based on what works. Not what you like.
One theme throughout: Testing proves the truth of the unexpected. What works is not what you expect to work.
Why does some design or technique or copy approach outperform others?
Doesn’t matter. Just be glad you know what works and use it.