This editorial is from a controversial issue on measuring usability. This special issue covered CIF usability testing (a seemingly innocuous if important topic). Upon publication many took strong esxception to our coverage of CIF testing. But perhaps these critics also took exception to our criticizing a practice all too often employed: how to lie with statistics and its designer equivalent using too high a fidelity to overwhelm an audience. --Jonathan
This issue is primarily about usability and metrics, measurements, and what we do with them. And thus our rant: deceiving with numbers.
At university you may have had the pleasure of a reading assignment instructing you to read and memorize a slim volume called How to Lie with Statistics [1]. This is a classic that presents the unimaginable- convincing people of something by manipulating unfamiliar concepts that sound scientific, and thus authoritative. One could, of course, achieve the same end via bullying or mesmerizing charisma. But lying with statistics is a venerable process engaged in by politicians, advertisers, and scientists- and even usability professionals.
One of the ingenious cheats of statistics is to "prove" that you are correct by accentuating the positive and eliminating the negative. To be a master of deception you do, of course, need tools. Not of the rabbit-and the-magic-hat variety. A simple high-fidelity prototype can do the trick. We can call this type of deceiving with numbers High Fidelity Deception, and it goes something like this:
Reduce the overall feature set of a target product.
Design a prototype that covers some of the features.
Ensure that it is attractive enough to resemble a finished product with a dose of glamor: nice animations, transitions, professional color palette (for extra points, make it blue-you do know about that, right?), refined font, and upscale branding elements.
Show it to a few high-placed individuals who (this part is important) don't have the time or inclination to dive into the details.
Get their enthusiastic endorsement.
Proceed to formalize this into a final product regardless of the paucity of credible detail.
You can get excellent marks on the prototype: high levels of user satisfaction, compliments on the prototype's refined sensibilities, and unequivocal approval to proceed. This prototype is, in fact, merely an attractive shell, but it passes muster, and you can quantify its success by repeating what percentage of users who have seen it were "wowed" by it (don't mention the actual number of users, though-after all, two out of two gives you 100 percent!).
Why does this happen?
A designer can impress stakeholders by producing an impressive-looking artifact. If it is, in fact, your product, but it isn't finished yet, you can call it a demo and all will be forgiven if your scripted click-through falls off track and your prototype crashes badly. Because it is lovely, it can even give the false sense of being usable.
Using early prototypes for usability evaluations is an important option for testing design concepts. However, if the fidelity of your prototype is too high, it may not be clear what you are really testing, and you risk gathering false positives on the concept.
Sometimes the consequences are high stakes: You convince a customer that you are much further along than you actually are, and deadline negotiations for delivering the actual product reflect the customer's reasonable demand for early delivery while you struggle to meet impossibly close deadlines. Is offering an unlovely collection of half-baked ideas to a user the only viable source of critique? One could say it's much better to go for a half-baked look without the bells and whistles, and simply test the concept with a low-fidelity prototype. It may not look nice, but it will result in a better product at the end of the day. But you can also do both: Show an attractive limited subset to illustrate the look and feel of the final product, and then get to work evaluating that drab gray collection of input fields and checkboxes.
In any case, the map is not the territory and the prototype is not the product.
Confusing the two will result in an impossible entanglement that no quantitative data can unravel.
-- Jonathan Arnowitz and Elizabeth Dykstra-Erickson
REFERENCE 1. Huff, D. (1954). How to
Lie with Statistics. New York: W-W Norton
and Company Inc.
This is a draft version (preprint) of material which was later published in substantially the same form in my Rant column in ACM’s magazine. The published version is a copyrighted feature of Communications of the ACM. All requests for reproduction and/or distribution of the published version should be directed to the ACM.
