Results Are In: Fidelity Deception Ranks High on Usability Problems

(Adapted from a previous editorial I wrote for interactions Magazine)

This editorial is from a controversial issue on measuring usability. This special issue covered CIF usability testing (a seemingly innocuous if important topic). Upon publication many took strong esxception to our coverage of CIF testing. But perhaps these critics also took exception to our criticizing a practice all too often employed: how to lie with statistics and its designer equivalent using too high a fidelity to overwhelm an audience. --Jonathan

This issue is primarily about usability and metrics, measurements, and what we do with them. And thus our rant: deceiving with numbers.
At university you may have had the pleasure of a reading assignment instructing you to read and memorize a slim volume called How to Lie with Statistics [1]. This is a classic that presents the unimaginable- convincing people of something by manipulating unfamiliar concepts that sound scientific, and thus authoritative. One could, of course, achieve the same end via bullying or mesmerizing charisma. But lying with statistics is a venerable process engaged in by politicians, advertisers, and scientists- and even usability professionals.
One of the ingenious cheats of statistics is to "prove" that you are correct by accentuating the positive and eliminating the negative. To be a master of deception you do, of course, need tools. Not of the rabbit-and the-magic-hat variety. A simple high-fidelity prototype can do the trick. We can call this type of deceiving with numbers High Fidelity Deception, and it goes something like this:
Reduce the overall feature set of a target product.
Design a prototype that covers some of the features.
Ensure that it is attractive enough to resemble a finished product with a dose of glamor: nice animations, transitions, professional color palette (for extra points, make it blue-you do know about that, right?), refined font, and upscale branding elements.
Show it to a few high-placed individuals who (this part is important) don't have the time or inclination to dive into the details.
Get their enthusiastic endorsement.
Proceed to formalize this into a final product regardless of the paucity of credible detail.

You can get excellent marks on the prototype: high levels of user satisfaction, compliments on the prototype's refined sensibilities, and unequivocal approval to proceed. This prototype is, in fact, merely an attractive shell, but it passes muster, and you can quantify its success by repeating what percentage of users who have seen it were "wowed" by it (don't mention the actual number of users, though-after all, two out of two gives you 100 percent!).

Why does this happen?

A designer can impress stakeholders by producing an impressive-looking artifact. If it is, in fact, your product, but it isn't finished yet, you can call it a demo and all will be forgiven if your scripted click-through falls off track and your prototype crashes badly. Because it is lovely, it can even give the false sense of being usable.
Using early prototypes for usability evaluations is an important option for testing design concepts. However, if the fidelity of your prototype is too high, it may not be clear what you are really testing, and you risk gathering false positives on the concept.
Sometimes the consequences are high stakes: You convince a customer that you are much further along than you actually are, and deadline negotiations for delivering the actual product reflect the customer's reasonable demand for early delivery while you struggle to meet impossibly close deadlines. Is offering an unlovely collection of half-baked ideas to a user the only viable source of critique? One could say it's much better to go for a half-baked look without the bells and whistles, and simply test the concept with a low-fidelity prototype. It may not look nice, but it will result in a better product at the end of the day. But you can also do both: Show an attractive limited subset to illustrate the look and feel of the final product, and then get to work evaluating that drab gray collection of input fields and checkboxes.
In any case, the map is not the territory and the prototype is not the product.
Confusing the two will result in an impossible entanglement that no quantitative data can unravel.
-- Jonathan Arnowitz and Elizabeth Dykstra-Erickson

REFERENCE 1. Huff, D. (1954). How to
Lie with Statistics. New York: W-W Norton
and Company Inc.

This is a draft version (preprint) of material which was later published in substantially the same form in my Rant column in ACM’s magazine. The published version is a copyrighted feature of Communications of the ACM. All requests for reproduction and/or distribution of the published version should be directed to the ACM.

Software that Behaves Like Spoiled Brats

As many people know, one way of analyzing interaction design and system behavior is to do something that has many different names, but usually sounds something like dialog analysis. Dialog analysis is the process of analyzing the software interaction as if the system is a human being who is performing a service for the client, aka the

woe-besieged user. After careful analysis of many products, it seems to us that if most commercial software and Web products were humans, they would be characterized as greedy spoiled brats. Just for an example, look at the way desktop products, customer services, and Web products behave.

Desktop Products

Office productivity tools make you do somersaults to undo their automatic formatting. They also drive you crazy with the collision of features they never knew would be used together. One word processor we were using to create a large document from many sub-documents refused to behave—auto-indenting, bulleting, or renumbering tables into items. It so destroyed the formatting and twisted the style usage that we had to resort to using flattened pictures of some tables and formats. When you try to do something that a product's designers didn't anticipate, some products exact their revenge: They do the digital equivalent of throwing everything into a heap, leaving you to clean up the mess all by yourself. This is marginally acceptable behavior from a two-year-old child learning the concept of patience. It is reprehensible behavior in adult human-computer interactions, and it turns the product into something we call a spoiled brat.

Customer Service Products

How ironic that some of the most abusive products around are software products designed for customer service. Let's just take call-center experiences as an example. Phone trees in service calls exhibit "do what I say or else" behavior. The user follows patiently along only to find 17 steps later that the end point is closed, or does not give you the information you expected, or transfers you to a busy line, or leaves you no way to return anywhere but to the beginning of the long depressing tree. Brat.

Web Sites and Services

There are also hidden brat interfaces that reflect the bratty intentions of their owners. All too many Web services abuse users by demanding all sorts of personal information, much of it completely unrelated to buying the service. They then turn around and sell your name and information to other people to spam and telemarket you to no end. These sites even brashly claim that your data is safe and that they have privacy rules and policies. Here the HCI brat is the very small print and the checkmark you will overlook that permits them to share your information with other companies with whom they have a relationship (even though that relationship only exists to sell your personal information!).

User experience as a field has a long way to go. Our bratty products do us little credit. Thank goodness we're not all brats.... There are certainly things to rave about as well!—eic, Jonathan Arnowitz and Elizabeth Dykstra-Erickson

This is a draft version (preprint) of material which was later published in substantially the same form in my Rant column in ACM’s magazine. The published version is a copyrighted feature of Communications of the ACM. All requests for reproduction and/or distribution of the published version should be directed to the ACM.

Whose profession is it? It’s Mine!

This article was the front editorial for Pabini Gabriel-Petit’s special issue on different stakeholders in the user experience.

We must step up to the plate and take ownership of experience design.

Focusing on "Whose profession is it, anyway?" you might ask, "Which profession are we discussing?" The professional interest is in "user-centered design," wherein professionals design and evaluate user experiences of human-computer interactions. There seems little consensus on who is responsible for that user experience.

Variously, we find that responsibility can fall to engineering, or marketing, or some collaborative group ownership. It seems like everyone's responsibility except our own. One might become resigned to the realization that it is politically inappropriate to claim our own expertise; but in the end this is shirking our obligation.

Taking responsibility

Attributing responsibility or accountability is different than attributing participation: We believe that UX (user experience) design is by its nature collaborative. Working with multi-disciplinary teams is a significant part of our work life. So why is it that UX design, unlike other disciplines, seems to promote a sense of collective work? Could it be that our conduct reflects our discipline's standards, rather than corporate standards?

Product management doesn't build or design products; their job is to own product vision and strategy (naturally with the other stakeholders' input). Engineers own code development and code quality, with a wide range of specialties (architecture, code design, QA, and release management, to name a few). Product marketers take clear ownership of marketing communications and product campaigns, keeping the pulse of the marketplace, and trying to detect what it will buy. Therefore, it's only logical that human-computer interaction professionals take ownership of the user experience. We are, after all, user experience experts, despite the fact that we depend on other development participants to meet user and business needs.

C-level ownership?

Why is it that we see few c-level managers for user experience? Chief User Experience Officer (CUXO) could be a formidable title. In most large software development organizations, user experience folks report in through engineering or marketing, whether there is a special group for HCI or not. That implies, however, that at the end of the day and after all possible design dilemma escalations, it is squarely the call of engineering or marketing (heaven forfend it should be our call) on what to create. We need our own high-level management who can directly compete with marketing and engineering management. (Please read that part again: This time and at this level we didn't say collaborate). Our agenda needs to be given the same respect as other disciplines. But this won't happen if we don't do our part. We must step up to the plate and take ownership of user experience design.

We've seen some positive changes in org charts: An HCI executive is no longer unheard of as we see more directors and senior vice presidents of User Experience. Little by little our agenda is slowly rising to the surface of formerly engineering-driven companies. But this puts all the more pressure on us to seize the moment and take control of our user experience.

If we aren't accountable for design, why does a software-making organization need us? If we don't fight for that responsibility, and we accept that our role is simply to suggest possible outcomes, our voices become weak and our protests unheard. We have a sneaking suspicion that our credibility is low because it's not our voices, but our rationales, that are weak. Insisting on ownership of the discipline comes with it a concomitant obligation to do the work with the highest possible standards, and that includes, like it or not, accommodating the quirks of marketing, the insanity of deadlines, and the caterwauling of engineers who have already thought of a cheaper, faster way to implement a requirement than the investment our designs require.

So what's the problem? Why don't they trust us with ownership? Are we inflexible, unschooled, uninformed, or just plain too crabby when asked to truly collaborate? Even in companies that proudly proclaim themselves "user-centered," all too often we see HCI activities included as an afterthought. When our work is conducted too late in the cycle it is pointless—if it has no effect on outcomes, it's a waste of time. Is HCI in your company just a checklist item deliverable? "Did you finish your usability testing? Great, check it off the project chart. You can look at those usability tapes when we're done with this project; we'd like you to focus on something else just now."

Some parting shots

Herewith we offer a few small pearls of truly sensible advice from some of the most difficult engineering managers ever:

• "Why bother with a usability test? It is what it is and it's too late; if you test it you'll expect us to do something about your results, and that isn't going to happen. Save the time for when it matters."
  
• "Please stop designing things. We've run out of implementation time."
  
• "Look, we're after a decent band-aid here, not elegance. Just get on with it or we'll do it ourselves."

You can always pack these concepts away for version n+1 (we call that type of cache of well-constructed design think-ahead as "blue sky"). Be a hero (if you're still there for the next release); revenge is sweet.

Other disciplines should be accountable for appreciating and abiding by our design and evaluation deliverables. It is not enough to require a design spec or a usability test's results. We've seen teams use lab testing to resolve internal disagreements when the number of test participants is barely larger than the team size; this is abdication of design responsibility to an extreme. We should not delude ourselves: There is a big difference between so-called discount methods and Machiavellian ones.
It could be that we all just need better persuasive powers, but we suggest that designating an accountable party (that would be us) in advance would (we guarantee) be much cheaper than extended debate. Engineers, product strategists, and others involved in delivering software experiences must commit to acting on our work. We'd like it if they understand it, too. Our commitment as a support to code development does not mean we can't have a controlling vote in a product partnership where consensus does not always rule.

Hold the soap

That is not to say that what a designer says is always the end of the line. We don't want to make a soapbox petition for control; we want a true dialogue on user advocacy and design sensibility. Designers need a thick enough skin to deal with critique, compromise, and collaboration, just like the developers, the marketers and the product managers. In each of our personal lives, someone, somewhere, told each and every one of us that people are supposed to just get along. This implies that the natural state of things is easy consensus. We beg to differ! It isn't supposed to be easy! You can admire and respect your competitors and your collaborators, but you do not always have to agree with them. And for the times that the discussion is on your turf, calling for your expertise, and demanding intelligent solutions from you, you need to be able to grab the ball and run with it.

When called for, we do need to be able to look any of our collaborative colleagues square in the eye and say: "I hear you. I understand your arguments; but we're going to do it this way."—eic Jonathan Arnowitz and Elizabeth Dykstra-Erickson

This is a draft version (preprint) of material which was later published in substantially the same form in my Rant column in ACM’s magazine. The published version is a copyrighted feature of Communications of the ACM. All requests for reproduction and/or distribution of the published version should be directed to the ACM.

Loser-Centered Design or let it be: the Frankenstein Persona's

This first entry is an editorial that covers the phenomenon of creating persona's without user research, which made quite a buzz at CHI2007, even though it first appeared in interaction in 2006. Frankenstein persona's, a particular egregious example of user-uncentered design, is something Elizabeth and I have witnessed not only in practice but also as reviewers for many conferences and journals (all of which incidentally have no name).

All too often, terms are misappropriated, if not outright hijacked, and their meaning becomes either obfuscated or totally changed. Have you noticed that user-centered design is quickly becoming such a victim? We feel the only way to protect this saintly term is to fight fire with fire and provide a counter weapon whenever the term user-centered design is misused: Enter the term loser-centered design.

Loser-centered design focuses on the wrong things, and worse, ignores the user-yet it still uses the term user-centered or user-focused; either to justify some Machiavellian ends or to cover up poor process. Unfortunately, loser-centered design, loser-focused design or loser-centric development (the terms are sadly synonymous) take many, many different forms. We want to give you some examples here; surely you have your own. (Feel free to add them as comments to this entry and you will have the immense gratification of commiserating other designers worldwide.)

Frankenstein Personas
One obvious example of loser-centered design is pandering to a marketing department's market research instead of conducting and using real user research (e.g., talk to someone who isn't in your company, on your team, or co-habitating with one of the aforementioned). You can smell these projects a mile away—the task analysis or information architecture just mimics the research, and neither questions basic assumptions nor adds anything that doesn't fit the pre-conceived marketing model. Loser-centered design in these projects also takes the form of user segmentation based on personas that were defined largely by internal focus groups and a few friends of the marketing department (or anyone else who has read too many in-flight magazines). All too often these personas are created from magazine clippings illustrating brand desires and wants based entirely on marketing mumbo-jumbo. Yet they become so tangible and visually vivid that designers rush in and design for these modern persona Frankensteins as if they really existed. Tell the truth, are you guilty?

The Loser Researcher The loser researcher is a charlatan or inexperienced user researcher, with just enough knowledge to be dangerous, who conducts user research for a product and instructs designers on what problems to solve. Instead of observing what users actually do, they ask users for instructions on what they should design. Loser research has some tell-tale signs: out-of-context telephone interviews, open-ended survey questions, and site visits conducted only in the on-site lunch room (assuming the software has nothing to with employee lunches).

One example of such folly follows. A designer was asked to look at improving communications since something was obviously wrong with the intranet. Indeed, upon being interviewed, senior management offered up numerous complaints that the junior members of the staff could not find the right information on their intranet. These complaints were addressed at great cost with much changing around of data layout and information architecture, but the results did not improve the situation.

A user (not loser) research firm came along and observed what the users actually did. The real problem, as became apparent by observation of the office, had nothing to do with the intranet and everything to do with a recent reorganization and the physical traffic through the work area. During the reorganization the company decided to reward high performers with window cubes, centralizing all the junior and new employees several walls away from the more senior employees. Not wanting to appear as uninformed as they felt, junior folks stopped asking for advice, since they had to expose their ignorance by making a public trek to a senior level staff cube to seek advice. A room redesign solved the problem.

Designing the Emperor's New Clothes
Reducing call-center-response time forms the crux of another loser research opportunity. Convinced that shaving seconds off of each support call would add up to entire individuals that could be dismissed, the user researcher was called in to analyze the business process and observe why results were still below expectations. Through interviewing the call-center staff, the user researcher discovered a huge number of problems—none of which management expected. Told that mistrust, disrespect, lack of information, and unfair treatment was the root of the problem, management reacted with outraged disbelief. "We already solved that problem!" Evidently not. The user research results were uncomfortably human. Subsequent loser research focused on technical issues; while the results showed no promise of improvement, focusing on machines rather than social practices "felt" better to management. After all, one should only have to "fix" people once. When loser research results were available, management breathed a sigh of relief and felt that the problem was well in control—and that its own management style was no longer to blame.

So the next time you hear the word user-centered design applied incorrectly, jump on the offensive and whip out the loser-centered design retort: "user or loser?" There is, indeed, a difference.—eic - Jonathan Arnowitz & Elizabeth Dykstra-Erickson


This is a draft version (preprint) of material which was later published in substantially the same form in my Rant column in ACM’s magazine. The published version is a copyrighted feature of Communications of the ACM. All requests for reproduction and/or distribution of the published version should be directed to the ACM.

Welcome to User Experience in ArnoLand

Hello and Welcome,

This Blog is dedicated to a discussion of the state of the User Experience profession. For the coming entries, we will be reprinting the commentaries on HCI that have appeared in interactions magazine for the past 3 years as written by myself and Elizabeth Dykstra-Erickson (as slightly edited/updated by me). These commentaries, though tongue in cheek sometimes, or sometimes very provocative were intended to draw a response from our audience. As a print magazine in an e-world, this was difficult to realize. Now putting this information in blog format I hope to generate the discussion our articles were intended.

Needless to say, this is also the space to discuss/confront one of the authors of Effective Prototyping.

I hope you enjoy.

Jonathan