Usability Engineers vs Designer: the process problem.

Another week has come (mine starts on Tuesday, you can do those things when you live in Europe). More and more, we see problems surfacing not from having the wrong people in place but rather the wrong process. My previous post discusses the rampant wrong process with prototyping. Here I want to touch on the process issues with usability testing and design. I want you to consider this familiar and completely unnecessary scenario:

A Designer works on a conceptual design with the customer. Then he works out a detailed design into a prototype that can be tested. So far so good. But what goes wrong is that the Usability Engineer is often disconnected to either the design concept or the detailed design. The usability engineer ends up suggesting new designs that totally contradict the conceptual design. The designer is gone. The engineering team implements the changes and the result is a Frankenstein's monster that despite the best UX resources, fails in the marketplace.

The obvious problem is the process disconnect between Designer and Usability. And the problems are serious. I want to discuss two aspects to these problems and how to resolve them: namely how False Negatives in a usability test and how to deal with Usability Engineer's design advice.

False negatives

When usability testing is conducted without input from the designer, this can lead to many false negative issues in the usability test. Examples of the errors that can result include:

• Early tests will report usability issues with conventions that a user is expected to learn over time or with a different task flow than being tested.
• Early tests especially lower fidelity ones may not catch learnability/system feedback issues due to lack of visual fidelity needed to communicate with the user
• Test moderators, not knowing the underlying concept may inadvertently introduce the topic or task in a way that is at odds with the design, thereby confusing the test subject.

This list is just the tip of the iceberg. These negative side effects are completely avoidable by making sure the Designer and Usability Engineer work together on the Usability Test script, identifying tasks and their importance. Also the task order when that is appropriate for a test (for example one step is a required gateway: e.g. sign up). Also let the Usability Engineer attend some of the conceptual design sessions and, OMG let them participate in the conceptual design; so they gain a thorough understand of it. Conversely, Designers should observe the usability tests whenever possible. The tests themselves can be so much more inspiring and vivid than even the best written report.

Usability Engineer's design advice

It is an expected part of the Usability Engineer's work to include not just data and analysis; but also design advice or alternative designs. This does not need to be a problem. But without setting expectations, the innocent Product Manager or software engineer confronted with new contradictory designs can quickly conclude that the UX profession is a screwed up group who cannot make up their minds.

Among the possible problems with blindly taking Usability Engineer design advice is:

• Designs may not be an ideal solution for the problems they have discovered
• Designs often recommend things that will cause usability problems elsewhere by introducing conceptually non-standard interactions
• Designs ignore larger issues since their advice focuses on the testing and not the larger issues (e.g. Business and Technical requirements which may lead to a different solution than suggested).
• A common example of this is when the Usability Engineer suggests something that is technically impossible for the requirements or constraints.

These and other issues with the Usability Engineer design advice harms everyone's credibility both designer and usability engineer. This is not to say that Usability Engineers should not give their advice. But it is absolutely important to set the right expectations. Usability Engineer design advice should be viewed as input to the problem not the solution. If the Usability Engineer includes the design rationale this will often give the vital information for coming up with a more ideal solution.

The design rationale should enumerate the objective reasons for the alternative design. This allows the designer to bridge the problem design with a solution based on objective criteria instead of personal taste.

[Objective information is one that either refers to the usability data itself (e.g. only 2 out of 12 users understood this command) or conceptual data based on requirements, (This design does not appeal to our target users or is not constant with the image/branding of the company). Both types of information can lead to a solution. Comments like, "I don't like that color" or "It doesn't look right to me." do not lead to workable solutions.]

Usability data misinforming design

Usability data is rarely communicated with the limitations or short-comings in the data and this is a real pity. All too often a usability engineering report reads like a set of demands and commandments without stipulating under what conditions this advice or anaylsis should be given. Things like the significance, persistence, sampling issues, etc. are often underplayed. Again a faulty process is the problem. Many usability engineers are under pressure to work quickly and also find dramatic and significant results. This can put a Usability Engineer between a rock and a hard place: asked to review a product with three of Janitor's friends and then come up with a list of "just the most important recommendations." Ah if life were only so easy? Yet we are constantly being put in this position. The client maybe always king, but findings that can include a little context setting would help the end-users of the usability reports.

Lessons learned

• Designers and Usability Engineers should insist on working together in projects. Meaning the Engineer is available during the concept design phase and the Designer is available during the testing design phase. (With iterative testing the designer must be available with each design cycle.)
• The customers should require design and usability engineers to work together. This will often require the usability engineer to come in early for 1-2 days in the conceptual design phase before their main work begins week(s) later. (Yes that also means if the engineer is an external contractor, the customer must pay the Usability Engineer for this work.)
• Customers should also realize the usability engineers do not provide solutions they propose challenges and problems that need to be solved.
• Usability Engineers may be great designers or maybe crap designers but as long as they include objective design rationale for their proposed solutions they will always be helpful

Prototyping software tools that work

The Blogosphere is full of posts over what is the best prototyping tool. This fight over Axure, Flash, Dreamweaver, PowerPoint, etc., is misplaced and looks at prototyping as something that is purely a single concrete one size its all activity, or at least one tool fits all. This is misplaced for two reasons. First, this discussion disempowers the designer, product manager, developer who wants to use software tools already at their disposal. Secondly, ignores the fact that prototypes are sometimes created extremely rapidly (like in an hour) or more thoroughly (in a month). The two prototypes on the opposite of these extremes have far different tool requirements. Therefore, it is far better to think of prototyping toolsets.

A prototyper should have many tools just like a mechanic has a toolbox. Otherwise the prototype who knows just one tool is not very effective. Like to the hammer every problem looks like a nail, a designer who knows just one prototyping tool offers just one type of solution when many many more are possible. If you look for a different perspective mainly what are the specific prototyping needs, some surprising prototyping tools emerge.

Specialities
Prototyping tools can usually do some kinds prototyping better than other kinds. Furthermore, you may also prefer to use a certain tool simply because you know it's special features better than another tool.

General Availability
A prototyping tool that is not generally available to the design team makes the team reliant on a chosen few with access. If the resulting file format is also unreadable outside the prototyping tool then access to the prototype is further limited. In some organizations this is a good thing; but when you want to maximize the knowledge of your design team having a prototyping tool set hat empowers people is important.

Interaction styles & Functionality

Tools support certain platforms, interaction styles and functionalities. These actually limit what types of prototypes you can create. For example, if the widget set in your prototyping tool does not support spinners, then spinners will not appear so easily on your prototype. Moreover, the more sophisticated the UX widget set and the interaction capabilities the more conservative and less innovative your prototype will be. Conservative because you are cornered into the widget set and interactive capabilities of the tool.

Design Team Talents
If a design team loves one product, making the switch over to another product robs them of their talent they already have. Of course that can be counterbalanced if there are long term advantages in adding the prototyping to the prototyping toolset.

Deliverable formats

How a prototype will be used is also essential. If the resulting prototype will be a paper prototype. For example, choosing a tool which can print out designs and portions of designs easily would be a factor.

Tools review

There are many different dimensions to look at since prototyping involves multiple dimensions (fidelity, audience, style, etc.) However as one handy table (I can make more if they are helpful, here is a grouping of prototyping tools and what they are most appropriate for. (I will update this table based on user comments.) The table is more or less an overview of a possible look on similar tools. For your personal organization, it might be helpful to list all the tools you know and list for each: advantages, disadvantages, etc.

Methods	Tools
Static wireframe	PowerPoint, Excel, Photoshop, Illustrator, Fireworks, Visio, OmniGraffle, HTML editors, Word
Storyboard	PowerPoint, or other presentation software, Acrobat, Excel
Paper	Word, PowerPoint, Excel, Photoshop, Illustrator, Fireworks, Visio, OmniGraffle, HTML editors
Wizard of Oz	WebEx or other video conferencing software or synchronous sharing tools.
Digital partial interactive	Excel, PowerPoint, HTML Editors, Acrobat, Visio, OmniGraffle, Axure
Digital fully interactive	Flash, Dreamweaver and other HTML Editors, Visual Studio, Director, Axure

It's all about the Concept

This editorial is one very dear to my heart, as it touches on the cornerstone of good design and something I miss in all to many HCI designer's work: the concept. In order to keep to the same points as in the original, I have done little editing from the original.

How can a design review be conducted on static interfaces?

What constitutes good design?

Good design is harder and harder to find these days. It is disheartening when people present a single window or Web page and ask for an evaluation, especially when the question is: "Is this good design?" How can a design review be conducted on static interfaces? What is possible to evaluate? What constitutes good design? Is it possible to judge a design from static screens?
As we said earlier—in fact way earlier in an article co-written with Wendy Mackay in March 2001—when discussing the importance of contextualizing design, the issue is not whether a design is good, but is it a good design of...what?
When starting to review a single screen, your heuristic-laden ego itches to pour out criticisms. A copy command on the file menu! Are you nuts?! You don't put buttons on the bottom left! Serif fonts! Are you mad? Where did you get that typeface? Spinners are so 1990s! Script typefaces on mobile devices? Split buttons! Are you sure about that? But instead of continuing that tirade, let's pause for a moment and ask: design of what?
First off, let's suggest some good questions. Do you really know the context to start judging a design correctly? What aspect of design are you reviewing? The visual design? The information design? The layout design? The interaction design? How do you "see" an interaction design in a static page?
There are of course many ways to represent all of the above. We're interested in how you do it. Do you divorce these aspects of design, or do you combine them in certain ways? Which combinations have been most successful for you?
It's difficult to divorce one from the others: All aspects of design must work together in a unified concept. That concept involves rich knowledge preceding the design activity: of the end users, their background, their tasks, their mental models. It doesn't stop there: You then need to understand your engineering constraints: what your developers' toolkit can and can't do, what sort of custom code your design will require, and whether your design needs to absolutely follow standards for future evolution and code maintenance, or if you're able to leap into new territory and design a new widget or two. Further rich knowledge that can and should influence the conceptual design: understanding the business model of the company producing the software. Is this a demo? Can it cut corners, or is it production quality? Is it a step in a long line of .dot releases? Is it the first version? Can you take risks? Do you need to reach feature and usability parity with other competitors, or do you need to excel and claim best-in-class?
Before you can understand how to design, you need to understand design. The conceptual design is more than any one facet of design; it's a gestalt that is more than the sum of the parts. Taking this perspective, how do you evaluate that single screen?—Jonathan Arnowitz and Elizabeth Dykstra-Erickson

This is a draft version (preprint) of material which was later published in substantially the same form in my Rant column in ACM’s magazine. The published version is a copyrighted feature of Communications of the ACM. All requests for reproduction and/or distribution of the published version should be directed to the ACM.