documentation

Usability reports (usability rant part 2)

Following on from yesterdays post about the usability process, today I’ll focus on the deliverables, the usability report and my contention that they are rarely grounded in any understanding of the project reality. Here are a couple of examples of usability findings from a (well respected) usability company’s report:

Finding: It was said that the ability to filter [the search results] would be important.
Recommendation: Add check boxes so the customer can choose between [result types]

“Add check boxes”.

That’s easy to say, three words “Add. Check. Boxes”. But what if the particular search engine the team are using does not allow such functionality?  Or such functionality will take significant effort to build.

Finding: When probed about the use of breadcrumbs on the site, 2 participants were confused by the structure that was displayed.
Recommendation: Consider using chevrons [for the breadcrumb] to better covey to the customer that these words show the journey they have been on [rather than ‘/’]

Let’s leave aside the basis of this recommendation; two participants commenting that they weren’t sure about the use of the ‘/’ (this sounds more like it is reinforcing the authors prejudice against the use of / in a breadcrumb and their preference for the ‘>’ symbol).  And let’s also leave aside the fact that it has taken three weeks to let the development team to know that.

It is presented on a powerpoint slide with a screen shot of the breadcrumb and a mockup of a preferred solution, e.g. “Home > UK > South East > News” (Rather than Home / UK / South East / News”).  I’d estimate this took twenty minutes elapsed time to produce this slide. It will take a further ten minutes to discuss when the page is presented to the product owner. And the product owner will spend ten minutes explaining it to the IT project manager who will take ten minutes to explain it to the developers to make the change…

Save your money

Usability testing is not a science. Investing in one or two formal usability tests is almost certainly money badly spent. The Cue reports give a good insight into this.  For example, they asked seventeen experienced professional teams to independently evaluate the usability of the website for the Hotel Pennsylvania in New York.

The teams reported 340 different usability issues. Only nine of these issues were reported by more than half of the teams, while 205 issues (60%) were uniquely reported, that is, no two teams reported the same issue. Sixty-one of the 205 uniquely reported issues were classified as serious or critical problems.

They go on to state…

The study also shows that there was no practical difference between the results obtained from usability testing and expert reviews for the issues identified.

This suggests getting a UI expert into the project team is probably money better spent than employing the usability company (and supports my assertion that usability testing is often just validating the work of a professional).  And when you do get a usability company to report back, as I’ve discussed above, don’t hold your breath for the quality or usefulness of the results:

They found that only 17% of comments in usability reports contained recommendations that were both useful and usable, and many recommendations were not usable at all [source]

So if the recommendations you get from one company are likely to be different to the recommendations from another; if the report is going to be full of recommendations that are impractical and not implementable; if an expert can pick up usability problems that usability testing can, why bother with the usability company testing at the back end of the project?  Indeed, why bother with the usability company at all?  Get an interaction designer on the project from the outset (call them an information architect, user centred designer ,UX person if you want), get them testing ideas and interfaces informally and regularly throughout, and truly embed usability into the project itself, not as an add-on process and report.

Usability rant part 3>

The scourge of Document Driven Design

Documents, or rather words are the scourge of product design. Because words can never convey the true meaning or emotion of what is really required. All to often, software development projects are driven by the documentation – agile projects can be equally guilty of this- driven by words on paper (or card) that convey what the requirement is. Issue #1. Developers don’t read!

Except for a few exceptions, software is all about a user interacting with a system in order to accomplish a goal. Trying to describe that premise in terms of features and functionality is fundamentally flawed as it will be nested in the language of technical implementation, not in the user interaction. “Find” becomes “search”, “buy” becomes “shopping cart”, “check” becomes “validation” and so on. Issue #2. The desired outcomes become lost in a smog of technical jargon.

Solution: Picture Driven Design. Up front. Yeah! I’m all for up front design! A picture tells a thousand words. It has the power to remove ambiguities, clarify the vision, showing what the journey to realise outcomes look like. Start with a day in the life of… scribble out the flow, the user journey. Nothing complicated, boxes and arrows. Then scribble out what the user interface might look like. Done. That’s your up front design. That’s your documentation. That’s your scope. What you do next is up to you, write loads of documents that describe it and produce the software in a waterfall way if you want. I’d prefer you were more lean and adopted agile practices, but whatever you do, start with the picture. I’m convinced it will save much pain later on.