Usability reports (usability rant part 2)

Following on from yesterdays post about the usability process, today I’ll focus on the deliverables, the usability report and my contention that they are rarely grounded in any understanding of the project reality. Here are a couple of examples of usability findings from a (well respected) usability company’s report:

Finding: It was said that the ability to filter [the search results] would be important.
Recommendation: Add check boxes so the customer can choose between [result types]

“Add check boxes”.

That’s easy to say, three words “Add. Check. Boxes”. But what if the particular search engine the team are using does not allow such functionality?  Or such functionality will take significant effort to build.

Finding: When probed about the use of breadcrumbs on the site, 2 participants were confused by the structure that was displayed.
Recommendation: Consider using chevrons [for the breadcrumb] to better covey to the customer that these words show the journey they have been on [rather than ‘/’]

Let’s leave aside the basis of this recommendation; two participants commenting that they weren’t sure about the use of the ‘/’ (this sounds more like it is reinforcing the authors prejudice against the use of / in a breadcrumb and their preference for the ‘>’ symbol).  And let’s also leave aside the fact that it has taken three weeks to let the development team to know that.

It is presented on a powerpoint slide with a screen shot of the breadcrumb and a mockup of a preferred solution, e.g. “Home > UK > South East > News” (Rather than Home / UK / South East / News”).  I’d estimate this took twenty minutes elapsed time to produce this slide. It will take a further ten minutes to discuss when the page is presented to the product owner. And the product owner will spend ten minutes explaining it to the IT project manager who will take ten minutes to explain it to the developers to make the change…

Save your money

Usability testing is not a science. Investing in one or two formal usability tests is almost certainly money badly spent. The Cue reports give a good insight into this.  For example, they asked seventeen experienced professional teams to independently evaluate the usability of the website for the Hotel Pennsylvania in New York.

The teams reported 340 different usability issues. Only nine of these issues were reported by more than half of the teams, while 205 issues (60%) were uniquely reported, that is, no two teams reported the same issue. Sixty-one of the 205 uniquely reported issues were classified as serious or critical problems.

They go on to state…

The study also shows that there was no practical difference between the results obtained from usability testing and expert reviews for the issues identified.

This suggests getting a UI expert into the project team is probably money better spent than employing the usability company (and supports my assertion that usability testing is often just validating the work of a professional).  And when you do get a usability company to report back, as I’ve discussed above, don’t hold your breath for the quality or usefulness of the results:

They found that only 17% of comments in usability reports contained recommendations that were both useful and usable, and many recommendations were not usable at all [source]

So if the recommendations you get from one company are likely to be different to the recommendations from another; if the report is going to be full of recommendations that are impractical and not implementable; if an expert can pick up usability problems that usability testing can, why bother with the usability company testing at the back end of the project?  Indeed, why bother with the usability company at all?  Get an interaction designer on the project from the outset (call them an information architect, user centred designer ,UX person if you want), get them testing ideas and interfaces informally and regularly throughout, and truly embed usability into the project itself, not as an add-on process and report.

Usability rant part 3>

User interface is a disruptive technology

Last year, according to Gartner, with belts tightening, technology executives need to focus upon disruptive technologies (that cut costs).  The top ten list of disruptive technologies makes strange reading.  How will social computing and mash-ups cut costs (enterprise 2.0?)  Most interestingly, coming in at number six on the list is “user interface”.  Now let’s leave aside the fact that a “user interface’ is hardly a technology (it is how technology manifests itself to the user) I’m interested by the fact that it can be considered to be disruptive. What is disruptive about user interface design?

But think a little further.  What is really disruptive is the realisation that good design is more than just adherence to functional requirements; good creative design is more than ‘bells and whistles’ or ‘gold plating’.  A good user interface will cut costs by enabling the internal user base be more efficient and productive.  A good user interface will enable customers to succesfully complete their transactions / goals.  In a world where poor UI on enterprise applications remains, maybe user interface is indeed a disruptive technology after all.

How are you managing the change?

To the development team ‘change’ relates to scope and requirements within the project, but change runs far deeper than that.

A question that I am often asked is how do you manage business change on agile projects. Release regular and often is an often quoted mantra, but what does that mean to the business where rolling new software across the large, multi-site organisation? How do you manage the piecemeal introduction of new technology, features and functions to hundreds or thousands of people, many levels removed from the project across remote offices and geographical locations? How do you ensure the recipients of the new technology rapidly adopt it and accept the change, even when change is occurring every few months.

What are the financial and human performance implications of each new release in terms of training, productivity and morale? What is the overall burden on people in frequent change?

The reality is that it is not unusual for projects deemed successful by IT and the immediate business team to ultimately fail when released to the broader organisation. Effective change management can be even more important when an organisation adopts agile software delivery.

An analogy as an example. If I expect a screwdriver and you only give me a cross-headed screwdriver when I really want a flat head one I am going to be unhappy. The core team may have prioritised the cross-headed one first for good reason, a flat headed one maybe coming just round the corner, but if you don’t deliver to my expectations I am going to be unhappy. Worse, I am likely to become resistant to future change and less likely willingly cooperate with the uptake of future releases, even if they do start to deliver to my needs.

Keep it on the shelf
The first point is that regular and often does not necessarily mean release to production for the entire organisation or marketplace. Running a number of internal releases, keeping them on the shelf until a complete and marketable product is ready is a strategy often employed. Significant value can be accrued by getting tested and working software into a pre-production environment and held “on the shelf” awaiting a full release. This maybe a UAT environment where a limited number of stakeholders test the functionality in an ‘as-live’ environment. Or it maybe a beta release to a small, selected number of interested people (e.g. a ‘friendly user trail’). This can often pay dividends with usability issues and minor gripes being picked up and addressed before a major roll-out.


Let’s assume that the team wants to roll out the application early and often to the whole target population. Critical to the success of managing the business change is communication. It is important to manage expectations on a timely and appropriate manner. Explain what the upcoming release will do and more importantly what it will not do (and when it will do it). Keep all stakeholders informed of the project progress (setting up a project blog can be a cheap and easy way of letting interested people know of progress), yammer maybe another way of broadcasting updates and information. Having a release count-down can also prepare stakeholders for the change. The techniques can be googled, the important thing is to communicate and manage the expectations (and be ready for inbound questions and comment after go-live).

Adaptable user interface
It is not unusual for the core team to drive for as much functionality as possible in the first release, considering UI enhancements as ‘nice to haves’ and consigning them to later releases. This is a false economy. Consider the cost of training and lost productivity through a hard to use interface. Now multiply that across multiple releases that focus upon utility before usability. Delivering a first release that is self contained and compelling will go a long way to driving organisational buy-in of the new application and greater acceptance of future change. (Jeff Patton writes some great stuff on using story maps to explain what the system should do. Using these will help focus on complete and useful slices through the application rather than random features that are perceived to be of value but do not make a coherent product).

A new user interface, however well designed will inevitably take time to learn the first time it is used. The challenge is with each subsequent release to introduce funcitonality and interactions that leverages the users existing mental model of the application, building upon what has been already been learned. Starting with the end-state, wireframes that articulate the final application then trimming out features, feields and controls to represent each notional release can be a good way of ensuring a UI that will scale as new functionality is added.

Agile organisation
Ultimately the most successful way of introducing agile is to build a beta culture with everyone as agents of change across the whole organisaiton. More importantly change becomes a cycle of learning and continuous improvement. And here I’ll borrow this most excellent graphic from David Armano. David compares what he calls conventional and unconventional marketing but the parallel with software development is obvious. His iterative cycle is “plan-design-launch-measure” but that is not a million miles away from the lean philosophy of “plan-do-check-act”. And critical to the journey is the learning cycle between iterations.