What’s the point of usability testing?

I’ve seen a couple of projects recently where the team have invested in usability testing of wireframe mock-ups of application processes. I’m beginning to wonder whether this is effort well spent.

Don’t get me wrong. I am a firm advocate of usability testing and incorporating it as early as possible into the design process. I’ve set up a few usability labs in my time, facilitated hundreds of usability sessions… I even once had the pleasure (is that the right word) of watching users struggling with boo.com. There is always much to learn from seeing users actually using what you are building. But I now ask the question at what point can you learn the most. If you’ve designed with usability in mind, where is the value in testing the basics of something that instinctively you know as a usability professional to be “right?”

Let me qualify this. The development of an on-line application processes / tools / wizards is now a well trodden path. Any interaction designer worth his or her salt will have experience of building one and will instinctively know what works and what doesn’t work.

In the first instance the interaction designer will create wireframes that illustrate the process; a visualisation of the pages representing the flow that has initially been created.

In creating the wireframes the Interaction Designer should be making reference to personas, will have been collaborating with other team members, validating their assumptions, and will probably be doing informal guerrilla testing around the office to confirm ideas that are puzzling them.

If the interaction designer is any good there will be few, if any issues with the wireframe flow that the users walk through. In a usability test the user is more likely to pick up on labels (and if it is not lorem-ipsumed up any copy that the interaction designer will have written – It is unlikely at this stage that the copywriter will have been involved).

That is a lot of cost and effort to validate the work of a professional.

The wireframes in usability test will generally be based upon a set scenario through a “happy path”. What the wireframes are unlikely to uncover are issues where the user wanders from the “happy path”. Producing reams of wireframes in Visio or PowerPoint and then linking them all up to simulate the finished product is painful and not particularly productive or useful, especially if there is a lot of interactive “web 2.0” stuff going on. To really simulate that you are going to have to build a prototype for real.

Here is something that the devs at ThoughtWorks are pretty good at. Using high productivity platforms such as Django or Ruby on Rails they can pull together a fully featured product that can be usability tested in earnest. It is not a dumb “smoke and mirrors” html prototype; it can have database and “intelligence” behind it. The benefits of this are clear. The business can try and break the process; we can get the business rules right. The developers are able to see how the vision actually functions. Again, they less likely to uncover corner cases that have been missed if a first pass of build has been done already.

Once we have this new avenues of testing are opened to us we can move away from the need to test in a clinical lab environment. Providing a user with a URL to the application form, and using collaborative software such as Skype and Net Meeting we can perform usability testing with greater ecological validity – the user can use the application form on their own PC at home.

4 Comments

  1. Rapid prototyping makes usability testing easier - Peter Krantz · Wednesday, 28 June, 2006

    […] In an article over at Dancingmango Marc McNeill writes about how new web development frameworks such as Ruby on Rails will have an impact on usability testing practices (“What’s the point of usability testing”). The only real reason to test a mockup instead of a real application is of course that it used to be more expensive and time consuming to create an application. With Rails there is no such barrier anymore and usability tests can (and should) be using the real application instead. It is likely that this will lead to a better understanding of how users behave in e.g. a task based system. […]

  2. Martin Dowson · Thursday, 29 June, 2006

    I agree wholeheartedly with everything Marc is saying and that Peter has supported (yes there is a but coming) it is true that if you follow a good methodology, have good people on board, have chosen front and back end technology that means you can develop using web-frameworks like Ruby on Rails… in these ideal situations then you will reduce the need to do “usability testing”

    I for one have moved on from “usability testing” to user experience testing… (whole other posting…)

    but what of those projects who

    1) arent developing on a web-framework… Large corporates developing in-house software solutions
    2) have chosen (for good or misguided reasons) out of the box industry solutions such as (and marc will shudder at this…) Chordiant
    3) dont have the user-centred design skills in house

    This is why we still do usability testing, because unfortunately we don’t all live or work in a perfect bubble where everyone understands user-centred design.

    take a look at IT teams these days… especially those in the mgt consultancy market… (again ANOTHER blog topic) Many of us in the usability/user experience profession don’t work on $500M, 5 year programmes rolling out software across 3 geographies to 100,000 people… when you do projects like that… agile programming, user-centred design are often very low down on the methodoloy musts… cost-control, cost-accountability are…

    I see usability testing within large corporates as an educational process for them, the more they go through it the more they learn. Sure it would be great if we could get them to be agile, user centred, persona-led, info arch savvy from day one… but they’re not so you end up with stuff that needs to be tested regularly to be brought back on track, with teams that don’t get user-centred design because they are in very formal methodologies… usability testing opens their eyes, changes mind sets, helps methodologies morph… I’m getting there eventually with my clients… my ideal is to have them follow more UCD methodologies and eventually not need to test as much…

    So if you are already a user-centred designer… you probably don’t need to do as much usability testing… just cos you don’t doesnt mean that there isnt a large need for it in the marketplace!

    🙂

  3. John Gibbard · Monday, 10 July, 2006

    All interesting points (otherwise why would I comment?!) but still maintain it’s essential to test our work, however well-considered during the design process. There’s nothing like a formalised statement with/without recorded video to indicate the issues and successes with a wireframed idea. I do agree, however, that the value of this diminishes depending on how lo-fi the wireframe is and how iterative the design has been to-date (i loved the guerilla testing image). More posts like this please 🙂

  4. dancingmango » Usability reports (usability rant part 2) · Tuesday, 25 June, 2013

    […] probably money better spent than employing the usability company (and supports my assertion that usability testing is often just validating the work of a professional).  And when you do get a usability company to report back, as I’ve discussed above, […]

Leave a Reply