Critiquing the critics (usability rant part 1)
Michael Winner may be a good food critic, but if you were looking for someone to cook you the finest meal for your budget, I doubt he would be your first choice.Same with film critics, they may be able to write an insightful and critical review, but would you want them directing a film for your budget? Would you want Jakob Neilsen, who is essentially a usability critic, to design your website? I mean, take a look at his site!
When you are building a product, you get a usability company in because you know that usability is a good thing that you want to have. If usability companies are the critics, what are you expecting?
The first usability test I ran was in 1991. I’ve set up usability labs, I’ve observed hundreds of people interacting with technology and products. My passion has always to do things at speed, turn around results ASAP and engage all stakeholders in the process. But I’ll talk about that in a later post. For now I’ll draw on experience of working with organisations that have commissioned usability companies to review their products. I’ll breakdown the process I have often observed from usability testing vendors, considering both the elapsed time and the actual ‘value added time’ taken.
The client (usually the business) engage the usability company to audit the usability of the product that is being developed. The consultants will come in and understand the user tasks, roles and goals; the target audience will be identified for recruitment. ‘Value added time’ = 1 hour.
The team go away and produce a test plan and a recruitment brief for a research agency to find participants. They promise to get it back to the client in a couple of days. They contact their preferred agency who set about recruiting people (let’s assume this is a simple brief for a retail website targeted at young mothers). Produce test plan (value added time = 3 hours). Send to client for review.
Client return test plan with a few comments. Update test plan. Value added time = 30 minutes.
Twelve usability sessions, each an hour long, they do three a day, that is four days of testing. Value added time = 12 hours
Days eleven – thirteen
The team spend three days analysising and synthesising the results, pull supporting video clips and produce a detailed report. Value added time – 15 hours
The client sees the report for the first time. (Value added time = 2 hours). Interesting results. (IT representation were not invited, they did not commission the report, the product owner wants to see the output first before sharing it with IT).
The product owner informs the dev team of the changes that need to be made in the light of the usability report. Project manager sucks air through his teeth and says “you’ll need to raise a change request for those items… ha! quick wins they say? hardly… Hmmm, OK, change the labels in the field, we should be able to do that…”
Value added Vs. Elapsed time
The usability company has delivered and their engagement is complete. From the start of the process to the recommendations hitting the developers who must ultimately action these, for this not-too-fictitious scenario sixteen days have passed, of which only four were spent on value-added tasks, actually doing stuff.
The product goes live. The usability company are aghast that so many of the changes they reccomended have not been implemented. They place the blame fairly and sqaurely at the door of the developers and reinforce their belief that IT just doesn’t listen, or worse, care about usability. The critics have critcised from their armchair, like the pigs and chickens they are the chickens, participated not committed.
It’d be interesting to see if you can convert this rant into a process that is feasible and helps in early engagement of dev team.
That comes in part three of the rant 🙂
Bringing in the usability people after the system has been implemented, is too late. It’s better to start the project with interaction design and bring in the implementation people after the it’s known what should be created and it has been tested to be what the users need.
[…] last couple of posts here and here have been critical of boutique usability companies that offer user testing services. In […]