dancingmango

Usability and the $1 trillion mistake

Is this a case of fat fingers, a usability flaw or poor design that enabled a Citigroup trader to have placed an order to sell $16 billion, instead of $16 million? P&G shares plunged by 23% because of this individual erroneous trade. What followed was the algorithms kicked in and automated trading saw the Dow loose a tenth of its value in less than half an hour. (And Accenture dropped to 4 cents down from $42!)

Before we go blaming human error, questions should be asked why that error occurred. How can someone make such a simple mistake so easily? Was it a case of entering two many 0s? (Don’t stop to look or think, answer the question as soon as you’ve read it – how many zeros are there in this number? 160000000. Same thing again, how many zeros are in this number 12,000,000. That’s a bit easier isn’t it. Only an ‘N’ separates the B from the M on a qwerty keyboard, in a hurry, easily mistaken?)

I’d start by looking at human factors and experience design, and question why (assumption here) the IT team who implemented the system didn’t have before a UX designer on the team to think about the human factors. Could this be the most costly example of poor design?

Article: Big drop, was it all a mistake?

If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!

Hey usability dude, join the devs (Usability rant part 3)

The last couple of posts here and here have been critical of boutique usability companies that offer user testing services. In the final part of my rant, here’s an alternative approach.  Seed into the development teams someone with a usability background (probably with a psychology degree), someone who can create as well as critique.  Align them to Marketing (or ‘the business’) but physically locate them in IT.  To be successful they must be a passionate advocate for the business vision (hence business alignment), but unless they have a daily relationship with the developers (on hand to support immediate design decisions) they will fail.  (The only way to do this is for them to be part of the development team, same building, same floor, same location).

It is better than they Eschew the formal usability testing process for more informal guerilla usability testing.  In an agile project treat this as akin to the showcase.  Testing sketches and wireframes prior to the development iteration, and putting working software when it has sufficient UI coherence in front of users and well as business stakeholders, with the developers as observers in the process.  One person can cover multiple workstreams.  This is a far better use of a limited budget than commissioning ad-hoc reports too late in the day.

Usability reports (usability rant part 2)

Following on from yesterdays post about the usability process, today I’ll focus on the deliverables, the usability report and my contention that they are rarely grounded in any understanding of the project reality. Here are a couple of examples of usability findings from a (well respected) usability company’s report:

Finding: It was said that the ability to filter [the search results] would be important.
Recommendation: Add check boxes so the customer can choose between [result types]

“Add check boxes”.

That’s easy to say, three words “Add. Check. Boxes”. But what if the particular search engine the team are using does not allow such functionality?  Or such functionality will take significant effort to build.

Finding: When probed about the use of breadcrumbs on the site, 2 participants were confused by the structure that was displayed.
Recommendation: Consider using chevrons [for the breadcrumb] to better covey to the customer that these words show the journey they have been on [rather than '/']

Let’s leave aside the basis of this recommendation; two participants commenting that they weren’t sure about the use of the ‘/’ (this sounds more like it is reinforcing the authors prejudice against the use of / in a breadcrumb and their preference for the ‘>’ symbol).  And let’s also leave aside the fact that it has taken three weeks to let the development team to know that.

It is presented on a powerpoint slide with a screen shot of the breadcrumb and a mockup of a preferred solution, e.g. “Home > UK > South East > News” (Rather than Home / UK / South East / News”).  I’d estimate this took twenty minutes elapsed time to produce this slide. It will take a further ten minutes to discuss when the page is presented to the product owner. And the product owner will spend ten minutes explaining it to the IT project manager who will take ten minutes to explain it to the developers to make the change…

Save your money

Usability testing is not a science. Investing in one or two formal usability tests is almost certainly money badly spent. The Cue reports give a good insight into this.  For example, they asked seventeen experienced professional teams to independently evaluate the usability of the website for the Hotel Pennsylvania in New York.

The teams reported 340 different usability issues. Only nine of these issues were reported by more than half of the teams, while 205 issues (60%) were uniquely reported, that is, no two teams reported the same issue. Sixty-one of the 205 uniquely reported issues were classified as serious or critical problems.

They go on to state…

The study also shows that there was no practical difference between the results obtained from usability testing and expert reviews for the issues identified.

This suggests getting a UI expert into the project team is probably money better spent than employing the usability company (and supports my assertion that usability testing is often just validating the work of a professional).  And when you do get a usability company to report back, as I’ve discussed above, don’t hold your breath for the quality or usefulness of the results:

They found that only 17% of comments in usability reports contained recommendations that were both useful and usable, and many recommendations were not usable at all [source]

So if the recommendations you get from one company are likely to be different to the recommendations from another; if the report is going to be full of recommendations that are impractical and not implementable; if an expert can pick up usability problems that usability testing can, why bother with the usability company testing at the back end of the project?  Indeed, why bother with the usability company at all?  Get an interaction designer on the project from the outset (call them an information architect, user centred designer ,UX person if you want), get them testing ideas and interfaces informally and regularly throughout, and truly embed usability into the project itself, not as an add-on process and report.

Usability rant part 3>

Critiquing the critics (usability rant part 1)

Michael Winner may be a good food critic, but if you were looking for someone to cook you the finest meal for your budget, I doubt he would be your first choice.Same with film critics, they may be able to write an insightful and critical review, but would you want them directing a film for your budget? Would you want Jakob Neilsen, who is essentially a usability critic, to design your website? I mean, take a look at his site!

When you are building a product, you get a usability company in because you know that usability is a good thing that you want to have. If usability companies are the critics, what are you expecting?

The first usability test I ran was in 1991. I’ve set up usability labs, I’ve observed hundreds of people interacting with technology and products. My passion has always to do things at speed, turn around results ASAP and engage all stakeholders in the process.  But I’ll talk about that in a later post.  For now I’ll draw on experience of working with organisations that have commissioned usability companies to review their products.  I’ll breakdown the process I have often observed from usability testing vendors, considering both the elapsed time and the actual ‘value added time’ taken.

Day one

The client (usually the business) engage the usability company to audit the usability of the product that is being developed. The consultants will come in and understand the user tasks, roles and goals; the target audience will be identified for recruitment. ‘Value added time’ = 1 hour.

Day two

The team go away and produce a test plan and a recruitment brief for a research agency to find participants. They promise to get it back to the client in a couple of days. They contact their preferred agency who set about recruiting people (let’s assume this is a simple brief for a retail website targeted at young mothers).  Produce test plan (value added time = 3 hours). Send to client for review.

Day three

Client return test plan with a few comments. Update test plan. Value added time = 30 minutes.

Days six-ten

Twelve usability sessions, each an hour long, they do three a day, that is four days of testing. Value added time = 12 hours

Days eleven – thirteen

The team spend three days analysising and synthesising the results, pull supporting video clips and produce a detailed report. Value added time – 15 hours

Day fourteen

The client sees the report for the first time. (Value added time = 2 hours). Interesting results. (IT representation were not invited, they did not commission the report, the product owner wants to see the output first before sharing it with IT).

Day sixteen

The product owner informs the dev team of the changes that need to be made in the light of the usability report. Project manager sucks air through his teeth and says “you’ll need to raise a change request for those items… ha! quick wins they say? hardly… Hmmm, OK, change the labels in the field, we should be able to do that…”

Value added Vs. Elapsed time

The usability company has delivered and their engagement is complete.  From the start of the process to the recommendations hitting the developers who must ultimately action these, for this not-too-fictitious scenario sixteen days have passed, of which only four were spent on value-added tasks, actually doing stuff.

Day n

The product goes live. The usability company are aghast that so many of the changes they reccomended have not been implemented. They place the blame fairly and sqaurely at the door of the developers and reinforce their belief that IT just doesn’t listen, or worse, care about usability. The critics have critcised from their armchair, like the pigs and chickens they are the chickens, participated not committed.

Usability rant part 2>

The journey doesn’t end with the buy button

Take a look at this and tell me what information it conveys to you.  Look at the order status…

Dispatched.

So five days later when I ring up Laskys to find out what has happened to my product I’m told “we’re sorry, the product is out of stock.  Would you like another similar product?”  I’m not exactly a happy chappy to find this out five days after the order has been placed. I’ve been to ‘my account’ to track my order and it tells me that the order has (click on the help icon) “been processed and either has a delivery date or has been delivered”.  Well no actually, it is out of stock and your eCommerce system hasn’t accommodated that scenario in the customer experience.  Sorry Laskys, you lied to me.

“We did actually send you an email on the 8th of March to let you know this”.  So I went back through my mailbox and indeed they had sent me a mail.  I use gmail.  The mail was unread, but I get so many mails I don’t always open them, especially as gmail gives you the title and the first line of the mail.  The mail in question from Customer Service was titled “Your Order” and the following first line; “Thank you for placing an order with Laskys. TheSEBO X4 EXTRA  on order 025″.  nothing there to suggest that it was out of stock.  Just a generic subject and first line of copy.  They had taken my phone number as part of the process; no-one thought to ring me.  Overall a poor experience.  I won’t be buying from Laskys again and I suggest you don’t either.

There’s a lesson here.  It is easy to focus upon the shiny stuff, to get customers converting, clicking on that buy button.  But if the post-buy button experience is lacking; if you haven’t factored in operational excellence into your process, in the long run it is likely to cost you.

Why it pays to think about the whole system, not just your local function

Ability to do bulk price mark-downs? Nice to have.

Today we are looking at a large UK supermarket stock control system.  At the end of the day the staff mark down prices on the short-life items (sandwiches etc).  They have a hand held scanner with a belt printer.  Scan item – print label – stick label on item.  Well that’s what the process is supposed to be, only this takes time (20 seconds per item) and when you have a whole shelf to do is a chore (12 items takes four minutes).  Far easier to just write down the new price on a ‘discount label’ with a sharpie and stick it over the barcode (do the whole shelf in less than a minute).

Where’s the problem in that?  In fact three minutes of waste (waiting time) has been eliminated.  Only it is a problem…

The customer takes the item to checkout and the mark-down label is covering the barcode.  The checkout colleague tries to peel it off to scan, but it doesn’t peel cleanly.  So she manually enters in the SKU. And the mark-down price.  And this has taken 2 minutes for one item and the queue has grown and because of the ‘one in front’ policy they have to open a new checkout and suddenly that small problem at one end of the value chain is replaced by a bigger costlier one at the front end.

But had we not observed this we would never know that bulk price mark-downs on the hand-held device is not a nice to have, it is million dollar requirement.

NFR 001: Make it easy to use

Designing an enterprise application.   Recently someone was grumbling to me about the statement “easy to use”. They felt it was a worthless statement; what does it actually mean? For them it had no meaning and thus should not be used at all.  This is nonsense.  “Easy to use” is a worthy statement that should either be treated as a non functional requirement with clearly defined acceptance criteria, or as a measurable KPI.  So to start the thinking on defining what easy to use must mean to your project, try using the BDD format of given, when, then:

As an Expert User
Given I have had no training
When I have to complete <insert goal>
Then I will be able to accomplish it in under five minutes

As a Novice User
Given I have had no training
When I have to complete <insert goal>
Then I will be able to accomplish it in under seven minutes

Links

Dan North introduces BDD

If you say Log out, log me out

You’ve logged into your on-line banking checked your balance, paid your bills.  What do you do now?  Click on the logout button?

What do you expect will happen now?  Well given that you have actively chosen to log-out (it’s not something you are likely to click on my mistake), you’d expect to do exactly that.  Logout.  The next screen you get will probably be something that thanks you for on-line banking, with a cross sell for a product or two.

That’s what I assume most customers would expect.  So what are Alliance and Leicester thinking about with this screen?

The customer has clicked log-out but they are still logged in?

Why?

“You are still logged in to Internet Banking – before you go have a look at Your offers.”

Excuse me, I logged out, I don’t need to be logged in for you to show me offers.

Worse: “Are you sure you want to log out?”

OF COURSE I WANT TO LOG OUT!!! Why else would I have clicked the link.

Alliance and Leicester fail here in a fundamental usability rule, that of managing the customer’s expectation. In an application where security isn’t paramount this would be an error, in an application where customers expect their action of leaving their secure accounts will do exactly that… but doesn’t, is inexcusable.

What time does the meeting start

Lotus Notes is a gem of a product for usability howlers. Barely a day goes by without me cursing or swearing at it.  Here’s an example of its dumbness…  Without thinking, what time does the meeting start?  I mean, what time do I, in the UK, need to dial in to the conference call?

Does this train go to Bangor?

Over the loudspeaker comes a garbled message “…this train divides at Chester.  Customers for Bangor must travel in the front four coaches of the train”.
There was a group of women behind me talking loudly, one of them picked out part of the message and was worried.  The train guard (sorry, Customer Revenue Protection Officer) walked by.
One of the women got his attention, “Excuse me, we’re going to Bangor?” she said.
“Oh” said the guard.  “You need to get out at Milton Keynes and walk to the front of the train”.
“What? We need to change trains?” the woman replied.
“No, it is the same train, just the front part of it.”
“Is it on the same platform?” Asked the woman.
“Yes, just walk up a little” replied the guard.
“We don’t need to cross over to another platform then?”
“No, it is the same platform, the same train”
“So why can’t we stay on this train then”
“Because this part of the train divides at Chester?”
“But we’re not going to Chester, we’re going to Bangor”
The guard was getting frustrated, “when the train stops at the next station, you just need to get out and walk up the platform, in fact to the next carraige and get on the train there”
“So why can’t we walk through the train to the next carraige?”
“Because it is a different train”
“but this train is going to Bangor isn’t it?  We are on the right train aren’t we?”

And so on until a fellow passenger jumped in “when we get to Milton Keynes, I’ll show you where to go” and at Milton Keynes he led them all off the train to walk past the train divide on the platform and I’ll assume they made it to Bangor in one peice.

The point of this narative is that not everybody “gets it”.  Just because you think something is straight forward or obvious doesn’t mean that your customers will.  You are not your customer, be wary of making assumptions on how people will use your Great New Product.