dancingmango

Usability and the $1 trillion mistake

Is this a case of fat fingers, a usability flaw or poor design that enabled a Citigroup trader to have placed an order to sell $16 billion, instead of $16 million? P&G shares plunged by 23% because of this individual erroneous trade. What followed was the algorithms kicked in and automated trading saw the Dow loose a tenth of its value in less than half an hour. (And Accenture dropped to 4 cents down from $42!)

Before we go blaming human error, questions should be asked why that error occurred. How can someone make such a simple mistake so easily? Was it a case of entering two many 0s? (Don’t stop to look or think, answer the question as soon as you’ve read it – how many zeros are there in this number? 160000000. Same thing again, how many zeros are in this number 12,000,000. That’s a bit easier isn’t it. Only an ‘N’ separates the B from the M on a qwerty keyboard, in a hurry, easily mistaken?)

I’d start by looking at human factors and experience design, and question why (assumption here) the IT team who implemented the system didn’t have before a UX designer on the team to think about the human factors. Could this be the most costly example of poor design?

Article: Big drop, was it all a mistake?

Critiquing the critics (usability rant part 1)

Michael Winner may be a good food critic, but if you were looking for someone to cook you the finest meal for your budget, I doubt he would be your first choice.Same with film critics, they may be able to write an insightful and critical review, but would you want them directing a film for your budget? Would you want Jakob Neilsen, who is essentially a usability critic, to design your website? I mean, take a look at his site!

When you are building a product, you get a usability company in because you know that usability is a good thing that you want to have. If usability companies are the critics, what are you expecting?

The first usability test I ran was in 1991. I’ve set up usability labs, I’ve observed hundreds of people interacting with technology and products. My passion has always to do things at speed, turn around results ASAP and engage all stakeholders in the process.  But I’ll talk about that in a later post.  For now I’ll draw on experience of working with organisations that have commissioned usability companies to review their products.  I’ll breakdown the process I have often observed from usability testing vendors, considering both the elapsed time and the actual ‘value added time’ taken.

Day one

The client (usually the business) engage the usability company to audit the usability of the product that is being developed. The consultants will come in and understand the user tasks, roles and goals; the target audience will be identified for recruitment. ‘Value added time’ = 1 hour.

Day two

The team go away and produce a test plan and a recruitment brief for a research agency to find participants. They promise to get it back to the client in a couple of days. They contact their preferred agency who set about recruiting people (let’s assume this is a simple brief for a retail website targeted at young mothers).  Produce test plan (value added time = 3 hours). Send to client for review.

Day three

Client return test plan with a few comments. Update test plan. Value added time = 30 minutes.

Days six-ten

Twelve usability sessions, each an hour long, they do three a day, that is four days of testing. Value added time = 12 hours

Days eleven – thirteen

The team spend three days analysising and synthesising the results, pull supporting video clips and produce a detailed report. Value added time – 15 hours

Day fourteen

The client sees the report for the first time. (Value added time = 2 hours). Interesting results. (IT representation were not invited, they did not commission the report, the product owner wants to see the output first before sharing it with IT).

Day sixteen

The product owner informs the dev team of the changes that need to be made in the light of the usability report. Project manager sucks air through his teeth and says “you’ll need to raise a change request for those items… ha! quick wins they say? hardly… Hmmm, OK, change the labels in the field, we should be able to do that…”

Value added Vs. Elapsed time

The usability company has delivered and their engagement is complete.  From the start of the process to the recommendations hitting the developers who must ultimately action these, for this not-too-fictitious scenario sixteen days have passed, of which only four were spent on value-added tasks, actually doing stuff.

Day n

The product goes live. The usability company are aghast that so many of the changes they reccomended have not been implemented. They place the blame fairly and sqaurely at the door of the developers and reinforce their belief that IT just doesn’t listen, or worse, care about usability. The critics have critcised from their armchair, like the pigs and chickens they are the chickens, participated not committed.

Usability rant part 2>

Follow fast

I’ll pick on a random industry. Let’s say you are an airline. Part of your digital strategy includes a refresh to your website (maybe you were inspired by this presentation I did a while ago on digital for airlines!). You’ve built a business case and secured funding for the project.  First things first, you get a design agency in and set them to work.

Some sort of competitor analysis is performed that proabably includes features and functions that “we like”, (for example ‘the tactile sliders in kayak.com. We like!  And an iPhone-like coverflow, got to have one of those…)

The information architect takes these ideas away and starts building wireframes and the creative team produce designs that bring these wires to life.  The team come up with lots of new, innovative ideas.  This is after all a ‘refresh’, and ‘innovation’ was probably one of the words in the brief.  The creative is fresh and ‘of the moment’, the IA has developed some new interaction models that are unique and compelling.  The business is sold on a new, innovative way of interacting with their customers, something that no one else does and will blow all their competitors away.

I’ve been bouncing ideas around with Luke Barrett (and because he doesn’t blog, I’ll write them down for him) around this approach; specifically the value of innovation against ‘follow fast’.

Luke reminded me of a project we worked on together many years ago. Before we started designing webpages we did usability testing. We did usability testing of the competitors, and of sites that were getting a lot of press as ‘innovative’.  This was at a time that boo.com had just started and the client were talking about how cool an avatar would be on their site, just like boo. We put people in front of boo.com and watched them suffer. Clearly the avatar was an idea good on paper, terrible on execution.  So we killed it.  Not on our site.

We observed what worked and what didn’t on a multitude of sites with real users. Then, like magpies, we took what was good and worked. Nothing particularly innovative, (let other people do that), we took the best of what existed and delivered on that.

So back to our airline. How about a different approach where they start by usability testing their competitors. Ask participants to book tickets on competitor websites. Understand what interaction elements work, what don’t.

Those kayak sliders, cool for geeks (maybe), but how about the target audience that flies and buys online with you?  It may not be cutting edge design, but Does a drop down work better?

The coverflow may be cool on your iPhone, but how successful is it for people seeking a holiday?  A static list has worked for websites till now (and it wasn’t so long ago that horizontal scrolling was the Great Taboo), just because Apple do something that looks cool for a particular purpose on their products, doesn’t mean you have to follow by scrapping your navigation.

There are no prizies for (design) innovation (other than for some award that the design agency may covet). The only metric that counts is conversion rates and the ability of the website to deliver the business case. Leave others to do the crazy innovation stuff, let others go through the dip when they launch new features, make sure you have got the platform and expertise right and be ready to follow fast.

Who would turn off the wrong engine?

In designing user interfaces there’s a lot we can learn from systems where failure to consider human factors has resulted in terrible consequences.

On 8th January 1989 British Midland Flight 92 crashed whilst attempting an emergency landing. There had been a fire on one of the engines which led to its malfunction. The captain reacted by shutting down the engine.  Only he shut down the wrong engine. With no power, approaching East Midlands airport the captain manged to glide the Boeing 737-400 to avoid Kegworth village but crashed short of the runway.  47 people died.

The investigation into the Kegworth air disaster identified engine malfunction (the engine used in the aircraft was an upgrade of an existing engine and had not been field-tested) as causal factor, however the report concentrated upon the failure of the flight crew to respond accurately to the malfunction.  Human error was the primary cause.

The truth is a little more complicated than that.  Why does a captain with over 13,000 hours flying experience and a first officer with over 3,000 hours experience shut down the wrong engine?

A number of human factors contributed to the disaster including organisational issues (refer to this paper for discussion of the role of managerial failures in disasters) and cognitive overload.  But of equal importance (and indeed buried in the appendices of the Investigation Report appendices) is the issue of design. Around 50% of accidents and incidents in the aircraft and nuclear industries have a root cause in design (source).

Take a look at the cockpit controls (taken from the investigation report). The left image is for the earlier 300 series and the right for the 400 series aircraft on which the captain had only 23 hours experience after a one day training course.

The actual cause of the engine malfunction was a broken turbine, itself the result of metal fatigue caused by excessive vibration (source).  Had the Captain noticed the Vibration Warning display he probably would not have made the wrong decision.

The Vibration Warning display on the new 400 series was in a different place to the 300 series, but more critically it was designed to be significantly smaller “the dial on the vibration meter was no bigger than a 20 pence piece and the LED needle went around the outside of the dial as opposed to the inside of the dial as in the previous 737 series aircraft” (Source: Wikipedia).  Take a look at the arrow on the left hand image, the display dials on the 300 series use mechanical pointers. On the 400 series they were redesigned with short LEDs rotating around the numbers. These, as the investigation report noted “are much less conspicuous than mechanical pointers, acting more as scale markers, and providing less immediate directional information).

The report criticised the layout of the instrumentation and helpfully suggested an improved layout.  The layout was (and as far as I can tell, remains in 737s) split into primary instruments and secondary instruments.  The issue with this layout is that the dials are not spatially aligned with their associated power levers.  If the pilot is focussing upon the primary instrumentation, the secondary instrumentation is in peripheral view.  This layout will lead to attention based around specific instruments rather than engines.

Compare this to an alternative design that the report provides where the dials are aligned to their associated power levers.  The report recognises the design trade-offs here but concludes that to break the left-right mental association with the engine position was probably not the most optimal solution.

This paper describes the issue well:

The 737 involved in the East Midlands crash had flight deck engine information that lead to confusion under mental pressure. Placing the secondary information sets for both engines to the right of the primary set broke the implied rule set by all the other engine information, that the left engine had left hand controls and indicators (and vice versa). If one assumes that the optimum positioning of indicators is the one that requires the least mental processing then a simple symmetry about the aircraft centre line seems appropriate. The actual positions required a mental spatial transposition of one set of dials to the other side… The readability of the indicators had been reduced by the substitution of electro-mechanical readouts with electronic readouts, but which simulated the old design. Possibly the redesign to electronic readouts should have taken the opportunity to use a rather different layout, possibly with linear indicators rather than rotary ones.

OK, so lots of words, but what is the point of this to what I usaully blog about?  The issue is one of design and layout and who’s responsibility is it.  In designing user interfaces UCD is often seen as a luxury, developers believe that they can design a GUI as well as anyone, and stakeholders (especially on internal projects) will question the value that a UCD person can bring to the project.  Does a developer or an engineer by training and instinct stop to ponder the human factor and the human consequences of the GUI layout? What are the consequences of this?  As can be seen from Kegworth, seemingly minor changes to the control layout can have a signficant impact on the safety of a complex system.

Thinking about value in terms of advantage and benefit

A product rarely sells itself.  What sells a product is the advantage it brings and the benefits it delivers to the customer.  It is the benefit of the product that sells rather than the product itself. What is the advantage of the requirement you are stating, and what is the benefit it will bring the customer?

Let’s start with a product.  Think broadband.  It’s dull.  Put 10MB in front of it and it is still dull.

Now think about the advantage that 10MB broadband brings.  The advantage is that it is fast.  Lightning fast.

Now think about the benefit which that advantage brings.  The benefit is that you can download an MP3 tune in seconds rather than minutes with your old dial up connection.  You are no longer selling broadband, but the experience that it brings.

Let’s consider IT requirements to be products.  A dull list, a thick document gathering dust. How do you prioritise one requirement over another?  What is more important?

Agile introduces ’stories’ as the requirement product.  They are written in the format ‘As a <role>, I want <a feature>, so that <some benefit is achieved>’.  It is the ‘So that’ which is usually the hardest part to articulate, yet it is the most important part of the story.

Liz Keogh describes how prompted by Chris Matts her preferred narative reads:

In order to <achieve some value>
As a <role>
I want <some feature>.

Applying the marketing thinking to how the story will “achieve some value”, don’t just define that value in the advantage it will bring, rather also consider the benefit it will deliver to the user.  The two are different.  There maybe a business advantage to delivering some feature, but if the benefit to the end user can’t be articulated, it’s real value must be questioned.

Design vision

Don’t be fooled into thinking that you don’t need to do any design when you adopt Agile.  Agile development strives to deliver business value early and often, focusing on getting working software to market as soon as possible rather than dwelling in documentation and ‘analysis paralysis’.  But let’s be clear, “business value” and “working software” are not the same thing.  You can quite easily get something into production that fails to generate revenue, decrease costs or whatever other yardstick you use for ‘value’.  What differentiates the two of them is design.  I don’t mean big up front design that details all the features and provides a concrete spec, I mean a design vision that articulates what the product goals are and a roadmap for getting there.  And what is a design vision?  A short statement of intent is a good place to start, and soon after a user interface mocked up in pen and ink.  It is cheap and easy and helps bridge the path from idea to execution.

What is the story?

One of the problems with IT development is that it is tactical and piecemeal in its approach. Functionality is added in response to competitor or broader market activity. Expect to see an increasing number of brands doing something ’social’ (and tactical) on the web, but don’t expect these new initiatives to be (strategic) seamlessly integrated into the existing digital channel offering.

This piecemeal approach extends to larger initiatives as well. In refreshing the website or developing new digital channels such as mobile and TV, IT will typically build out features and functionality prioritised upon their perceived individual business value regardless of what the sum value of the proposed release is. (Focusing all your effort of building functionality that delivers to your bottom line will seldom be as successful as you predict if it is not supported by features that meet the customers needs).

So when it comes to thinking about new features and functionality, where’s the best place to start? I’d suggest collaboratively, thinking around the customer. Collaboration is important to ensure that everyone starts with the same vision. It needs to be shared it with the broader audience, the product teams, IT; anyone whose day to life life will be touched by the project when it starts. I’d argue that you cannot start this soon enough. You don’t need to spend time doing analysis, interviewing all stakeholders individually, coming up with a document that is circulated and reviewed and re-written (with all the delays and waste that such a process incurs). Start the process getting all those stakeholders off-site for an afternoon and get the thoughts out on the table.

Commence with a presentation that introduces thinking in terms of customers and customer journeys. The below SlideShare presentation does this for the airline industry, addressing a new customer experience across channels. I acknowledge that it is pretty simple and doesn’t touch on half the ideas that airline executives may have. That is the point, it is just enough to get people thinking about different customer types and their touchpoints without getting bogged down in detail. This is what we want the participants of the off-site to share.

Once we’ve been through the presentation we break out into small groups a, each taking an individual customer (or persona) and build up a story; a day in the life of… (It is important not to forget the internal users of the system). These breakouts last 15-20 minutes with ten minutes for the team to play back their findings. As they build out a richer picture of the customer interactions they are asked to sketch out what the user interfaces may look like. The process is rapid, intense and iterative, but always focussing upon the customer journey; how will the customer realise their goals. When the teams tell their stories an analyst captures the essence of the requirements on index cards. The final exercise is to lay all these cards on the table and ask the team to group them into similar areas then prioritise them according to their perceived importance. In an afternoon you will have achieved four things. Firstly, you will have captured a vision for the new product in less than a day, with all stakeholders understanding not only the vision itself, but also the process that developed it and the concerns and issues that different parts of the business have with it. Secondly you will have an initial prioritised roadmap for its development. This will change, but it is a good strawman to circulate. Thirdly you will have introduced all the stakeholders together – projects succeed or fail based upon the strength of relationships and getting people engaged from the start will go a long way to creating shared ownership. And finally you will have generated energy, engagement and traction; to do the business case and to get the project started, recognising that just one part of the business having a vision is not going to bring it to the life that they dream.

Are you experienced?

“For you who have had the experience, no explanation is necessary. For you who have not, none is possible.”

I’m going to attribute that saying to Ram Dass, a Harvard professor who via psychedelic experiences ended up a spiritual teacher in the Eastern Tradition.

The problem with too much software/web design is that it is produced by people who have just not had the experience, or do not see the experience as relevant to their organisation or domain. They just don’t “get it”.

(“For you who have an apple product, no explanation is necessary, for you who have not, none is possible?” Cue “it’s an enterprise application we’re buiding, not a ****ing iPhone”).

If we want to build memorable and compelling products, we need to focus upon the experience. To dwell on the feature list or functional requirements is to build mediocrity. Nothing wrong with mediocrity if you don’t want to delight your customers or increase the performance of your workforce. Without considering experience you will miss innovation and added value.

So how to focus upon experience? Get your team to undertake different tasks to get under the skin of what customers go through.

Telco product?
Spend time in a retail outlet and watch different customers buy phones
Go into all the phone shops on the high street and ask the rep “hello, I want a mobile phone”. Suspend all your knowledge about phones and tariffs. How do they sell?
Leave your blackberry at home for a day (how dies it feel? How does it change what you do?)
Download instruction manuals from different phones from manufacturers websites

Travel product?
Go into a travel agents and ask for a holiday “somewhere hot and cheap in February”

Credit card product?
Ask to borrow money from someone you don’t know (how does it feel?)
Apply for a credit card at another bank
Collect all the Credit Card / loan direct mail and emails that you and you get sent over a week, photo / scan all the credit card advertisements you see in a week
Go into a car sales room and look to buy a car on credit

Supermarket product?
Get behind the till for a day (In the UK, at least a few years ago, all senior executives in both Tesco and Sainsburys spent time in the stores over the Christmas period)
Ask a shop assistant to help you find an obscure product that is not in stock
Go into a store with a shopping list and a single bank note, (no credit cards)
Go to the pharmacy when it is busy and ask to buy the morning after pill

Extend your team
Bring in representatives from completely unrelated parts of the business to participate in brainstorming sessions. Building a “youth” social networking website? Get someone from legal or corporate finance to join in. (Get’s you thinking along the lines of extreme characters – here and here [pdf]). Working on a complex exotic financial instruments? Get a few PAs to join in. You may learn something (that your product is too complicated and even you can’t explain what it really is).

I’m sure you can come up with better exercises. The object is that with this collection of experiences and related emotions new ideas can be brought to the table. They can offer insights from another, different perspective, providing more chance of business innovation being realised. More importantly, if you have an emotional attachment to the product you are building through real experience, you are more likely to build a better product that will fullfil the needs of and goals of the target audience in the way they want. The day your enterprise application team all have iPhones will be the day you start building better enterprise applications. For them, no explanation will be necessary. They’ll just “get it”.

A new page

This blog was getting tired in its design so I’ve given it an overhaul, including introducing some widgets.  (What an awesome piece of software Wordpress is).  I’ve also added a new page with a bunch of published papers.  some classics in there (if I do say so myself) such as “Heat stress in night clubs” and “Occupational disorders in Ghanaian Subsistence farmers” !!.  The rest of the dancingmango site was not built using wordpress, so to update it all in one go would be time consuming and of little value.  OK, so there is inconsistency across the site and as a UI guy that hurts, but it is one of the trade-offs that needs to be made.

Where are the missing floors?

It is fairly standard practice in Hong Kong for buildings to have no thirteenth or fourteenth floors. They are considered unlucky numbers. Not sure what happened to the first, second and fifth floor here. And back-to-front button numbering that is neither in the telephone format nor the phone format. There’s a couple of lessons to learn here; when designing human-technology interactions consider cultural norms and existing design stereotypes. (Sorry, its the Human Factors conditioning in me that notices such things).