The most tempting question to ask when trying to understand somebody’s behavior is to ask them ‘why?’. ‘Why do you prefer option A or option B?’; ‘why did you choose template A?’. These are all questions users might be faced with during a usability test. The assumption behind this logic is that people have the ability to introspect into the reasons behind their behaviors. However, as we already know from Thinking fast and slow in UX, people not always make deliberative choices. In fact, not only are we most of the time using System 1, we are genuinely poor at introspection into the reasons behind our own behavior.
Nisbett and Wilson and their pantyhose study
Over thirty years ago Nisbett and Wilson demonstrated that when asked to introspect people tend to confabulate the reasons for which they have made a particular decision. In their classic study the researchers asked customers outside a brain store to evaluate the quality of four pairs of ladies’ pantyhose, arranged side by side and labelled A, B, C and D respectively from left to right. Most people preferred pair D (right-most), and fewest pair A (left-most). Little did the customers know, but all the pairs of stocking were identical.
The reason behind the clear preference for pair D was simple an order effect. However, when the researchers asked people why they have chosen that particular pair, people identified all types of different attributes as a potential reason, such as superior knit or elasticity.
Under no time pressure people show a natural preference for the right most item. Under time pressure people prefer the left most item.
Nisbett and Wilson even asked people whether they may have been influence by the order of the items, nobody but one (as it happens, psychology student) thought anything else than the attribute they have names had affected their choice. Instead people, unaware of the true motivations behind their choice, made up plausible reasons for their pick.
What does this mean for UX research?
This well established phenomenon has potentially two important implications for UX research.
Firstly, testing should also focus on indirect measures of usability. Consider task completion, task time and number of errors as potential markers for usability.
Secondly, take order effects into consideration when running comparison studies. When asking users to choose between three different potential layouts (A, B, & C) use counterbalancing.
An example for counterbalancing three potential layouts across users.
A B C
B C A
C A B
Taking psychological principles into account in usability testing can not only save time and money, but also prevent potentially changing your design for all the wrong reasons.
Listen to your users by all means, but remember to support those insights with behavioral data.