WHY MEASURE UX?
In the working environment, we measure things in order to make performance visible because we want to evaluate how well we’re doing, define targets and allocate resources for achieving those targets. Making performance visible means it can be managed objectively in order to uphold a certain standard or improve the bottom line. Performance measurement improves communications internally among employees, as well as externally between the organization and its customers.
WHAT IS THE USERINDEX?
The USERindex is a standardised means of measuring the user experience of a digital interface. While usability refers to how successfully a user can carry out a task with a given product, User Experience takes a broader view, taking the user’s individual perception of his entire interaction with the product and even the brand into consideration. Today, the success of a digital interface is determined not only by the functions it offers but also, and maybe even more importantly, by how the user feels about interacting with it.
HOW DO WE MEASURE USER EXPERIENCE?
We have developed the USERindex because we want to measure the user experience of a digital interface in a repeatable, scientific manner. This enables the product owner to determine the quality of the user experience. As a result, the product owner can derive insights for improvements, monitor the evolution of the user experience over time and benchmark it against other products.
“When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarely, in your thoughts advanced to the stage of science.”
― William Thomson, 1st Baron Kelvin
What is the USERindex?
The USERindex is a standardised means of measuring the user experience of a digital interface.
Each dimension is in turn defined by aspects which comprise both very objective facets of good user experience such as the accuracy of the information and the responsiveness of the interface, as well as very subjective ones such as whether the user liked using the interface.
View image below.
The USERindex comprises 10 questions which have been worded neutrally, avoiding jargon and which can be applied to any digital interface, be it a web application or a wearable device. This makes the index very powerful because it can be used to compare the user experience of different products even if they are very dissimilar in function and interface type.
DIMENSION | ASPECT | STATEMENTS |
---|---|---|
Reliability | Performance | I felt the site responded quickly |
– | Accuracy | I found the information on the site to be accurate |
Usefulness | Effectiveness | The site does what i need it to do |
– | Usefulness | The information on the site was useful to me |
Ease of use | Ease of use | I found the site easy to use |
– | Navigation | It was easy to find what I was looking for on the site |
– | Learnability | It didn’t take me long to learn how to use the site |
– | Efficiency | It didn’t take me long to complete my task on the site |
Satisfaction | Appeal | I found the design of the site appealing |
– | Engagement | I liked using the site |
Summarising:
-
The USERindex measures more than just the usability of a digital interface; it takes the emotional aspect of user engagement into consideration
-
The USERindex is a tool for measuring the holistic user experience of a digital interface
-
The USERindex can be used to compare the user experience of different digital interfaces regardless of the devices or technology they run on
-
The USERindex can be used to continuously monitor the evolution of the user experience of your digital interface over time.
Why measure UX?
Today, we are constantly measuring how we perform
-
it was developed prior to the emergence of the web (and digital interfaces such as mobiles, wearables and intelligent fridges)
-
it does take the design of a digital interface into consideration
-
it does not measure the user satisfaction.
“If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.”
― H. James Harrington
How to use the USERindex
Ask your users to rate the 10 USERindex statements on a 5 item Likert scale
-
The USERindex can be used during formative usability tests to determine, mould and monitor the user experience during the development phase.
-
After go-live, it can be used in summative usability tests to validate the expected user experience; to determine a benchmark value and monitor its improvement during the subsequent product lifecycle.
-
It can also be used to compare the user experience of different digital interfaces in a competitive analysis.
-
Make the questions optional in order to avoid forcing users to evaluate an aspect they might not find applicable.
-
Replace the word ‘site’ with a more meaningful term if you feel ‘site’ does not work for your digital product. For instance, use ‘mobile site’ for a mobile website, ‘app’ for a native app, ‘watch’ for connected watch, etc…
Downloads
Use this spreadsheet for entering your user data and generating charts.
Unformatted statements for copying into your online survey
Printable questionnaires
Applications
Measuring Inclusive Experiences
The USERindex is a benchmarking score created in 2014. To date, the USERindex has been used in over 700 studies across 50 countries and has been adopted by companies worldwide as their default benchmarking system.
The USER in USERindex represents 4 dimensions: Usefulness, Satisfaction, Ease of use and Reliability. Capturing these aspects of the experience has become crucial as digital technology is now more prevalent in consumers’ daily lives than ever.
One USERindex for All
The USERindex was originally developed with digital products such as websites and mobile apps in mind. Soon after, it was successfully used to measure both the digital and hardware experiences of IoT devices.
In the past few years, we have piloted the USERindex in inclusive design UX studies with participants with disabilities and people who use assistive technologies. Our conclusion is that the USERindex is suitable for capturing and measuring the experience of participants in inclusive design UX studies.
This is encouraging, since a significant portion of the world’s population has a disability of sorts; about 15 percent of the world’s population and 30 to 50 percent of older adults in the US. And let’s not forget people all over the world who break their arm or lose their glasses, resulting in a temporary disability.
Best Practices
Here are a few best practices for interpreting scores in inclusive design UX studies.
Track Disability and Assistive Tech
Benchmarks such as the USERindex rely on self-reported feedback as the user rates the statements. A person’s frame of reference can significantly influence their ratings. For example, screen reader users are accustomed to challenging user experiences. As a result, what may seem a poor experience to some is a “good” experience for screen reader users. This can skew the ratings and it is thus a good idea to track both the disability and the assistive technology used.
Define User Personas Based on Assistive Tech
The different types of disabilities (e.g. vision, cognitive, motor impairment) are broad categories that cover a number of conditions and severity of these conditions. This variety makes it challenging to define user personas based on condition. We have found that defining user personas primarily based on assistive technology works well to collect consistent data points since the experience heavily depends on the type of assistive tech used.
Learn Through Detailed Analysis
The USERindex is represented as a single score. Yet, reviewing the scores for each statement and dimension across participants can be very insightful and help identify specific areas where the digital product may be particularly lacking.
Our world needs to be inclusive; access to a benchmarking score to measure the digital experience for everyone is simply a must have.
Measuring Delightfulness
What is delightful UX?
Delightful UX is a user experience that literally puts a smile on your face (or an inner smile, if you’re not the type to show emotion easily 😉) simply because the interaction brings you joy. Feelings matter, and creating a positive emotional reaction influences the user’s behaviour and opinion towards the digital product they are using.
However, if the product itself falls short of the user’s needs and expectations, its delightfulness will not make up for it. This is expressed very nicely in Aaron Walter’s hierarchy of user needs, which is based on Maslow’s hierarchy of needs.
Walter remaps Maslow’s hierarchy of needs to the needs of users. He defines the most basic need a user has as functionality, followed by reliability, usability and then pleasure. Taking an app to purchase train tickets as an example: a user’s first need would be that the app does just that: allow the user to buy a train ticket (= functionality). The next need the user would have is that the app is reliable i.e. the data it provides is accurate, the payment is secure etc. and once these needs are met, the user will look for usability. Was it onerous to buy that ticket, or did the app make it seem effortless because it was so intuitive? Each of these needs is a potential exit for the user and a reorientation versus another app that might satisfy their needs better. And only once all of these needs are met, the user will appreciate if the app is pleasurable.
“Designers creating interfaces that are just usable is like a chef creating food that’s just edible.” – Aaron Walter
How do you measure delightfulness?
Nielsen differentiates between surface and deep delight; surface delight being local and contextual, while deep delight is more holistic and the one we should aim to achieve.
You can measure the surface delight simply by asking users to rate how delighted they were on a 5-point (or 7-point) labelled (or non-labelled) scale. Similarly, you could use a Likert scale and ask users to rate how strongly they felt delighted. Any of these would correctly measure the level of delight the users felt. But it wouldn’t inform you how the other aspects of the experience were perceived.
The USERindex measures a users’ subjective user experience by evaluating all 4 dimensions of their experience: Usefulness, Satisfaction, Ease of use and Reliability. These 4 dimensions map directly to Walter’s pyramid. The USERindex can thus be used to measure a product’s delightfulness as well as evaluate how the other 3 dimensions of the experience performed and identify on which level of the pyramid the product fails to live up to expectation. Try it out and do please share your thoughts!
—
Aaron Walter – Designing for emotion. http://www.abookapart.com/products/designing-for-emotion
Nielsen Norman Group – A Theory of User Delight: Why Usability Is the Foundation for Delightful Experiences https://www.nngroup.com/articles/theory-user-delight/
About Us
The USERindex was created by 2 user experience professionals
Contact Us
Share your data