WHY MEASURE UX?
In the working environment, we measure things in order to make performance visible because we want to evaluate how well we’re doing, define targets and allocate resources for achieving those targets. Making performance visible means it can be managed objectively in order to uphold a certain standard or improve the bottom line. Performance measurement improves communications internally among employees, as well as externally between the organization and its customers.
WHAT IS THE USERINDEX?
The USERindex is a standardised means of measuring the user experience of a digital interface. While usability refers to how successfully a user can carry out a task with a given product, User Experience takes a broader view, taking the user’s individual perception of his entire interaction with the product and even the brand into consideration. Today, the success of a digital interface is determined not only by the functions it offers but also, and maybe even more importantly, by how the user feels about interacting with it.
HOW DO WE MEASURE USER EXPERIENCE?
We have developed the USERindex because we want to measure the user experience of a digital interface in a repeatable, scientific manner. This enables the product owner to determine the quality of the user experience. As a result, the product owner can derive insights for improvements, monitor the evolution of the user experience over time and benchmark it against other products.
“When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarely, in your thoughts advanced to the stage of science.”
― William Thomson, 1st Baron Kelvin
What is the USERindex?
The USERindex is a standardised means of measuring the user experience of a digital interface.
We say “digital interface” because we are well aware that today, when we speak of “digital” we are no longer only referring to computers and desktop/web applications; we are referring to a series of devices such as wearables and smartphones as well as products which are recently being equipped with digital interfaces such as cars and fridges, for example.
While usability refers to how successfully a user can carry out a task with a given product, User Experience takes a broader view, taking the user’s individual perception of his entire interaction with the product and even the brand into consideration. Today, the success of a digital interface is determined not only by the functions it offers but also, and maybe even more importantly, by how the user feels about interacting with it. The subjective, more emotional factor is becoming the deciding factor. The USERindex takes this into consideration.
Each dimension is in turn defined by aspects which comprise both very objective facets of good user experience such as the accuracy of the information and the responsiveness of the interface, as well as very subjective ones such as whether the user liked using the interface.
View image below.
The USERindex comprises 10 questions which have been worded neutrally, avoiding jargon and which can be applied to any digital interface, be it a web application or a wearable device. This makes the index very powerful because it can be used to compare the user experience of different products even if they are very dissimilar in function and interface type.
We believe that an exceptional user experience is determined by 4 factors (which we call USERindex dimensions): Usefulness – Satisfaction – Ease of use – Reliability
We researched our choice of positively expressed statements (as opposed to a mix of negative and positive statements) and deliberately decided to go with them because we don’t believe that we need to negatively word the statements in order to achieve more accurate results. This has also been proven in a study where the SUS was deployed in both its original format of both negative and positive statements as well as an entirely positively worded version and the variance was found to be negligible (SUStisfied? Little-Known System Usability Scale Facts). We believe that the 4 dimensions are correlated and thus improving one dimension will have an impact on the others.
| DIMENSION | ASPECT | STATEMENTS |
|---|---|---|
| Reliability | Performance | I felt the site responded quickly |
| – | Accuracy | I found the information on the site to be accurate |
| Usefulness | Effectiveness | The site does what i need it to do |
| – | Usefulness | The information on the site was useful to me |
| Ease of use | Ease of use | I found the site easy to use |
| – | Navigation | It was easy to find what I was looking for on the site |
| – | Learnability | It didn’t take me long to learn how to use the site |
| – | Efficiency | It didn’t take me long to complete my task on the site |
| Satisfaction | Appeal | I found the design of the site appealing |
| – | Engagement | I liked using the site |
Summarising:
-
The USERindex measures more than just the usability of a digital interface; it takes the emotional aspect of user engagement into consideration
-
The USERindex is a tool for measuring the holistic user experience of a digital interface
-
The USERindex can be used to compare the user experience of different digital interfaces regardless of the devices or technology they run on
-
The USERindex can be used to continuously monitor the evolution of the user experience of your digital interface over time.
Why measure UX?
Today, we are constantly measuring how we perform
We have smart devices that record the number of steps we take, the GPS coordinates of a hike we walked, the depth and temperature of a dive we made. Cheap data storage allows us to accumulate this data, use it to plot our progress in charts and determine correlations. The cloud enables us to easily share our data with friends and like-minded strangers in order to compare achievements, harvest accolades and find new insights.
In the working environment, we measure things in order to make performance visible because we want to evaluate how well we’re doing, define targets and allocate resources for achieving those targets. Making performance visible means it can be managed objectively in order to uphold a certain standard or improve the bottom line. Performance measurement improves communications internally among employees, as well as externally between the organization and its customers.
So the question actually should not be ‘why measure user experience?’, it should be ‘why aren’t you measuring the user experience of your digital interface yet?‘
How do we measure user experience?
We have developed the USERindex because we want to measure the user experience of a digital interface in a repeatable, scientific manner. This enables the product owner to determine the quality of the user experience. As a result, the product owner can derive insights for improvements, monitor the evolution of the user experience over time and benchmark it against other products.
Why not use the SUS?
We like the SUS and we have used it extensively over the past years, but we find it lacking in the following areas:
-
it was developed prior to the emergence of the web (and digital interfaces such as mobiles, wearables and intelligent fridges)
-
it does take the design of a digital interface into consideration
-
it does not measure the user satisfaction.
In our digital era, where features and functions of a digital product are comparatively similar, you differentiate yourselves by the way you engage with your user. This dimension is lacking in the SUS.
“If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.”
― H. James Harrington
How to use the USERindex
Ask your users to rate the 10 USERindex statements on a 5 item Likert scale
The USERindex is very simple to use. Ask your users to rate the 10 USERindex statements on a 5 item Likert scale (from strongly disagree/somewhat disagree/neutral/somewhat agree/strongly agree) and then use this USERindex Calculator spreadsheet to calculate the overall USERindex as well the individual score of each dimension.
When to use the USERindex
-
The USERindex can be used during formative usability tests to determine, mould and monitor the user experience during the development phase.
-
After go-live, it can be used in summative usability tests to validate the expected user experience; to determine a benchmark value and monitor its improvement during the subsequent product lifecycle.
-
It can also be used to compare the user experience of different digital interfaces in a competitive analysis.
How to interpret the score
The value of USERindex ranges from 1 to 5. A digital product with an exceptional user experience will have a USERindex of 4.7 or above; anything between 4.2 and 4.7 is good and a USERindex of below 4.2 means you have uncovered areas for improvement.
If the USERindex is used in a test environment, questions about performance will not reflect realistic use by end users. To minimize the impact of non-production-like circumstances, the score calculation in the USERindex Calculator can be configured accordingly.
Some tips:
-
Make the questions optional in order to avoid forcing users to evaluate an aspect they might not find applicable.
-
Replace the word ‘site’ with a more meaningful term if you feel ‘site’ does not work for your digital product. For instance, use ‘mobile site’ for a mobile website, ‘app’ for a native app, ‘watch’ for connected watch, etc…
Downloads
Use this spreadsheet for entering your user data and generating charts.
Unformatted statements for copying into your online survey
Printable questionnaires
Applications
Measuring Inclusive Experiences
The USERindex is a benchmarking score created in 2014. To date, the USERindex has been used in over 700 studies across 50 countries and has been adopted by companies worldwide as their default benchmarking system.
The USER in USERindex represents 4 dimensions: Usefulness, Satisfaction, Ease of use and Reliability. Capturing these aspects of the experience has become crucial as digital technology is now more prevalent in consumers’ daily lives than ever.
One USERindex for All
The USERindex was originally developed with digital products such as websites and mobile apps in mind. Soon after, it was successfully used to measure both the digital and hardware experiences of IoT devices.
In the past few years, we have piloted the USERindex in inclusive design UX studies with participants with disabilities and people who use assistive technologies. Our conclusion is that the USERindex is suitable for capturing and measuring the experience of participants in inclusive design UX studies.
This is encouraging, since a significant portion of the world’s population has a disability of sorts; about 15 percent of the world’s population and 30 to 50 percent of older adults in the US. And let’s not forget people all over the world who break their arm or lose their glasses, resulting in a temporary disability.
Best Practices
Here are a few best practices for interpreting scores in inclusive design UX studies.
Track Disability and Assistive Tech
Benchmarks such as the USERindex rely on self-reported feedback as the user rates the statements. A person’s frame of reference can significantly influence their ratings. For example, screen reader users are accustomed to challenging user experiences. As a result, what may seem a poor experience to some is a “good” experience for screen reader users. This can skew the ratings and it is thus a good idea to track both the disability and the assistive technology used.
Define User Personas Based on Assistive Tech
The different types of disabilities (e.g. vision, cognitive, motor impairment) are broad categories that cover a number of conditions and severity of these conditions. This variety makes it challenging to define user personas based on condition. We have found that defining user personas primarily based on assistive technology works well to collect consistent data points since the experience heavily depends on the type of assistive tech used.
Learn Through Detailed Analysis
The USERindex is represented as a single score. Yet, reviewing the scores for each statement and dimension across participants can be very insightful and help identify specific areas where the digital product may be particularly lacking.
Our world needs to be inclusive; access to a benchmarking score to measure the digital experience for everyone is simply a must have.
Measuring Delightfulness
What is delightful UX?
Delightful UX is a user experience that literally puts a smile on your face (or an inner smile, if you’re not the type to show emotion easily 😉) simply because the interaction brings you joy. Feelings matter, and creating a positive emotional reaction influences the user’s behaviour and opinion towards the digital product they are using.
However, if the product itself falls short of the user’s needs and expectations, its delightfulness will not make up for it. This is expressed very nicely in Aaron Walter’s hierarchy of user needs, which is based on Maslow’s hierarchy of needs.
Walter remaps Maslow’s hierarchy of needs to the needs of users. He defines the most basic need a user has as functionality, followed by reliability, usability and then pleasure. Taking an app to purchase train tickets as an example: a user’s first need would be that the app does just that: allow the user to buy a train ticket (= functionality). The next need the user would have is that the app is reliable i.e. the data it provides is accurate, the payment is secure etc. and once these needs are met, the user will look for usability. Was it onerous to buy that ticket, or did the app make it seem effortless because it was so intuitive? Each of these needs is a potential exit for the user and a reorientation versus another app that might satisfy their needs better. And only once all of these needs are met, the user will appreciate if the app is pleasurable.
“Designers creating interfaces that are just usable is like a chef creating food that’s just edible.” – Aaron Walter
How do you measure delightfulness?
Nielsen differentiates between surface and deep delight; surface delight being local and contextual, while deep delight is more holistic and the one we should aim to achieve.
You can measure the surface delight simply by asking users to rate how delighted they were on a 5-point (or 7-point) labelled (or non-labelled) scale. Similarly, you could use a Likert scale and ask users to rate how strongly they felt delighted. Any of these would correctly measure the level of delight the users felt. But it wouldn’t inform you how the other aspects of the experience were perceived.
The USERindex measures a users’ subjective user experience by evaluating all 4 dimensions of their experience: Usefulness, Satisfaction, Ease of use and Reliability. These 4 dimensions map directly to Walter’s pyramid. The USERindex can thus be used to measure a product’s delightfulness as well as evaluate how the other 3 dimensions of the experience performed and identify on which level of the pyramid the product fails to live up to expectation. Try it out and do please share your thoughts!
—
Aaron Walter – Designing for emotion. http://www.abookapart.com/products/designing-for-emotion
Nielsen Norman Group – A Theory of User Delight: Why Usability Is the Foundation for Delightful Experiences https://www.nngroup.com/articles/theory-user-delight/
About Us
The USERindex was created by 2 user experience professionals
Why we created the USERindex
In our usability testing projects, we’ve used a number of different metrics over the years. While all metrics have their merits, we’ve been looking for something that is ‘quick and easy’ (think SUS), but also collects information on today’s metrics such as appeal and satisfaction. And we want it to be freely available (we don’t believe anyone should be paying for a good metrics system).
Our vision for the USERindex
We created and fine-tuned the USERindex, and find that it’s working well for us. We encourage anyone to use it. Please refer back to the USERindex.org site for further information. If you wish to anonymously submit your data to be added to the historical database, please send your file to feedback at userindex.org.
Contact Us
Do you have questions or feedback you would like to share with us? Then mail us at feedback at userindex.org. We’d love to hear from you.
Share your data
If you wish to anonymously submit your data to be added to the historical database, please send your file to feedback at userindex.org. We will use your data to continuously improve our analysis of the correlations between the dimensions.