UMUX (Usability Metric for User Experience)

What you’ll find in this article:

• What is UMUX?
• UMUX vs. UMUX-Lite
• When to Use UMUX 
• Score calculation 
• How to get the most out of UMUX data 
• Pros and Cons of UMUX 
• Using UMUX with Qualaroo 
• Conclusion

What is UMUX (Usability Metric for User Experience)? 

UMUX is a short qualitative assessment designed to measure the general usability of a system.

Developed at Intel in 2010 by Kraig Finstad and his colleagues, the UMUX questionnaire was intended to offer a shorter alternative to the 10-item SUS questionnaire. Additionally, it was meant to address the new definition of usability set forth by the International Organization for Standardization or ISO. Whereas SUS assesses perceived usability and learnability, UMUX targets usability by assessing effectiveness, efficiency, and satisfaction.    

UMUX vs. UMUX-Lite

UMUX is a simple four-item questionnaire listing two positive and two negative statements that respondents rate agreement with on a five or seven-point Likert scale. UMUX-Lite is the shorter version which only contains the positive statements. 

While length is an obvious difference between the two, the tone is perhaps the most important difference between them. After conducting follow-up research on UMUX, Jim Lewis and others from measuringU developed UMUX-Lite in 2013. Their argument was that the alternating tone of UMUX created an artificial two-factor structure even though it was only meant to measure one factor, usability. That’s why UMUX-Lite only contains the positive items. 

Both assessments can be placed on either a 5 or 7-point Likert Scale. If you need help deciding which one is best for you, here’s an article comparing the benefits of each. The overall recommendation though is that if you are running this questionnaire for the first time, you’re better off using a 7-point scale in order to get more granular data.  

UMUX Statements 

1. This website/ product/ tool/ software/ prototype capabilities meet my requirements.
2. Using this website/ product/ tool/ software/ prototype is a frustrating experience
3. This website/ product/ tool/ software/ prototype is easy to use.
4. I have to spend too much time correcting things with this website/ product/ tool/ software/ prototype 

UMUX-Lite Statements 

1. This website/ product/ tool/ software/ prototype capabilities meet my requirements.
2. This website/ product/ tool/ software/ prototype is easy to use. 

When to Use UMUX 

Like most of our usability templates, UMUX is a great questionnaire to use after usability testing. 

However, UMUX and UMUX-Lite are also considered great tools for evaluating usability throughout product use because of how compact they are. You can consider using them as a one-time solution for specific use cases or during different phases of the design process like prototyping and validation. 

After onboarding 

First impressions matter. When a user is using your product for the first time, you want their experience to be seamless and straightforward. They should feel like your tool is easy to use and that it can help them quickly accomplish what they need to do. If you want to know how your first impression or onboarding experience is perceived by users, the UMUX statements allow you to understand just that. 

Following a usability test 

While there are a variety of quantitative metrics you can measure during a usability test, like error rates and task timings, the feedback you get from a questionnaire like UMUX following a usability test can help you understand your users attitudes. If your product isn’t user-friendly, it’s very unlikely to be adopted by users at the level you need for your business to grow. 

Testing the usability of your prototype will help you catch issues early on while testing the usability of products that have already been shipped will help you make continual improvements. 

Testing New Features

Once you’ve released a new feature, you’re probably eager to know whether or not your users will be able to use it easily. Will they be able to adopt it quickly? Will they find it easy to understand, use and integrate into the flow of how’re they’re already using your product? Since your users are already taking in new information about the new feature, UMUX-Lite is the better option here. You’ll be able to assess usability quickly without overburdening your users with too much info. Just make sure you only target those who are actually experiencing the new feature.

Score Calculation 

UMUX Scoring 

• Odd items are scored as [user score - 1]. Even items are scored as [7 - user score]. 
• Add up these differences and divide the sum by 24 (the highest possible score). 
• Multiply your quotient by 100. 
• Average your results across users.

If you’re using a 5-point Likert scale, you will only have to make a few adjustments to the steps above. Even items should be scored as [5 - user score] and you’ll have to divide the sum by 16 instead of 24. 

UMUX-Lite Scoring [7-point scale] 

• Items are scored by subtracting one from the user response: [user score - 1] 
• Add the two adjusted scores and divide the sum by 12 (the highest possible score). 
• Multiply your quotient by 100. 
• Average your results across users.  

If you’re using a 5-point Likert scale, you only have to make one change to the steps above: divide the sum by 8 instead of 12. 

Interpreting UMUX results 

Because UMUX is still fairly new in comparison to other standardized usability questionnaires like SUS, not much benchmark data exists to help you interpret your score. However, the more UMUX data you track over time, the better understanding you’ll have of what your targets should be with UMUX as well.

If you’re interested in converting your UMUX-Lite score to an SUS score, the creators developed a formula to help you do just that. 

SUS comparison score = 0.65 *((UMUX-Lite Item1 + UMUX-Lite Item2 − 2) ∗ (100/12))+22.9.

This formula, however, only works if you are scoring the UMUX-Lite on a 7-point scale. 

Keep in Mind

Because the UMUX Questionnaire is evaluated on a Likert scale, we recommend keeping potential respondent biases in mind that are associated with Likert scale reporting.  

• Social desirability bias - Most people tend to give answers that they consider as socially acceptable rather than being fully honest. 
• If your results are overwhelmingly positive, dig in further by reaching out to some of your respondents for in-depth user research interviews. 
• Central tendency bias - Most people avoid choosing extreme responses such as 1 “Strongly disagree”  or 7 “Strongly agree” 
• While the UMUX questionnaire was tested and retested to help mitigate this bias, if you are finding that the majority of your responses are neutral, you might consider how the central tendency bias is at work here. 

Pros and Cons of UMUX 

Pros

• The UMUX Questionnaire is an ideal length. Shorter than the System Usability Scale. Longer than the Single Ease Questionnaire. UMUX gives you a greater amount of insights that your users can provide quickly! 
• UMUX evaluates usability based on the official definition and standard established by the ISO.

Cons

• Scoring is somewhat more complex when it comes to UMUX, especially if you’re interested in comparing your results to an SUS score. 
• Because UMUX combines both positive and negative questions, some researchers have found that it creates a two-factor test. However, UMUX-Lite can be used instead, if you find your data showing the same concern. 

Using UMUX with Qualaroo 

Getting started with UMUX or UMUX-Lite is easier than ever with Qualaroo Templates. We’ll walk you through how to get it set up in the steps outlined below. 

Set-up 

Start Running the UMUX Questionnaire with Qualaroo in 3 simple steps! 

1. Select the right channel. 
• Decide where you want to run your survey (desktop, mobile web, prototype, mobile app or link) and select “Choose Template.”
• Don’t worry–you can run this survey on as many channels as your organization needs by creating additional surveys so that you cover the different domains your organization spans.
2. Find your template. 
• Select one of the three “UMUX” templates. 
• You can even type “UMUX” into the search bar to find it.  
3. Create, edit and publish your survey! 
• All questions already been entered for you. 
• The only thing left to do is select the right word choice to describe your system: website/product/software, etc., Make sure to keep it consistent for all of your statements!

Additional Tips 

Getting the most out of the UMUX Questionnaire Template

To get the most out of the UMUX Questionnaire, we recommend taking advantage of Qualaroo’s targeting options. Depending on the area of your company you wish to evaluate or the audience segment you are seeking to survey, targeting options may vary. However, these general recommendations can help you get your Nudge in front of the right users at the right time. 

If you have a more specific use case, feel free to get in touch with your designated customer success manager to discuss further options that you can take advantage of. 

Recommended targeting options

• Where should this survey appear?: 
• Only where active users will see it
• This might be behind the login wall after a user has completed a specific task or after the individual has gone through a customer support experience.
• Type in the URL (simple or advanced) where you expect the experience to naturally conclude.  
• Who should be prompted to take this survey?: 
• Select 100% of all visitors who meet the conditions specified below
• "Came directly to your site (typed in your URL, or bookmark)" +  "Only returning visitors". 
• When should this survey be displayed?: 
• After the log-in wall to visitors who are returning
  • Displaying a Nudge very quickly to first time visitors or users can disrupt their first impression of the user experience.
• You can also schedule a time delay that aligns best with your user behavior. Use Google Analytics as your guide to see what the average length of time people spend on a page is.
• If you have pages set up for the end of a certain path like thank you (for purchasing, downloading, subscribing) screens or success pages designated for a specific experience, you might also select “immediately after the page loads.”
• How often should this survey be displayed?: 
• Select “Continue showing until a visitor provides a response” 

If you need help understanding all the options available within the Targeting Section, please visit this section of our Help Center.

Before and After you begin collecting responses 

Before

• Determine how long you want to run this campaign. Setting an end-date can allow you to analyze data over a fixed interval while also making sure that you’ve allowed enough time for user exposure. 
• If you want to combine objective metrics like error rates and task completion times as part of your usability testing, consider using a session recording tool like fullstory or SessionCam to integrate with Qualaroo. 

After

• Analyze your results to determine which patterns or trends are emerging.
• If need be, follow-up with outliers to gather more descriptive and qualitative feedback on their responses. This can be done by inviting them to more precise usability testing to nail down what parts of the interface were difficult. 
• If you have a central database where you are aggregating user or customer data, Qualaroo’s integrations (i.e. Segment, Tableau, Adobe Analytics and more) can help you funnel responses seamlessly for further analysis in context. 
• If you are on our Premium Plan, you can take advantage of our REST Reporting API to send Qualaroo response data to any endpoint of your choosing. 

Conclusion 

The Usability Metric for User Experience is a great option for companies looking to understand what usability looks like for their products. As one of the youngest usability questionnaires in the field, there isn’t a ton of benchmark data for comparison. However, UMUX and UMUX-Lite were built upon the years of research that went into their predecessors, like SUS. Although scoring might be slightly more complicated with this specific template, your results will ensure that your measuring usability according to its standardized definition and with very little time spent on set-up! 

If you have any additional questions or would like to get in touch with a customer success manager about setting up this template, please submit a request here. We are always looking for ways to ensure that Qualaroo can become more efficient, effective, and satisfying to your needs.