Understanding Your Best Customers #3: User Testing


Validating that an idea is worth pursuing is crucial. Including the user’s consideration in the decision-making process helps make sure that you’re building the right product for the right people. Instead, commonly there is only stakeholder feedback on a prototype. It occurs for 2 reasons: Perceived additional costs of user testing, and a concern for scope creep.

Perceived ROI of user testing

UX designers know all too well what it’s like to watch a user struggle and become frustrated with a prototype. Unfortunately, most executives, engineers, and project managers have never witnessed a user test, and need to be convinced of the importance of beta testing another way. Have you ever given a gift to someone who turned out not to like it?

Data supports ROI increase

product usability is intrinsically tied to customer loyalty and purchasing behaviour – when 10% of a redesign budget is allocated for user testing, an average increase is seen of +135% in usability. Decreased frustration means users can easily find what they want on their own (less customer support costs, less bounces, etc.). Identifying and addressing pain points help to highlight the biggest business opportunities

Avoiding Scope creep

“Due to tight deadlines, there is no time for user testing.”
Time, cost and scope are the name of the game for Project Managers and any adjustments in one area will shift the others. Not all user testing methods are time-consuming or costly.

Many types of user tests
• Attitudinal: What people say
• Behavioural: What people do
• Qualitative: Direct observation
• Quantitative: Indirect measurement
Each type has a set number of participants. Each of these areas falls into Formative or Summative.  “When the cook tastes the soup, that’s formative. When the guests taste the soup, that’s summative.” – Robert E. Stake, Professor Emeritus of Education at the University of Illinois.

Qualitative Behavioural User Testing

These user tests notice what a small number of participants do in various situations, in their normal routine, or within a prototype. These tests need the fewest number of participants and can be easier to pitch. Generally, 5 users will reveal 85% of the issues needing solutions. Ideally, you would prototype solutions for those issues, and retest with 5 new people, two more times. So 3 rounds of 5 participants, with a total of 15 people. You would do this once you’re about halfway through your project, while things are still easy to change. It’s much less expensive and disastrous than waiting until the product has been fully developed and launched to discover usability problems. Usually people won’t come back if their first impression was negative.

Qualitative Attitudinal User Testing

UX researchers investigate how small groups of participants feel about a product, architecture, or design. A few more people are needed, but still not many.

Card Sorting

Depending on the complexity of information architecture (how menus are laid out for logical navigation, intranet structure or portals), around 15-30 participants are ideal. Card sorting is often performed in the middle of a project after assessing the information architecture. It involves lying cards out labelled with product names for example, and having people organize them into categories.

Focus Groups

3-6 focus groups of 8-10 people can identify 90% of the themes within a dataset. These should happen during the strategy phase of a project, and many types exist, from participatory design exercises to open discussions.

Qualitative Attitudinal Example

The UX company Etre took on a project to improve the Eurostar information architecture. The website included 11,000 web pages and PDFs, of which 80 pages were core to the architecture. Etre turned each fundamental page into a card for a sorting exercise. They had 20 participants per user group, with 9 groups total. So 180 participants sorting 80 cards each resulted in 14, 400 placements to plot, which they used an automatic software tool to process. Making complex information easy to navigate by especially diverse audiences of various languages, is crucial to the success and with only 2.2% increase in website traffic, they had 24% more revenue.

Quantitative Behavioural User Testing

At least 20 participants are required, because with this type UX Researchers looking for patterns in the data, so larger datasets produce more accuracy.


At least 40 users are required per page, for heatmaps using eye tracking exercises. Eye tracking is typically performed after a new design is executed.

Quantitative Studies

At least 20 participants are recommended. Like usability testing, these tests are to see user behaviour when completing a set of tasks. The difference is that users are observed remotely instead of being one-on-one with the UX designer. They are often performed just before the product or an interactive prototype released, or is already hosted online. These can take more time and be more expensive to conduct, but depending on the scale of reach for those changes, it can ensure a higher degree of success and catch valuable details.

Quantitative Attitudinal User Testing

The test requiring most participants is a quantitative attitudinal exercise. The goal is to understand what a large group of individuals think about a product, and then compiling those into meaningful charts and graphs.


Survey participants required by a project is calculated depending on the total number of users currently on that platform. Optimal results have a 5% margin of error and at least 95% confidence level. A small database of 100 users needs 80 survey participants, while 100,000 would require 383. Surveys occur the beginning and end of a project.

The user experience makes or breaks a product. You have to be certain that you are building the right product for the right people, and testing with the right people. Implementing some of the testing methods to gather qualitative and quantitative data to prove what works and what doesn’t should be a top priority.


No Comments

Sorry, the comment form is closed at this time.


Don’t hesitate to get in contact with us and make your digital aspirations a reality. Book a consultation with one of our digital experts and start down the road to growing your business now.