Validation and Evaluation

Once you begin design and development, you’ll need to continue to conduct research and set up a cadence for regular testing to ensure your product or service is still meeting the needs of its users. Want to add a new feature? Let’s test it in the context of the current system to make sure there is no conflict or doesn’t cannibalize other features and functions. Have a list of new features you want to add but don’t know which are vital for MVP? Perform a Kano Study to ensure the features actually matter to users.

Activities
  • Kano Studies
  • Remote or In-Person Validations
  • Alpha/Beta Trials
  • Surveys
  • User Testing Cadence

Kano Studies

A Kano study is a technique used in product development and customer satisfaction research to understand the relationship between customer needs and product features and prioritize a set of features based on those needs.

Different product features can have different impacts on customer satisfaction, and that not all customer needs are equal. The study involves surveying customers about their preferences for different product features, and categorizing those features into three categories:

  1. Must-haves or basic requirements: These are features that customers expect to be present in the product, and their absence would result in dissatisfaction.
  2. Performance features: These are features that customers consider important and their presence will increase satisfaction.
  3. Delighters or excitement factors: These are features that customers may not expect to be present, but their presence would result in high levels of satisfaction and delight.
A summary slide from one Kano question

Validations (remote or in-person)

A formal usability study is typically conducted after an initial heuristic evaluation is complete and changes or revisions are made to the web site or prototype based on expert recommendations. If the study is to be conducted for an existing web site or application, the heuristic evaluation can be done in parallel, without changes or modifications to the site/app.

Requirements and Deliverables for conducting a formal usability study:

  1. Test plan document
  2. Recruiting contract
  3. Recruiting screener
  4. Scheduling and compensation
  5. Physical set-up
  6. Participant wrangling
  7. Pre and post-test surveys
  8. Moderating guidelines
  9. Data-logging the session
  10. Reporting results

Can be done manually using zoom or thru a third-party provider like usertesting.com or userinterviews.com

Alpha/Beta Trials

If time and budget allows, we’ll conduct an alpha software trial with a small group of users testing the product in a to identify and fix any issues before it is released to a larger audience. The primary goal of an alpha/beta software trial is to ensure that the software product functions correctly, meets the requirements of the users, and to identify and fix any technical or usability issues before it is released to a wider audience, thereby reducing the risk of major issues and negative impact on reputation and user satisfaction.

A beta trial unboxing

Surveys

Surveys are a quick way to get feedback about interfaces, naming conventions and collect demographic information. While the licenses are quite expensive, I prefer to use Qualtrics for most online data collection. It is an extremely versatile piece of software. The development team is very responsive to the needs of their clients and they even added new features based on my team’s recommendations while working at Verizon.

Some things you can do with online user surveys in addition to traditional data collection:

  • Pre and Post-test questionnaires
  • Card Sorting exercises
  • Heat Map studies
  • Recruit users for product trials
  • Product trials feedback
  • Interface comparisons
  • Feature ranking