What’s wrong with our home monitoring and automation platform? It tested well in the lab.
A national telecommunications company was launching a home monitoring and control service with new hardware and software components. Sales were sluggish, those who purchased the service weren’t activating it or were cancelling their subscription after the initial promotion. They had conducted some basic user testing in their controlled lab environments and weren’t seeing any major issues.
Activities
- Ethnographic Studies (Consumers)
- Heuristic Evaluation
- Prototype Design
- Panel Curation for Surveys and Testing
- Prototype Testing
- Case Study
- Telecom
- Engagement Type
- B2C
Approach
I recreated the home monitoring system in a ‘fake’ living room to conduct a heuristic evaluation of the entire platform – from ordering and unboxing to daily home use.
I also recruited 10+ users to trial the software and hardware over the course of a month, observing their installations and use of the project.
I also created a panel of over 100 users to test future versions of the prototype I created with designers, which allowed us to quickly get feedback before writing a single line of code.
Data Synthesis
Users behave differently in canned environments such as a lab or ‘fake’ living room. Observing users in their own environments can surface issues that might not appear in controlled areas. This is particularly effective when dealing with physical products. I visited many product trial participants in their homes. I scheduled a timeframe in the evening or weekend for a given user, provided them with the equipment and then simply said, “Install and activate this.” I informed them I could not help them unless they said “I’d call customer support now.”
Specific learnings that were barriers to success:
1. The shipped HDMI cable is too short for most home entertainment centers
2. The television interface is not useable on screens smaller than 36″
3. The printed instructions do not match the online versions
4. The thermostat is not compatible with the user’s HVAC system
5. Batteries are not included
6. Users don’t remember their WiFi passwords or where to get them
7. Instruction manuals are ignored – users don’t read
8. Installation flows vary by user and are often interrupted
9. The interface does not scale for a given device
10. Customer support is unaware of new features and functions NONE of these issues would have been identified in a lab, on a survey or using web analytics.”
Output
In addition to prototyping, I created a series of 10 second installation videos that showed exactly how to pair devices and activate them. I revised the installation guides to short, 1-page visual diagrams along with QR codes to scan for more detailed information. I identified several hardware pieces that were ultimately incompatible with the overall system for a majority of users’ home environments and found replacement devices from other vendors that worked better.
Installations and activations rose and calls to customer service were greatly reduced. Ultimately, the product still failed as the initial product offering was so poorly received. If they would have conducted ethnographic research prior to building out the core system, perhaps the product would have been more successful.
