Our client Pavlok was a MassChallenge-backed startup that produces a wearable that can shock you to change your habits. The clock can wake you up with vibrations and beeps, make you do jumping jacks (that it can motion-sense) to turn it off, and give you an electric shock if you hit snooze too often. All the experiences are customizable, and we were assured that youâre always in control of what happens to you. Â
It had a nifty app, and a founder with big dreams for a comprehensive, research-backed behavior change platform that would put users in charge of their sleep/wake cycle. He also had a tiny team, so we were invited to pour some fuel on this product development cycle.
Discovery: understanding the product, its users, and the ecosystem around it
To better understand the product, its market, and leadershipâs vision, our team held a Design Studio exercise with the Pavlok team.
We learned that though they are currently viewed as a hardware company, they see themselves as a habit change company.
They'd already built out extensive user support and enrichment programs, and have existing forums, tutorials, and videos that cover the psychology and methodology of habit change in general to get the most out of Pavlok.
They wanted us to help get them started on the path of positioning themselves to the user in a way that accurately reflects their product vision. As they saw it, the wearable and the app would work together to create a dynamic sleep-wake experience.
So whatâs the problem?
As the CEO said:
"the advanced features are the most valuable parts of the product."
Users werenât finding or using them.
Heuristic Analysis
Our first look at the existing screen and a quick run-through with Nelson's Heuristics helped explain why. The appâs visual structure overwhelmed userâs ability to parse what was relevant, and what to look at/do first.
A lack of happy-path tends to make people bounce off, at best. At worst, users can experience outright distres, as with that feeling of dread you get with an overflowing inbox, or when there are too many warnings on your analytics dashboard. This app was meant to be used before bed and on first waking, when none of us are at our sharpest. Low cognitive reserve could reasonably be expected to enhance negative emotional effects. Instead of inviting users to dive deeper and make it theirs, they're likely to open this screen, stare at it for a few seconds, and disengage.
Task-oriented User-Studies
User tests confirmed that users found Pavlokâs current sleep homepage very difficult to use.Â
We found some folks who had no exposure to Pavlok, but who were familiar with habit tracking and wellness apps in general, and watched them use the app. We asked asked them to narrate their experience while setting an alarm, and secretly timed them. They struggled. People didnât know where to click or what they should do first. Some of the largest issues were centered around the fact that there is a reminder that itâs time to go to bed (called the âSleep Goalâ) and an alarm clock. People didnât understand that these were separate features, and that setting up the âSleep goalâ was a separate function from setting a clock. None of our testers successfully set up their sleep reminders or alarms in the time allowed.Â
Event-Tracking Review
Most mobile apps log events for quality assurance, and I wanted to take a look to see if we could see any footprints that could give us some insight.
There was an expected pattern of users seeing the sleep screen for the first time and then backing out, but more troublingly, a common pattern of users getting to the sleep screen, then activity going quiet entirely. Users were sending the app to the background and doing something else. While the Pavlok team had been logging and trying to understand behavior around force-quits, they did not have a way to get insight into this idling with digital metrics.
Tracking an absence of anything is difficult. These days, you can set rules like 'if after event A, nothing happens for x duration, generate event marker B' but the tech accessible to a young startup back then didn't support this. Behavioral inference by a human was required. Behavioral inference from data can be an incredibly powerful tool, or an incredibly expensive money sink. It's not practical (or actually useful) to review every user's data. Creating useful event tracking systems for user research and QA can be one of the most valuable contributions a behavioral scientist or experienced product designer can bring to a company. As this project moved forward, I worked with this team to create representative user cohorts, learn how to code and infer behavior in their data, and make the most of the event-tracking tooling they had access to.
Peer-Reviewed Research
One of the most wasted resources in tech is peer-reviewed behavioral science research. I stand on the shoulders of giants whenever I can. Reviewing the research on behavior change and sleep/wake cycles put our mental models on a strong foundation, and helped us ask the best questions as we moved into primary research that generated original data. I was particularly interested in how failing at habit change impacts our sense of self, and how inflicting a shock on yourself might impact that.
My review highlighted that habit change is a deeply emotional process, intrinsically tied to the experience of failure, and implicit beliefs about our own value and worthiness.
Marketing & Domain Research
We wanted to understand how competitors and other experts viewed this product space, so we reviewed how products like FitBit and Android's sleep management app approach things, including their surrounding online communities.
User Interviews
Ethnography and Interviewing is the most powerful and complex skill in a researcher's arsenal.Though I had a decent foundation from others'peer-reviewed work, nothing replaces talking to people. I interviewed non-users about their relationship with the concept of habit change, paying attention to their expressions, stress signs, word choices, and asking them to tell me stories. They spoke very vividly about emotional experiences of powerlessness and frustration. They did not have clear ideas about deliberate habit change as a process and what may help them reach their goals. They were generally unsatisfied with digital solutions, though this sample size would not be enough to gain real insight on trends there.The purpose of these sessions were not to gain ground-breaking new information, but step away from the ultra-processed data I'd been reviewing and ground my choices going forward in real, undistilled human experience, and nurture my own empathy.
Surveys (and necessary validity testing)
Surveys are a decent way to get generalizable data. The Pavlok had a well built-out user group on Facebook, and they wanted us to use that as a user pool.It's what they'd been doing.
I had some concerns about selection bias. Early adopters of new technology tend to be more tech savvy, risk-tolerant, and educated than the average user of established products. This is compounded if you select out of enthusiast groups and online communities. In those cases you've selected for people who are already likely to be predisposed to offering --in fact ARE offering-- the kind of engagement you hope to make more likely with your product design choices. It's an easy way to end up with a rosy picture.
I was concerned about the validity of testing with this sample, so I built in some validity checks.
I made two identical surveys, asking about attitudes, beliefs, and practices around habit change. I sent one out to the Pavlok Facebook user group, and one out into the wilds of several wearable tech, sleep, and habit change web forums.If the two sets of survey results were very different, it would have suggested that user research Pavlok had done previously may not be reliable.
My fears did not come to pass. The results for both surveys were similar. I was able to verify the validity of the pre-existing research we could build on, and collect over 60 detailed survey responses.
We had a lot of data at this point. How do you go about making it useful?
Information Synthesis
Affinity Mapping
Personas
Task analysis and flow design
Understanding the emotional landscape to craft the digital one
Journey Mapping
Mapping the flow itself
Solution Proposal
Prototyping & Testing
Sketches:the earliest stage of prototyping
They help you figure out what you need while maintaining the right incentives.
Testing & Iterating With Paper Prototypes
Paper prototyping helps you iterate quickly without getting attached to a particular idea, and helps testers focus on the right thing. No one is critiquing the font of a paper prototype.
We created multiple iterations of paper prototype and tested them on other technologists at the MassChallenge offices, on students in a technology course, and unsuspecting members of the public that we approached in the street and tempted with gift cards.
Checking In With The Pavlok Team
Hand-off