Helping caregivers understand “if it’s working” with the Caregiver Assessment
Overview
EndeavorRx is the world’s first FDA-approved prescription video game developed for pediatric ADHD. EndeavorRx Insight is their companion app for parents that’s meant to give them a window into their child’s treatment experience. After building the MVP, we knew there was a lot of opportunity to add valuable features to the app based on user feedback. One of the first user needs we tackled was the need to better understand if the treatment was working. We developed the Caregiver Assessment, a subjective bi-weekly questionnaire to help them recognize symptom progression over time. Week 3 & 4 compliance went up by ~10% after releasing the feature.
Research & Discovery
We started out by conducting in-depth interviews with caregivers to get their input about which types of data/behavior tracking would be most valuable to include in the app. In the spirit of moving quickly, we developed several low-fidelity concepts for the caregivers to review.
The various solutions ranged from daily symptom tracking, a 3rd party reaction-time test, surfacing in-game data like “current level” or “XP”, a proprietary gameplay-derived attention metric (like the Focus Score), and what we ended up building; the subjective biweekly Caregiver Assessment.
Doing this rapid research provided us with an immense amount of insight into how caregivers are thinking about the treatment, measuring progress, and general comprehension of our concepts.
The 3rd party assessment, Biweekly assessment, and Gameplay-derived performance metrics were all rated equally appealing. We decided to start with the Caregiver Assessment because some of the infrastructure was already built and the data science team needed more time to develop a “Focus Score”-like metric.
All 9 parents we interviewed said the biweekly assessment would be useful and would be interested in completing it to help track symptoms and their improvement over time.
“The whole purpose is to minimize the symptoms. By tracking the symptoms we are able to see if he is minimizing his symptoms. We can track if he is improving” - test participant
“These are the things that lead people to do this in the first place, so measuring how it is going in the real world is helpful… If the attention skills are changing, but the symptoms aren't it would be helpful to find the right clinical path to keep trying to address the issue” - test participant
A few parents requested notifications or emails to remind them to complete the assessments.
With this research to validate our direction, we moved towards developing high-fidelity designs for the feature.
Design & Prototyping
We collaborated with our clinical science team to determine the best clinically validated symptom assessment questionnaire to use in the app. We interviewed doctors to determine which assessments they most commonly use and gauged their interest in including assessments within the caregiver app. They recommended we use the “Vanderbilt Symptom Assessment for parents” as it’s most commonly used in the field and requires only 5-10 minutes to complete.
Based on our research, we wanted to ensure we were meeting several design goals:
Caregivers should understand what the survey is and why they should take it
It should be as easy/effortless to complete as possible and take no more than 5-10 minutes
They should be sent reminders to complete it on a biweekly cadence
They should be able to understand the results of their ongoing assessments at a glance
They should be able to share the results with their healthcare provider
Learnings & Outcomes
Week 3 & 4 user retention went up by ~10% after releasing the Caregiver Assessment indicating that having the assessment helped caregivers stay engaged in their child’s treatment.
60% of caregivers who visit the assessment landing page end up starting an assessment, indicating that the screen was moderately successful at conveying the value of the assessment. As a next step, we are performing an A/B test for this screen to improve its conversion rate.
95% of caregivers who began the assessment completed all 18 questions, suggesting the UI design for the questionnaire was easy and intuitive to use, and the survey questions we selected were relatively straightforward to complete.
Only 4% of caregivers are exporting their data to share with their doctors. Research initiatives are underway to determine why most caregivers aren’t taking advantage of the ability to share data with their healthcare providers.
Conclusion
Though we still have a lot of work to do, our data suggest that the Caregiver Assessment was one step towards helping caregivers understand if EndeavorRx is working for their child and is contributing to improved patient outcomes.
We will continue to evolve the feature based on ongoing data collection and user feedback.
Next, we are working to incorporate an objective measure of attention since that was also a top-rated concept from our discovery research.