Building Better Ways to Measure Digital ADHD Treatment

This case study is an overview of our efforts and learnings during my 5 years as the lead designer on the product team at Akili.

The Challenge

How do you know if a video game is actually helping improve attention? This was the core question that kicked off my five-year journey as the lead designer at Akili, makers of the first FDA-approved video game treatment for paediatric ADHD.

When we started, we faced a crucial challenge: while we had clinical proof that our game improved attention, families had no way to understand if the treatment was actually working for their child.

Starting Point: ADHD Insight

Understanding How Families Track ADHD Symptoms

The Akili product team set out to develop a companion app for caregivers that would help them understand if EndeavorRx, a digital prescription therapeutic, was working for their child.

As a first step they investigated how families currently track ADHD symptoms. The thinking was that by studying how they were currently solving this problem we would reveal insights into their needs. Through interviews and home visits, they found parents doing their best with whatever tools they had – notebooks, spreadsheets, and mental notes. They were trying to spot patterns but struggling to make sense of day-to-day changes.

Persona created by Product & Marketing team

“I’ve been using a notebook, but it’s hard to see patterns. Some days are great, others aren’t, and it’s impossible to tease apart all of the various factors.”

— Caregiver

They mapped out what parents were trying to track:

Daily behaviors and mood
Medication effectiveness
School performance
Specific ADHD symptoms
Patterns across different times and situations

A code prototype of the app was built by the team prior to my involvement and it was tested with a group of caregivers.

Our early testing looked promising:

Parents could easily complete the main tasks
They loved seeing patterns emerge
Many asked when they could start using it

Building ADHD Insight

When I was hired my priority was to lead the design efforts and turn the beta version of the app into a production-ready mobile app on Android and iOS platforms, as well as to apply a new brand identity.

I worked with a product manager and 4 engineers to build ADHD Insight, an app for daily symptom tracking.

Discrepancy Between Test Environment & Real Life

Unfortunately, real-world usage told a different story than what was originally uncovered through research. The app worked well in testing, but in real life, parents struggled to keep up with daily tracking. It was like giving someone a really nice food journal – great in theory, but hard to maintain when life got busy.

Branding & Design System

I collaborated with the marketing team to create social media content to promote the app.

I collaborated with the engineering team and built a modular design system to make our web and mobile app development efforts efficient moving forward.

What We Learned

Parents wanted to help but were already overwhelmed
Daily manual tracking was too much work
The data wasn't actionable enough to justify the effort
We needed to focus on automatic data collection where possible
Algorithms for discerning subtle trends within the data could have enhanced user insights

Pivot: EndeavorRx Insight

A Less Hands-On Approach

Instead of asking parents to do more work, what if we could show them data that was already being collected? Since kids were playing our therapeutic game, we had data about:

How often they played
How much effort they were putting in (measured by adherence to the rules of gameplay)

We stripped down our focus to answer two key questions:

Is my child playing enough?
Are they playing correctly?

We knew from user research and feedback that caregivers felt blind as to how the treatment was going for their child and found it cumbersome to closely supervise their 25 minutes of daily play. We chose to focus on revealing these two pieces of information as an MVP approach for a new caregiver companion app; EndeavorRx Insight.

Designing a Data-Driven Platform for Caregivers

Working alongside the product manager, I created design artifacts that facilitated stakeholder alignment, enabled us to validate concepts through user testing, and guided the development of the new app in just 3 months from conception.

The Impact

The app garnered an average of 4.5 out of 5 on a scale of "usefulness" from an average of 16 study participants.

We heard encouraging feedback that supported the direction of the app.

“I would have renewed my prescription if I had this information”
“It would give me freedom so I don’t have to constantly observe”

As of October 2023, Approximately 71% of caregivers in our ecosystem use the app at least weekly and we continue to aim to improve this number.

Interestingly, we also observed that Insight usage by caregivers has a direct impact on the child’s treatment compliance.

Non-Insight users: 20% treatment compliance in week 4
Insight users: 72% treatment compliance in week 4

This may imply that the Insight app helps caregivers stay engaged in encouraging their children to play as prescribed. We cannot definitively say whether this is correlation or causation, however, since these might be highly engaged parents with or without the app.

Ongoing Research Uncovered Areas For Improvement

We also received some valuable constructive feedback that will help us iterate moving forward.

“It would be better if…

…it could measure improvements in my child’s attention
…the data was sent via text or email
…If I could give my child a reward in-game to show my engagement and support”

Measuring Real-World Impact With The Caregiver Assessment

Additional user research & product feedback revealed that while gameplay data was helpful, parents still wanted to know if the treatment was improving real-world behavior.

“The whole purpose is to minimize the symptoms. By tracking the symptoms we are able to see if he is minimizing his symptoms. We can track if he is improving”

— User Test Participant

Working with our clinical team, we integrated the Vanderbilt Assessment – a standard ADHD symptom questionnaire – right into the app.

The trick was making a clinical assessment feel less like paperwork and more like a helpful tool.

We:

Broke questions into manageable chunks
Added progress indicators
Created clear before/after comparisons
Made results easy to share with doctors

This seemingly simple addition had a big impact

Week 3 & 4 user retention went up by ~10% after releasing the Caregiver Assessment indicating that having the assessment helped caregivers stay engaged in their child’s treatment.
60% of caregivers who visit the assessment landing page end up starting an assessment, indicating that the screen was moderately successful at conveying the value of the assessment.
95% of caregivers who began the assessment completed all 18 questions, suggesting the UI design for the questionnaire was easy and intuitive to use, and the survey questions we selected were relatively straightforward to complete.

Room for improvement

Only 4% of caregivers are exporting their data to share with their doctors. Research initiatives are underway to determine why most caregivers aren’t taking advantage of the ability to share data with their healthcare providers.

The Caregiver Assessment was one step towards helping caregivers understand if EndeavorRx is working for their child and is contributing to improved patient outcomes.

Meeting The Needs of A New Target Demographic

Creating The Focus Score for Adults

“Seeing improved focus would be a huge benefit... like a graph or number that shows focus level, which is different than game scores”
“It can give a reinforcement that this is helping you and that it’s okay if you’re struggling because you can see how much progress you’ve made over time... It’s cool to see that you’re improving and sometimes that’s hard with games like this because you can feel kind of defeated when you’re not doing well. Having results can help.” - User research participant

In addition to user feedback, there is a lot of health industry evidence that suggests providing individuals with real-time feedback on their performance can significantly increase their engagement with a task; see Measurement-Based Care. The Focus Score aims to leverage this principle by providing them with a way proactively manage their focus.

“What you can measure you can manage.”

Designing a Meaningful Metric

Design a game-play-derived metric that quantifies a player's attention and focus during gameplay.
Ensure the metric positively correlates with improved attention function in ADHD patients.
Design an intuitive and visually appealing UI to display the metric and related information.
Incorporate gamification elements into the design to improve it’s effectiveness.
Incorporate this metric into a free-trial experience that increases subscriptions and perceived product value.

Ideation & Design: Crafting the Experience

The Focus Score metric was developed by the Cognitive Data Science team at Akili. Game-derived cognitive performance metrics were mapped onto cognitive processes (Focus/Sustained Attention, Distractibility, Impulsivity, etc) to develop the metric.

To design how the data would be presented to the users, I performed a competitive analysis and collected external inspiration from other relevant offerings to help guide our creative collaboration.

Next, we held a cross-functional ideation workshop where all stakeholders could sketch low-fidelity ideas and think visually about how we might convey this information.

A Functional Prototype Validated Our Approach

To validate our direction we created a functional prototype that simulated a 7-day free trial of the treatment with the Focus Score concept included and conducted usability testing sessions to collect feedback on the game mechanics, Focus Score comprehension, and value.

Bringing Focus Score to Life

Informed by the findings from our research, we created high-fidelity mockups and functional specifications for the final list of features that were to be included in the release. We collaborated with developers to integrate the Focus Score into the game's codebase and existing game flows.

In parallel, our data science team conducted thorough testing to ensure the accuracy and reliability of the Focus Score calculation. Across 3 clinical studies (pediatric, adolescent, adult ADHD), we found that improvement in Focus Score is significantly correlated to improved ADHD symptoms, validating that this was a meaningful measurement for our users.

Early Success Indicators and Ongoing Analysis

User feedback indicated high satisfaction with the Focus Score - it was rated as 2nd most important (after clinical validity) on a list of what people liked most about the product.

The Focus Score became a game-changer:

Users who tracked their score played 40% longer
Treatment adherence improved by 25%

Continuous Improvement and Patient Outcomes

By providing players with a meaningful and engaging metric that correlates with improved attention function, the game will motivate players to manage their ADHD symptoms actively. The project demonstrates the potential of gamification and UX design in contributing to digital therapeutic treatment compliance. Continuous monitoring and user feedback will remain essential for ongoing improvements and refinements to ensure the best possible outcomes for patients.

What I Learned

Looking back at this journey, a few key lessons stand out:

Start with the Problem, Not the Solution Our first instinct was to build a tracking app because that's what parents said they wanted. But the real problem wasn't a lack of ability to track symptoms – it was a need to understand if treatment was working. Sometimes you have to look past the requested solution to find the real need.
Automate What You Can Every manual task you ask of users better provide clear value. We succeeded when we focused on showing valuable insights from data we already had, rather than asking users to input more data.
Make Progress Visible People need to see that their efforts are paying off. Whether it's through scores, assessments, or celebrations, showing progress in a meaningful way keeps users engaged.
Balance Clinical Validity with User Experience In healthcare, being scientifically sound isn't enough – you also need to create experiences people want to use. Finding this balance is crucial for creating effective digital health tools.

This journey taught me that good design isn't just about making things look nice or work smoothly. It's about deeply understanding problems and crafting solutions that fit into people's lives while driving meaningful outcomes. Sometimes that means throwing out your first solution and starting over. Other times it means finding ways to make clinical tools feel more human. But always, it means keeping focused on the real impact you're trying to create.