By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
Health Works CollectiveHealth Works CollectiveHealth Works Collective
  • Health
    • Mental Health
    Health
    Healthcare organizations are operating on slimmer profit margins than ever. One report in August showed that they are even lower than the beginning of the…
    Show More
    Top News
    bowl of vegetable salad
    Raw Foods: benefits and harms
    November 9, 2021
    pros and cons of the keto diet
    Read This Before You Follow the Keto Diet
    May 18, 2022
    spinal cord injuries
    4 Potential Causes of Spinal Cord Injuries (and How to Seek Compensation)
    May 25, 2022
    Latest News
    7 Most Common Healthcare Accreditation Programs: Which Should You Use?
    August 20, 2025
    Hospital Pest Control and the Fight Against Superbugs
    August 20, 2025
    Hygiene Beyond The Clinic: Attention To Overlooked Non-Clinical Spaces
    August 13, 2025
    5 Steps to a Promising Career as a Healthcare Administrator
    August 3, 2025
  • Policy and Law
    • Global Healthcare
    • Medical Ethics
    Policy and Law
    Get the latest updates about Insurance policies and Laws in the Healthcare industry for different geographical locations.
    Show More
    Top News
    TBI: Some Surprising Statistics
    February 9, 2016
    Your Keys to Safer, Even More Secure Healthcare Cloud Services
    January 13, 2015
    4 Career Options in Healthcare Industry that Combine Big Data & Healthcare
    February 5, 2021
    Latest News
    How Social Security Disability Shapes Access to Care and Everyday Health
    August 22, 2025
    How a DUI Lawyer Can Help When Your Future Health Feels Uncertain
    August 22, 2025
    How One Fall Can Lead to a Long Road of Medical Complications
    August 22, 2025
    How IT and Marketing Teams Can Collaborate to Protect Patient Trust
    July 17, 2025
  • Medical Innovations
  • News
  • Wellness
  • Tech
Search
© 2023 HealthWorks Collective. All Rights Reserved.
Reading: Challenges Evaluating mHealth’s Success
Share
Notification Show More
Font ResizerAa
Health Works CollectiveHealth Works Collective
Font ResizerAa
Search
Follow US
  • About
  • Contact
  • Privacy
© 2023 HealthWorks Collective. All Rights Reserved.
Health Works Collective > eHealth > Mobile Health > Challenges Evaluating mHealth’s Success
BusinessMobile HealthTechnology

Challenges Evaluating mHealth’s Success

JosephKvedar
JosephKvedar
Share
7 Min Read
SHARE

I saw another exciting news story on a mobile health intervention the other day.  I honestly don’t remember the company or product, but what stuck with me was the declaration of success based on 10 patients using the product for three months.  Success was touted in terms of cost reduction and resource utilization reduction in a before/after analysis. This inspired me to collect some thoughts on some of the challenges around evaluating success in mHealth.

I saw another exciting news story on a mobile health intervention the other day.  I honestly don’t remember the company or product, but what stuck with me was the declaration of success based on 10 patients using the product for three months.  Success was touted in terms of cost reduction and resource utilization reduction in a before/after analysis. This inspired me to collect some thoughts on some of the challenges around evaluating success in mHealth.

Challenges evaluating mHealth success

mHealth represents the collision of two interesting worlds — mobile, which changes on what seems to be a daily basis, and health care, which changes infrequently, only after significant deliberation and usually much empirical analysis.  In the tech (mobile) world, companies are talking about creating a minimally viable product (MVP), getting it out in the market, assessing adoption through metrics such as downloads and customer feedback, and iterating accordingly.  This would seem to make sense in the consumer world where the goal is to sell a game, an information app or productivity app.  If people use it and are willing to pay, that proves its utility, right?

More Read

Ablation in Ischemic Stroke
Smart Phone Device Takes EKGs in 30 seconds
5 Fun Healthcare Tools to Help Build Patient Relationships
Healthcare Providers Should Be the Driving Force of Mobile Health
Why Are Hospitals Still Using Pagers?

There is something to this line of thinking. Empiric market success is in some ways the ultimate success, at least for those who want to make a big difference in how humanity benefits from technology.

But does this work in health care?  I’m not so sure.  As clinicians, we’re trained to turn our noses up at this sort of measure of success.  But maybe we’re the ones who are wrong.  Let me use the 10-patients-for-three-months example to illustrate some issues.

  1. Selection bias. Virtually all pilots and trials of any sort suffer from this to some extent.  These days, it seems that patient/consumer engagement is the holy grail and we all must realize that people who show up to enroll in any sort of study are already engaged to an extent.  What about the people who are great candidates for an intervention (conventional wisdom says the disengaged are sicker and more costly) but are too unmotivated even to show up to enroll? Does anyone know how to handle this one? 

    Activity_tracking_mHealth cHealthBlog_selection bias

  1. Regression to the mean. This is a pesky and annoying one — and a favorite of folks trained in public health — but unfortunately it is a real phenomenon.  This is the stake in the heart of virtually all before/after studies.  If you follow a group of people, particularly sick ones, a certain percentage of them will get better over time no matter what you do.  The more sick the starting sample, the more dramatic the effect.  This is why some sort of comparison group is so helpful and why before/after studies are weak.
  1. Small sample size bias. This one can go either way, meaning you can exaggerate an effect or miss one.  If you want to run a proper study, find someone who has training in clinical trial design to estimate the size of the effect of your intervention, and thus the size of the sample you need, to show its efficacy.  Lots of technical jargon here (power calculations, type I error, type II error, etc.), enough to make your head spin. But bottom line, you can’t really say much about the generalizability of data based on 10 patients.
  1. Novelty effect. I made that up, and there is probably a more acceptable scientific term for it. But what I’m referring to is, when you take that same group of people that was motivated enough to enroll in a study and apply an intervention to them, the newness will drive adoption for a while.  We see this all of the time in our studies at the Center for Connected Health.  The novelty always wears off over time.  In fact, I’d say the state-of-the-art in understanding the impact of connected health is one of cautious optimism because we haven’t yet done long term studies to show if our interventions have lasting effects over time.  There is room for argument here, I guess, but three months is awfully short.

    mhealth success

Why is health care tech different than finding the MVP in the rapidly-changing, market-responsive world of mobile tech?  One reason may be that we’re dealing with health and sickness which are qualitatively different than sending a friend the latest snapshot from vacation. It is cliché to say it, but lives are at stake. So we’re more careful and more demanding of evidence. Is this holding us up from the changes that need to occur in our broken health care non-system?  Possibly.

It is true that a well designed trial with proper sample size is expensive and takes time.  Technologies change faster than we can evaluate them.

One thing we’ve done at CCH is design studies that use a large matched data set from our electronic record as a comparator.  This speeds things up a bit, eliminating the need to enroll, randomize and follow a control group.  Results are acceptable to all but the most extreme purists.

What ideas do you have on this dilemma?

Share This Article
Facebook Copy Link Print
Share

Stay Connected

1.5kFollowersLike
4.5kFollowersFollow
2.8kFollowersPin
136kSubscribersSubscribe

Latest News

engineer fitting prosthetic arm
How Social Security Disability Shapes Access to Care and Everyday Health
Health care
August 20, 2025
a woman explaining the document
How a DUI Lawyer Can Help When Your Future Health Feels Uncertain
Public Health
August 20, 2025
physiotherapist at work
How One Fall Can Lead to a Long Road of Medical Complications
Health care
August 20, 2025
Common Healthcare Accreditation Programs
7 Most Common Healthcare Accreditation Programs: Which Should You Use?
Health News
August 20, 2025

You Might also Like

Patient Record on Parking
Mobile HealthRemote DiagnosticsTechnology

The Case for Patient Video in Doctors Visits: Take a Selfie and Call Me In the Morning

August 28, 2015
intranet healthcare
Technology

How Intranet Software Enhances Efficiency in Healthcare

February 15, 2025
patient-centered care
Hospital Administration

The Myth of Patient-Centered Care

November 15, 2013

Aetna Acquires mHealth StartUp, iTriage: mHealth is Here to Stay

December 20, 2011
Subscribe
Subscribe to our newsletter to get our newest articles instantly!
Follow US
© 2008-2025 HealthWorks Collective. All Rights Reserved.
  • About
  • Contact
  • Privacy
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?