Why Pilot Projects Don’t Work

January 4, 2012
63 Views

Remember, the entire ObamaCare approach to controlling costs and improving quality is based on running pilot projects. As the president has said on more than one occasion, “Let’s find out what works and then go do it.” So I want to make sure you are aware of an impressive post by Megan McArdle before Christmas, explaining why this approach is doomed to fail.

Remember, the entire ObamaCare approach to controlling costs and improving quality is based on running pilot projects. As the president has said on more than one occasion, “Let’s find out what works and then go do it.” So I want to make sure you are aware of an impressive post by Megan McArdle before Christmas, explaining why this approach is doomed to fail. In a word, she says, “promising pilot projects often don’t scale.”

They don’t scale even when you put super smart people with expert credentials in charge of them. They don’t scale even when you make sure to provide ample budget resources. Rolling something out across an existing system is substantially different from even a well-run test, and often, it simply doesn’t translate.

Here are five reasons:

  • Sometimes the “success” of the earlier project was simply a result of random chance, or what researchers call the Hawthorne Effect, named after a study in which workers became more productive no matter what the stimulus — a change from the ordinary, or the mere act of being studied.
  • Sometimes the success was due to what you might call a “hidden parameter,” something that researchers don’t realize is affecting their test.
  • Sometimes the success was due to the high quality, fully committed staff, whose work won’t be duplicated by folks who are just looking for a job and don’t see the pilot project goal as their life’s mission.
  • Sometimes the program becomes unmanageable as it gets larger.
  • Sometimes the results are survivor bias – the subjects who stay in the project are very different from the random population.

On this last point she explains:

This is an especially big problem with studying health care, and the poor. Health care, because compliance rates are quite low (by one estimate I heard, something like 3/4 of the blood pressure medication prescribed is not being taken 9 months in) and the poor, because their lives are chaotic and they tend to move around a lot, so they may have to drop out, or may not be easy to find and re-enroll if they stop coming. In the end, you’ve got a study of unusually compliant and stable people (who may be different in all sorts of ways) and oops! that’s not what the general population looks like.

You may be interested

Care On The Road: How Telemedicine Can Reach Truck Drivers
Mobile Health
12 views
Mobile Health
12 views

Care On The Road: How Telemedicine Can Reach Truck Drivers

Larry Alton - August 21, 2017

Telemedicine is considered a powerful tool for individuals living in rural areas, far from adequate services or in need of…

Where Is The Balance? Pushing Back Against Consumer Health Tech
eHealth
3 views
eHealth
3 views

Where Is The Balance? Pushing Back Against Consumer Health Tech

Larry Alton - August 18, 2017

When Republican Congressman Jason Chaffetz glibly remarked that Americans struggling to afford insurance should choose between that and their smartphones,…

What to Look for in Patient Solutions Software
eHealth
365 views
eHealth
365 views

What to Look for in Patient Solutions Software

Robert Cordray - August 17, 2017

The medical sector is one area where technology has had a significant impact, largely by providing tools that simplify many…