BusinessPolicy & Law

Why Pilot Projects Don’t Work

2 Mins read

Remember, the entire ObamaCare approach to controlling costs and improving quality is based on running pilot projects. As the president has said on more than one occasion, “Let’s find out what works and then go do it.” So I want to make sure you are aware of an impressive post by Megan McArdle before Christmas, explaining why this approach is doomed to fail.

Remember, the entire ObamaCare approach to controlling costs and improving quality is based on running pilot projects. As the president has said on more than one occasion, “Let’s find out what works and then go do it.” So I want to make sure you are aware of an impressive post by Megan McArdle before Christmas, explaining why this approach is doomed to fail. In a word, she says, “promising pilot projects often don’t scale.”

They don’t scale even when you put super smart people with expert credentials in charge of them. They don’t scale even when you make sure to provide ample budget resources. Rolling something out across an existing system is substantially different from even a well-run test, and often, it simply doesn’t translate.

Here are five reasons:

  • Sometimes the “success” of the earlier project was simply a result of random chance, or what researchers call the Hawthorne Effect, named after a study in which workers became more productive no matter what the stimulus — a change from the ordinary, or the mere act of being studied.
  • Sometimes the success was due to what you might call a “hidden parameter,” something that researchers don’t realize is affecting their test.
  • Sometimes the success was due to the high quality, fully committed staff, whose work won’t be duplicated by folks who are just looking for a job and don’t see the pilot project goal as their life’s mission.
  • Sometimes the program becomes unmanageable as it gets larger.
  • Sometimes the results are survivor bias – the subjects who stay in the project are very different from the random population.

On this last point she explains:

This is an especially big problem with studying health care, and the poor. Health care, because compliance rates are quite low (by one estimate I heard, something like 3/4 of the blood pressure medication prescribed is not being taken 9 months in) and the poor, because their lives are chaotic and they tend to move around a lot, so they may have to drop out, or may not be easy to find and re-enroll if they stop coming. In the end, you’ve got a study of unusually compliant and stable people (who may be different in all sorts of ways) and oops! that’s not what the general population looks like.

Related posts
Global HealthcarePolicy & Law

Do HMOs Deserve A Bad Reputation? The Truth Of Managed Healthcare

3 Mins read
Have you ever heard people say that HMO health insurance is no good? It’s no secret that HMOs have a bad reputation,…
Policy & Law

Top 4 Important Things To Understand About Insurance

2 Mins read
When it comes to Medicare, the enrollment process can be a little tricky. That’s why it’s important to understand the relevant health…
Health carePolicy & Law

Elmiron Controversy Highlights Need For Longterm Clinical Drug Testing

4 Mins read
Elmiron has recently been the subject of a number of lawsuits brought about over claims that the drug causes vision problems and may lead…