When I worked at Microsoft, we managed by the scorecard. The scorecard was meant to provide key indicators of the business health. If something wasn’t on the scorecard, it didn’t get focus from the worldwide sales and marketing groups, and if a product or initiative didn’t get this focus it would die. The scorecard had tremendous power and was a rallying and focal point for a sometimes unwieldy global organization. So powerful was the scorecard that if any errors were made in how something was tracked, it could drive exactly the wrong behavior.
When I worked at Microsoft, we managed by the scorecard. The scorecard was meant to provide key indicators of the business health. If something wasn’t on the scorecard, it didn’t get focus from the worldwide sales and marketing groups, and if a product or initiative didn’t get this focus it would die. The scorecard had tremendous power and was a rallying and focal point for a sometimes unwieldy global organization. So powerful was the scorecard that if any errors were made in how something was tracked, it could drive exactly the wrong behavior.
One year, a metric was introduced to measure sales of a new product, in relation to an existing product. The thought was that the new product was a good “upsell” from the existing product so tracking one in relation to the other was a logical measurement. The intention of the metric was to show the new product growing as it “attached” to the existing product. The metric was calculated as:
The sales teams behaved rationally and stopped selling the existing product, because if they sold the existing product, they had to sell even more of the new product to meet their target since the denominator of the equation kept increasing. They met their targets and got their bonuses, but their behavior was exactly the opposite of what the product teams and the company wanted which was for both businesses to grow or at least for the existing product to stay steady while the new one grew.
Last week, ProPublica caused a flurry by releasing a report of complication data for US surgeons. Using their database you can look up any surgeon and find how their patients fared on average for complications after surgery.
As with any measure, it is fraught with controversy about both the accuracy of the data or whether we are measuring the right things. On the surface complication data seems like it’s a good way to track surgeons, and it is if the complications are caused by surgeon error. The problem is that complications are caused by lots of things including patient behavior (for example not caring for a wound properly or taking too many narcotics and falling down after surgery) or by the patient situation, for example, age or co-morbidities. Looking at complication data alone, as Dr. Jennifer Gunter points out eloquently in her blog post, does not give the whole picture. Dr. Gunter’s mother had two surgeries, one that would be recorded as “no complications” and one full of complications. From the raw data, the first surgery looks like a success with a 7-day hospital stay, and the 2nd a failure with a 90-day hospital stay and many complications. (Note that the 2nd surgery could be counted as a “readmission” which would be counted against the hospital.) Regardless, in this situation data alone does not tell the whole story.
In addition to not telling the whole story, looking at complication data alone can drive the wrong behavior, which is surgeons only taking on the “easy” cases, those who are younger, in perfect health, and have no other diseases, for example diabetes. There are many things that patients can do before surgery to ensure successful outcomes like quitting smoking or losing weight, there are things they can’t do, like get rid of a chronic disease or suddenly shed 10 years. Judging surgeons on only complications can encourage them to “cherry-pick” patients so that they have low complications and high scores. In turn these surgeons will be sought out by the “best” patients, and we could end up with a bifurcated system where the “worst” surgeons (looking only at complications) operate on the hardest cases.
There’s a saying that you can’t manage what you can’t measure. It’s important as well to consider what you are measuring, the behavior that you intend to drive, and the long-term implications of it . Healthcare is making small steps to become more data and outcome-driven and we need to encourage and commend that. At the same time, let’s make sure we are looking at the right metrics.
