The promise and challenge of comparative effectiveness research

Anupam Jena, Jonathan Skinner, Amitabh Chandra

19 June 2011



Healthcare continues to account for a sizeable and growing share of national spending in the US and other industrialised countries (Breyer 2011). In the US, ongoing concerns about inefficient healthcare spending – spending on medicines, procedures, or services that yield few if any measurable health benefits – have prompted efforts to ensure that new spending is met with a tangible gain in patient benefits.

As we argue in a recent review article (Chandra et al. 2011), markets solve such problems reasonably well for non-health products and services, but healthcare is different. Government is a large purchaser and care is consumed locally thereby reducing the scope for competition. Most importantly, for nearly all diseases, the landscape of potential medical treatments is complex and the benefit of one treatment over another (either in reducing mortality or alleviating symptoms) is often unknown to both patients and physicians. In all countries, insurance complicates matters by insulating patients from the true costs of medical therapies.

It is in this context that researchers have argued that 20% to 30% of US healthcare spending could be eliminated without adversely affecting the health of Americans (Skinner et al. 2005, Buntin and Cutler 2009). In discussions of US health reform, comparative effectiveness research (CER) has been advocated as a strategy to understand better what works in healthcare and as a means to reduce wasteful spending by eliminating spending on treatments that are expensive but ineffective.

Put simply, CER compares the efficacy of two or more tests, treatments, or healthcare delivery methods without including costs. While not including costs in an efficiency analysis might seem nonsensical to economists, especially when much of US healthcare reform is focused on limiting cost growth (Garber and Sox 2010), we believe CER may still hold promise, but with several challenges.

Knowledge is lacking

Medical treatment options are complex and little or no information on comparative effectiveness exists for many treatments. For example, new and expensive biologic treatments for cancer and inflammatory diseases such as rheumatoid arthritis are developed yearly, yet little is known about how some of these treatments compare to conventional approaches. Put simply, the dearth of comparative efficacy data leaves physicians and policymakers in the dark about how treatments compare to one another. Simply knowing this information will improve productive efficiency by eliminating treatments that are ineffective no matter the price.

Comparative effectiveness could reduce healthcare costs

By demonstrating that expensive medical therapies are equally or less effective than cheaper alternatives, comparative effectiveness could certainly lower costs by eliminating the use of costly treatments. For example, as we highlight in Chandra et al. (2011), in a recent randomised controlled trial of lung cancer patients, half were randomised to early palliative care – a treatment by physicians and social workers who are trained in the management of symptoms at the end of life – and half were not. Both groups chose to undergo chemotherapy equally often, but the group randomised to early palliative care experienced better quality of life, better survival, all at lower cost (Temel et al. 2010). Similarly, amid the growing use of minimally-invasive arthroscopic knee surgery for osteoarthritis of the knee, one study demonstrates that “placebo surgery” – skin incisions and simulated (but not actual) surgery – was no more effective than actual arthroscopic knee surgery. Use of the procedure declined in the years following the study (Hawker et al. 2008).

Comparative effectiveness ≠ cost effectiveness

While maximising value in US healthcare would clearly require us to also consider the costs of treatments (i.e. cost effectiveness analysis), political resistance to using prices has proved formidable. Comparative effectiveness, by ignoring prices and focusing only on effectiveness, may overcome this resistance and at the very least inform policymakers, physicians, and patients about what treatments work and what do not.

While comparative effectiveness may lead to productive efficiency, it won’t lead to allocative efficiency. To illustrate this, Figure 1 displays the association between health spending on the horizontal axis and health on the vertical axis. Greater spending will generally yield better health up to a point (A), after which spending may lead to harm (e.g. overuse of procedures with no measurable benefit but procedure-associated risks to patients). Point B is the greatest level of health a country could hope to attain and is the point where productive efficiency is maximised. But at what cost? While point B may be productively efficient, the foregone consumption required to achieve it may be so high that it is not allocatively efficient. In healthcare, policymakers often invoke $100,000 per year of life as the value of consumption worth foregoing for an additional year of life. Economists would prefer point (B) to point (A) since the foregone non-health consumption required to get from point (B) to point (A) would exceed the value of improved health.

Figure 1. Productive and allocative efficiency in healthcare

Heterogeneity in patient benefit and provider skill

Comparative effectiveness research may demonstrate that a treatment is better on average, even though what is best for an individual patient (or group of patients) may vary. If the results of CER are used to make unilateral coverage decisions in which only those treatments demonstrated to be superior on average are covered, some patients may be worse off (Basu et al. 2011). The effectiveness of a treatment may also depend on provider skill, so that the results of CER may not generalise to lower-volume or less-skilled providers who are likely to provide these treatments in practice. For example, Wennberg et al. (1998) demonstrate large differences in outcomes among patients undergoing carotid endarterectomy – a surgery which removes plaque from the arteries that supply blood to the brain – in original clinical trial hospitals, non-trial hospitals with large volumes, and non-trial hospitals with low volumes.

How much will CER cost?

Clinical trials are notoriously expensive – e.g. pre-approval clinical testing for a single drug being evaluated by the US Food and Drug Administration costs roughly $86 million (DiMasi et al. 2003). Identifying comparative effectiveness trials where the value of information will justify the potentially high cost will be important moving forward.

Will CER change practice?

Publishing new information doesn’t necessarily mean it will change physician behaviour. For example, despite evidence that percutaneous coronary intervention – a procedure in which the arteries of the heart are ballooned open to increase blood flow to the heart – is no more effective than medicine alone for certain heart conditions, it continues to be heavily used by cardiologists (Borden et al. 2007). Similarly, despite published US national guidelines on breast and colon cancer screening, between 30% and 40% of Americans eligible for screening are not screened.

Future directions: Comparative effectiveness of healthcare delivery systems

Despite its limitations, we are optimistic that comparative effectiveness research will improve healthcare, potentially at lower cost. An underappreciated avenue of CER and a source of potential health benefits and cost-savings, however, is the comparative effectiveness of healthcare delivery systems. As little as we know about what treatments work best relative to others, we know equally little about how to efficiently deliver healthcare. Leading US medical centres vary by nearly two-fold in risk-adjusted costs of treating heart attack patients, mainly due to how often patients are seen after hospitalisation, how frequently they are referred to specialists, and how much diagnostic testing is performed (Fisher et al. 2004). In our view, studying how care is delivered in addition to what treatments work better than others will be crucial to harnessing value from comparative effectiveness research.



Basu, Anirban, Jena Anupam, and Tomas Philipson (2011), “The Impact of Comparative Effectiveness Research on Health and Health care spending”, forthcoming in Journal of Health Economics

The COURAGE Trial Research Group (2007), “Optimal Medical Therapy with or without PCI for Stable Coronary Disease”, New England Journal of Medicine, 356(15):1503-1516.

Buntin, Melinda Beeuwkes, and David M Cutler (2009), “The Two Trillion Dollar Solution: Saving Money by Modernizing the Health Care System”, Centre for American Progress.

Chandra, Amitabh, Anupam B Jena, and Jonathan Skinner (2011), “The Pragmatists Guide to Comparative Effectiveness Research”, Journal of Economic Perspectives,25(2):27-46.

DiMasi, Joseph A, Ronald W Hansen, and Henry G Grabowski (2003), “The Price of Innovation: New Estimates of Drug Development Costs”, Journal of Health Economics,22(2):151-185.

Fisher, Elliott, Dan Gottlieb, and David Wennberg (2004), “Implications of Regional Differences in Spending”, Annals of Internal Medicine, 140(2):148-149.

Garber, Alan M, and Harold C Sox (2010), “The Role of Costs in Comparative Effectiveness Research”, Health Affairs, 29(10):1806-1807.

Hawker, Gillian, Jun Guan, Andy Judge, and Paul Dieppe (2008), “Knee Arthroscopy in England and Ontario: Patterns of Use, Changes over Time, and Relationship to Total Knee Replacement”, Journal of Bone and Joint Surgery, 90A(11):2337-2345.

Temel, Jennifer S et al. (2010), “Early Palliative Care for Patients with Metastatic Non–Small-Cell Lung Cancer”, New England Journal of Medicine, 393(8):733-742.

Shah, Nilay D, Victor M Montori, Harlan M Krumholz, Karen Tu, Caleb Alexander, and Cynthia Jackevicius (2010), “Responding to an FDA Warning--Geographic Variation in the Use of Rosiglitazone”, New England Journal of Medicine, 363(22): 2081-2084.

Skinner, Jonathan S, Elliott S Fisher, and John Wennberg (2005). “The Efficiency of Medicare”, in David A Wise (ed.), Analyses in the Economics of Aging, 129-160, University of Chicago Press.

Wennberg, David E, FL Lucas, John D Birkmeyer, Carl E Bredenberg, and Elliott S Fisher (1998), “Variations in Carotid Endarterectomy Mortality in the Medicare Population: Trial Hospitals, Volume, and Patient Characteristics”, JAMA, 279(16): 1278-1281.

Breyer, Friedrich, Joan Costa-i-Font and Stefan Felder (2011), “Does ageing really affect health expenditures? If so, why?”,, 14 May.




Topics:  Health economics

Tags:  US, healthcare, efficiency, comparative effectiveness research

Massachusetts General Hospital and Harvard Medical School

Professor of Economics, Department of Economics, Dartmouth College

Amitabh Chandra

Professor of Public Policy, Harvard Kennedy School of Government