Canadians are missing out on the health and wider societal benefits of new healthcare technologies because of our country’s overly simplistic drug evaluation process. In adopting a narrow health system perspective and in focussing its assessment of value exclusively on a single parameter known as the cost per quality adjusted life year ($/QALY), the current Health Technology Assessment (HTA) process used in Canada fails to account for other variables that can greatly affect patient outcomes and add to the overall societal value of a medication.

While it may be appealing in its perceived simplicity and purported objectivity, the $/QALY is neither simple nor objective. Its calculation requires complex modeling and a myriad of accompanying assumptions. In reality, at its very best it should be considered a mere estimation, comprising a wide distribution of possible values. Which value most closely approximates “reality” depends largely on the perspective of the observer and which assumptions they choose. In other words…it is highly subjective. Why then does our current HTA process resign itself almost exclusively to this mediocre surrogate for value? Proponents will argue that it is the best of a bad lot. But this seems to me to be too convenient a response when one considers the profound impact that reimbursement decision making has on the health of individuals and our society. No doubt the decisions in question are difficult. But decision makers and those around them have an obligation to constantly improve upon their methodology and to conduct their evaluations from the perspective of society overall, not simply that of the health system budget silo.

A truly comprehensive HTA process should account for variables other than $/QALY and should strive to incorporate factors from outside of the health system. The objective of this approach should be to compliment and improve traditional HTA; not replace it.

At Amgen, we believe that if the framework used to assess new drugs is to evolve, it is essential to stimulate debate and dialogue among stakeholders, including patients, healthcare providers, academics and governments. We have conducted research that looks at alternative models for decision-making in other parts of the world and found that multi-criteria decision-making is used effectively in such countries as France, Italy and Sweden.

In work reported in Health Economics, Goldman and colleagues described a model that supports the use of a broader definition of value¹. Called Comprehensive Benefits of Value, or CBV, this framework goes beyond traditional HTA considerations to assess such important factors as the level of innovation, societal benefits, disease severity, unmet need and enhanced patient experience. We believe this type of qualitative review in tandem with traditional HTA will ultimately help improve quality of reimbursement decision making.

It has been said that one could place all of the economists on earth end to end and that they still would not reach an agreement. This is no less true of their more specialized health economist brethren. But one thing health economists, healthcare providers, patients, governments and all other stakeholders must absolutely agree on is that the individual and societal consequences of reimbursement decisions are profound. The weight of this responsibility compels all involved to subject the decision making apparatus to unrelenting scrutiny and debate and to strive for continuous improvement. It is a mission in which “failure is not an option”.

More information on Novel Frameworks for Reimbursement Decision Making.

Reference:
¹ Goldman D, Lakdawalla D, Philipson T, and Yin W. Valuing Health Technologies at NICE: Recommendations for improved incorporation of treatment value in HTA. Health Econ. 19:1109-1116(2010)

When deciding on reimbursement criteria for pharmaceutical products, it’s understandable that makers must take a hard look at cost-effectiveness. After all, health expenditures, which are on the rise, must be managed within our public system.

Yet the emphasis on cost-benefit analyses, the lens through which we view public coverage, often leaves gaps in appropriate care options for Canadians.

Often, when new and innovative therapies emerge, the focus of decision making can centre on cost-effectiveness, rather than clinical effectiveness.

As a result, access to medicines can be uneven across Canada, depending on the decision-making process in each jurisdiction.

This problem isn’t new. But if we’re going to evaluate drug reimbursement in large measure on the basis of cost, let’s make sure we factor in all the costs.

For instance, taking the example of osteoporosis, let’s say Drug A is publicly reimbursed and works well for Patient X. That’s cost-effective. But what about Patient Y, who’s vulnerable because Drug A doesn’t work for him? What happens if Patient Y goes on to break a hip, is hospitalized, requires surgery, and time to rehabilitate? How does that now play into the cost-benefit analysis of Drug A?

Or consider the case of a 50-year-old woman with osteoporosis, who manages the household, is a working mother, and is the primary caregiver for an elderly parent. For whatever reason, she either can’t take the approved osteoporosis drugs that are currently covered, or they aren’t effective for her. So there’s a high likelihood that she’ll eventually break a hip. Any fall could do it.

What happens now? Well, not only is the woman affected; so is her entire family. The household can’t run as effectively with her out of commission. While off work, her income might suffer. Certainly, her ability to care for her own parent is hampered.

Those impacts might not be health care costs, but they are societal costs. And one way or another, we do pay for them.

The question is this: how do we account for such a scenario when using traditional health technology assessment (HTA) principles to make reimbursement decisions?

We simply must find a way to factor in these issues and other qualitative measures to complement the traditional HTA.

Without question, this is complex. For example, a drug could be less expensive, but might not be as effective from a clinical perspective. So how do we assess the long-term effects and costs of a disease, of which drug costs are merely one element?

There have been some promising developments. For instance, there’s now a vehicle in place to allow patient input into the Common Drug Review (CDR), which provides formulary listing recommendations to the publicly-funded drug plans in Canada. But it’s too early to tell how much the patient experience will influence those recommendations.

The reality is that healthcare costs will only increase as the Canadian demographic ages. The toll of chronic diseases and related conditions will also continue to grow. Yes, we need to rein in costs. But we need to paint a fuller picture of what certain decisions really cost us.

Dr. Famida Jiwa is President and CEO of Osteoporosis Canada (www.osteoporosis.ca).

I read with great interest the paper by Geoff Sprang entitled :”Traditional reimbursement decision-making misses the big picture”. The drawbacks and flaws of $/QUALY, so dear to Canadian Health Technology Assessment, are too numerous and too well recognized to deserve further listing. This does not mean it should be totally abandoned since, at the very least, it will provide a frame of reference on which to measure potential alternatives. An interesting alternative is the Comprehensive Benefits of Value (CBV) acting as a modifier of the Cost-effectiveness threshold (CET).

The currently used CET is a mythical figure that is often discussed but very rarely officially defined. The ongoing rumor being that it corresponds to the cost/QUALY of hemodyalisis. This makes sense given that a society willing to pay for hemodyalisis should, to be consistent, pay for interventions with lower costs/QUALY. Besides all the problems inherent in the quantification of the cost/QUALY of a specific intervention one should realize that the CET should be continuously (almost in real-time) adjusted to the value of the highest cost/QUALY technology reimbursed by society at the moment of the evaluation of the new intervention. Furthermore, from a practical point of view, more often than not the CET sits somewhere between the highest and the lowest value of the cost/QUALY produced by the sensitivity analysis.

The proposal to use the CBV to adjust the cost-effectiveness threshold has considerable merit. As a matter of fact it is just a formalization and systematization of a process that has been ongoing for at least twenty years. How else can you explain the fact that given the same cost/QUALY value the drug reimbursement decisions made by different Canadian provinces show such a high degree of divergence? It is quite obvious that informally, almost subconciently, they have been applying a modifier to the cost/QUALY information they receive. This modifier will vary from province to province. When I look at the components of the CBV total score I recognize most of the factors that are informally taken into consideration by Canadian provincial drug-reimbursement organizations.

I strongly believe that attempts at quantification are a good thing, as long as they are followed by a formal evaluation. The components of the proposed CBV are comprehensive and I do not see a need for additional parameters. One still faces the daunting task of assigning weights which are proportional to the “value” of the component and the quality of the evidence on which it is based. In my experience the members of drug reimbursement advisory committees give a high value to innovation and tend to disregard patient convenience. I am not convinced that patient representatives would arrive at the same conclusions. Which leads to the question: who should participate in the exercise and which mechanism should be used? The following challenge would be the validation of the CBV given the fact that there is no gold standard. At worse we would end up with terms of reference that would certainly make the process better structured and hopefully more transparent.

In spite of the huge challenges, the present process is too unsatisfactory, and the stakes too high, for us to continue being complacent and to persist in using a methodology in which must of us do not believe. The Amgen proposal has the merit of providing a basis to at least initiate a long overdue discussion which should focus on how to best serve the patients interest.

Healthcare reimbursement decision-making at the level of prescription drugs is complex and is based on a number of decision-making criterion like cost per quality-adjusted life year ($/QALY). Like a staged or hurdle process, the first and foremost consideration for prescription medication decisions is safety followed by efficacy and effectiveness. Cost-effectiveness analysis is a component of the decision-making process, but it is by no means the ‘decision-making rule’ for any decision-making body in Canada (drugs and non-drugs included). Although it is certainly true that cost-effectiveness is a consideration, safety, effectiveness, cost-effectiveness and budgetary impact all play important roles. These four decision-making criteria are the mainstay considerations used by almost all prescription drug decision-makers at the national and provincial levels. Another point to emphasize at the outset is that although cost-effectiveness has been the subject of criticism and debate, it is important to note that cost-effectiveness is made up of two components, costs and effects. Regardless of the tools used to measure, project, value or represent costs and outcomes, the point remains that any measure of value for money is driven largely by the level of effectiveness of a new drug and the price set by manufacturers for the new drug.

These rather basic issues aside, I agree entirely with the notion that prescription drug decision-making, and healthcare decision-making more broadly, should be based on a larger set of criteria than has historically been the focus. This does not minimize the role of cost-effectiveness as an input into the decision-making process, but only grounds it more firmly into the process as opposed to perceptions that cost-effectiveness is a ‘decision-making rule’. Approaches like Multi-Criteria Decision Analysis (MCDA) and Comprehensive Benefits of Value (CBV) share the common purpose of expanding the list of possible decision-making criteria beyond the four which have traditionally been used in prescription drug reimbursement decision-making. As pointed out by Sprang in his article, using multiple criteria is not a new concept in decision-making and has been used more comprehensively in other fields and in other jurisdictions. Of course one could argue prescription drug reimbursement decision-making is already based on multiple criteria (i.e. safety, effectiveness, cost-effectiveness, budgetary impact), and perhaps we are simply talking about an expansion of the list of criteria that should be considered. An expanded list of criteria is certainly welcome in a climate of transparency since if ‘other factors’ are important considerations, then what these other factors are, and how they are measured and assessed, should be fully disclosed and open for comment and debate. This is exactly the approach taken by the Ontario government for device and procedure evaluations through the Ontario Health Technology Advisory Committee (OHTAC) where an expanded list of 9 criteria is used for its deliberations.

But the story does not end with simply an expanded list of criteria. Instead of a ‘holistic’ approach like that used by OHTAC where evidence and information for each decision-making criteria are considered altogether (i.e. unweighted), MCDA and CBV type approaches also recommend weights be applied to each criteria and levels be assessed for each so that total quantitative scores can be calculated. Although very appealing on grounds of both forcing consideration of an expanded list of criteria and forcing transparency on the relative weights assigned to these other factors in the decision-making process, these approaches also present a number of challenges which will need be to addressed and refined as the expanded list of criteria develop.

The first challenge relates to how this expanded list of criteria is developed for each decision-making body. Different bodies have different philosophies and missions/objectives and obviously face different constraints. Guidelines and tools need to be developed to assist different decision-making bodies in determining what these criteria should be for their organization.

The second challenge relates to the collection of information for each of the identified criteria. Researchers have decades of experiences generating, collecting and synthesizing scientific evidence around safety, efficacy and effectiveness. However, where do we get information on decision-making criteria that may be identified like social values (e.g. patient acceptance), ethical issues (e.g. reducing health disparities) or unmet patient need? What literature or survey methods need to be used to collect the evidence and information needed to properly assess these criteria?

The third challenge relates to assessing the ‘level’ of evidence/information for each criterion. Important issues for consideration here include the number of levels needed, how cut-points are defined, how the defined levels relate to available data, and what are the interval scale properties across the chosen cut-points.

The fourth challenge relates to weights assigned to the criteria itself. Issues such as how to arrive at weights, how to validate weights, and how to re-assess weights over time are all important considerations here.

The fifth challenge relates to how criteria weights and criteria levels are combined and aggregated. Is it as simple as multiplying criteria weights by levels and summing to arrive at a total score? Is the functional form of this mathematical equation additive, multiplicative, multilinear,…?

I certainly do not want to give the wrong impression that an expanded list of decision-making criteria should not be considered or used, or that the data sources for collecting evidence or information for these criteria should not be fully explored, or that perhaps even weights or relative preferences should not be assigned to the identified criteria. I believe these are all important for a more comprehensive and transparent decision-making process, for which I am a big advocate. However, research and guidelines are needed before the rigor of these approaches even comes close to the rigor which has been developed around the assessment of safety, effectiveness and even cost-effectiveness. It is certainly true that some components of cost-effectiveness analyses are subjective. However, decision-making based on multiple criteria (i.e. holistic and unweighted approaches, MCDA or CBV) already has substantially more assumptions and subjectivity inherently built into these systems and processes. If someone is concerned about the ‘softness’ of some of the assumptions/approaches used in traditional decision-making, this concern should be compounded significantly more when the list of criteria is expanded, when weights and levels are defined, when ‘hard’ and ‘soft’ information is combined, when scientific and colloquial evidence are combined and when all this is combined into assumed mathematical formulas and function forms.

Nevertheless these developments are very encouraging and I look forward to participating in the development of more rigorous tools, instruments and processes for incorporating this expanded list of criteria in more comprehensive and transparent healthcare decision-making processes.

Geoff Sprang argues that traditional HTA misses the “big picture”. Traditional HTA is all about cost per QALY. Because this is inadequate, we need additional models of reimbursement decision making. He suggests that one such framework is the “Comprehensive Benefits of Value” framework, as described by Goldman. Several comments follow.

• Who is it that’s doing the asking, and why? Is this issue being raised now because the rate of reimbursement for new drugs is seen to be too low by industry? This type of motivation is suggested by the opening rationale, which expresses concern about Canadians’ inability to access new technology? Of course, this doesn’t mean that the questions being raised are not legitimate, but it is helpful to keep the broader agenda of this debate in mind.
• “Conventional HTA is all about cost per QALY gained”. If only it were so. There is, first of all, really no such thing as “conventional HTA”. To the extent to which there is a common ground, HTA is mostly about evidence synthesis, to a less extent about economic evaluation, particularly in Canada, and to some extent about the broader social and ethical considerations associated with technology. HTA is by definition a mongrel, with a mixed pedigree derived from evidence based medicine, health economics/decision analysis, and bioethics/social science. In Canada, most HTA decision making bodies are rooted in a tradition of evidence based medicine- an HTA will nearly always include an evidence synthesis or review, but not always an economic evaluation. Canadian decision making frameworks always include factors other than value for money. The framework of the Ontario Health Technology Advisory Committee, for example, includes four factors: i) Evidence of effectiveness and safety; ii) Social and ethical concerns; iii) Cost effectiveness; iv) Feasibility- system sustainability.
• What about drug reimbursement decision making in Canada? Is it really all about cost/QALY calculations? There’s no doubt that questions of value (one measure of which is incremental cost per QALY gained) play a larger role than they used to. But decision making around new drugs is highly complex, varies widely between provinces, and takes many other things into account. Nearly always, pride of place goes to effectiveness and safety. Health economic considerations are playing a larger role, as they should, in my view, but we are a long way, say, from the situation in the UK, where cost effectiveness plays a very substantial role. Economic analyses vary widely in quality, are nearly always conducted by those who have a large incentive to produce a particular result, and are poorly understood by decision makers. The role of health economic considerations, in my view, is still fairly modest in Canada.
• What about the role of “broader questions”- i.e. issues beyond effectiveness and cost effectiveness? Sprang has a point here- conventional analyses can be reductionistic, in that the presence of randomized trial evidence and a favourable cost effectiveness ratio are seen as “hurdles” that new technologies must surmount. HTA agencies and reimbursement groups globally are trying to work out how to include broader dimensions of value in decision making. Patients are sitting on guidelines committees and reimbursement decision making bodies. Citizens’ panels are proliferating. New methods are being explored to consider patients’ perspectives and values through primary and secondary research. This is a welcome, and overdue development. The danger, though, is lowering the bar. There is a tradeoff between considering broader perspectives and weakening the science. Allowing values and preferences to take a seat at the table may attenuate the influence of “conventional” considerations like, “does it work?”, and “is it a good deal?”. We need to work out how to include those considerations too- without weakening the scientific perspective that gives our current work its credibility.

Geoff Sprang correctly notes that the use of traditional Health Technology Assessment (HTA), including a formal systematic review of the clinical evidence (safety and effectiveness in comparison to current care), and consideration and critique of a manufacturer-provided economic evaluation, provides the backbone for the evaluation of drugs within the Canadian Common Drug Review. He also notes that traditional HTA does not have a formal framework for considering other aspects of a drug that are important to consider during the reimbursement process, including the severity of the disease condition, whether there are any other treatment options for the condition in question, and whether the condition is rare or common (among other considerations).

As others have proposed, Geoff proposes that the Canadian Common Drug review should implement a formal process that considers these other factors, to reduce the subjectivity in drug assessment. As he notes, these factors are already considered informally for all drugs reviewed by the Canadian Expert Drug Advisory Committee (CEDAC), and a conscious effort to increase consideration of these factors was made when public committee members were added to CEDAC in 2006.

While the quality adjusted life year continues to be a lightning rod for criticism (and undoubtedly always will be), it remains a reasonable proxy for what health care systems are trying to achieve (maximizing life expectancy and quality of life), with uncertainties around its measurement often relating mostly to inadequacies in the clinical data that exist at the time of drug submission. Similarly, there are many issues with multi-criteria decision analysis, including what other factors to consider and how to reduce subjectivity in item measurement. Relating to the comprehensive value assessment scheme offered, inclusion of “innovation” adds an additional dimension of complexity since the definition of innovation differs between decision makers and manufacturers. Most health care decision makers consider innovation in terms of whether a drug offers improvement in either life expectancy or other measures of health compared to treatments that are currently being used (both of which are incorporated into traditional HTA) whereas manufacturers often consider innovation to reflect a drug’s mechanism of action or delivery method.

It is clear that recommendations made by drug reimbursement committees vary across countries (even for the same drugs)1, though it is less clear whether this is due to differences in the drug plans supported, different interpretations of the clinical or cost-effectiveness evidence, or relates to different interpretations of these other subjective factors noted above. Given that these factors will always remain subjective, even when codified, the true test of the value of frameworks like multi-criteria decision analysis would be to determine whether funding recommendations differ (or become more consistent across countries) if a formal framework to facilitate comprehensive value assessment were incorporated into decision making. Given the complexity of the decision-making process, and reliance in the end on experts in the field, necessitating expert committees like CEDAC, I suspect recommendations would not change substantively.

Reference:
¹ Clement FM, Harris A, Li JJ, Yong K, Lee KM, Manns BJ. Using effectiveness and cost-effectiveness to make drug coverage decisions: a comparison of Britain, Australia, and Canada. JAMA 2009;302:1437-43.