Healthcare reimbursement decision-making at the level of prescription drugs is complex and is based on a number of decision-making criterion like cost per quality-adjusted life year ($/QALY). Like a staged or hurdle process, the first and foremost consideration for prescription medication decisions is safety followed by efficacy and effectiveness. Cost-effectiveness analysis is a component of the decision-making process, but it is by no means the ‘decision-making rule’ for any decision-making body in Canada (drugs and non-drugs included). Although it is certainly true that cost-effectiveness is a consideration, safety, effectiveness, cost-effectiveness and budgetary impact all play important roles. These four decision-making criteria are the mainstay considerations used by almost all prescription drug decision-makers at the national and provincial levels. Another point to emphasize at the outset is that although cost-effectiveness has been the subject of criticism and debate, it is important to note that cost-effectiveness is made up of two components, costs and effects. Regardless of the tools used to measure, project, value or represent costs and outcomes, the point remains that any measure of value for money is driven largely by the level of effectiveness of a new drug and the price set by manufacturers for the new drug.

These rather basic issues aside, I agree entirely with the notion that prescription drug decision-making, and healthcare decision-making more broadly, should be based on a larger set of criteria than has historically been the focus. This does not minimize the role of cost-effectiveness as an input into the decision-making process, but only grounds it more firmly into the process as opposed to perceptions that cost-effectiveness is a ‘decision-making rule’. Approaches like Multi-Criteria Decision Analysis (MCDA) and Comprehensive Benefits of Value (CBV) share the common purpose of expanding the list of possible decision-making criteria beyond the four which have traditionally been used in prescription drug reimbursement decision-making. As pointed out by Sprang in his article, using multiple criteria is not a new concept in decision-making and has been used more comprehensively in other fields and in other jurisdictions. Of course one could argue prescription drug reimbursement decision-making is already based on multiple criteria (i.e. safety, effectiveness, cost-effectiveness, budgetary impact), and perhaps we are simply talking about an expansion of the list of criteria that should be considered. An expanded list of criteria is certainly welcome in a climate of transparency since if ‘other factors’ are important considerations, then what these other factors are, and how they are measured and assessed, should be fully disclosed and open for comment and debate. This is exactly the approach taken by the Ontario government for device and procedure evaluations through the Ontario Health Technology Advisory Committee (OHTAC) where an expanded list of 9 criteria is used for its deliberations.

But the story does not end with simply an expanded list of criteria. Instead of a ‘holistic’ approach like that used by OHTAC where evidence and information for each decision-making criteria are considered altogether (i.e. unweighted), MCDA and CBV type approaches also recommend weights be applied to each criteria and levels be assessed for each so that total quantitative scores can be calculated. Although very appealing on grounds of both forcing consideration of an expanded list of criteria and forcing transparency on the relative weights assigned to these other factors in the decision-making process, these approaches also present a number of challenges which will need be to addressed and refined as the expanded list of criteria develop.

The first challenge relates to how this expanded list of criteria is developed for each decision-making body. Different bodies have different philosophies and missions/objectives and obviously face different constraints. Guidelines and tools need to be developed to assist different decision-making bodies in determining what these criteria should be for their organization.

The second challenge relates to the collection of information for each of the identified criteria. Researchers have decades of experiences generating, collecting and synthesizing scientific evidence around safety, efficacy and effectiveness. However, where do we get information on decision-making criteria that may be identified like social values (e.g. patient acceptance), ethical issues (e.g. reducing health disparities) or unmet patient need? What literature or survey methods need to be used to collect the evidence and information needed to properly assess these criteria?

The third challenge relates to assessing the ‘level’ of evidence/information for each criterion. Important issues for consideration here include the number of levels needed, how cut-points are defined, how the defined levels relate to available data, and what are the interval scale properties across the chosen cut-points.

The fourth challenge relates to weights assigned to the criteria itself. Issues such as how to arrive at weights, how to validate weights, and how to re-assess weights over time are all important considerations here.

The fifth challenge relates to how criteria weights and criteria levels are combined and aggregated. Is it as simple as multiplying criteria weights by levels and summing to arrive at a total score? Is the functional form of this mathematical equation additive, multiplicative, multilinear,…?

I certainly do not want to give the wrong impression that an expanded list of decision-making criteria should not be considered or used, or that the data sources for collecting evidence or information for these criteria should not be fully explored, or that perhaps even weights or relative preferences should not be assigned to the identified criteria. I believe these are all important for a more comprehensive and transparent decision-making process, for which I am a big advocate. However, research and guidelines are needed before the rigor of these approaches even comes close to the rigor which has been developed around the assessment of safety, effectiveness and even cost-effectiveness. It is certainly true that some components of cost-effectiveness analyses are subjective. However, decision-making based on multiple criteria (i.e. holistic and unweighted approaches, MCDA or CBV) already has substantially more assumptions and subjectivity inherently built into these systems and processes. If someone is concerned about the ‘softness’ of some of the assumptions/approaches used in traditional decision-making, this concern should be compounded significantly more when the list of criteria is expanded, when weights and levels are defined, when ‘hard’ and ‘soft’ information is combined, when scientific and colloquial evidence are combined and when all this is combined into assumed mathematical formulas and function forms.

Nevertheless these developments are very encouraging and I look forward to participating in the development of more rigorous tools, instruments and processes for incorporating this expanded list of criteria in more comprehensive and transparent healthcare decision-making processes.