With the increasing impact of artificial intelligence (AI), researchers are exploring the role of technology in clinical decision-making and explanations in healthcare.
“Clinicians must calibrate their judgement against a whole constellation of other factors, even if they are using an AI tool that is well validated and highly accurate,” said Dr Ian Stedman, a lawyer and professor of public policy at York University in Toronto, Canada, who has co-authored a paper on AI with Australian researchers.
An analytical review conducted by the University of Adelaide outlines explainability as one of the major challenges in health AI and explores whether explanations of specific predictions for individuals is absolutely necessary to make a good decision.
Explainability, also referred to as ‘interpretability’ is the concept that a machine learning model and its output can be explained in a way that makes sense to a human being at an acceptable level. It is a part of explainable AI, which is AI in which humans can understand the reasoning behind decisions or predictions made by the AI.
The field of explainability which focuses on individual-level explanations is a developing one, said Dr Melissa McCradden, of the University of Adelaide’s Australian Institute for Machine Learning (AIML).
“We are optimistic about where the field can go, but with where we are right now, requiring prediction-level explanations for clinical decision-making is problematic,” she said.
There are two types of explainability – inherent explainability and post-hoc explainability, Dr McCradden explained.
Inherent explainability refers to understanding how the model as a whole functions and post-hoc refers to attempts to understand the means by which a specific prediction was generated by the model.
“Some models are directly interpretable, meaning that the operations from inputs to outputs are easy to follow and clear such as decision trees. Others are more opaque, meaning that the process from inputs to outputs is difficult or impossible to follow precisely, even for developers,” said Dr McCradden.
“The issue is in health AI, clinicians typically believe an explanation is what they are getting when they see something like a heatmap, or a prediction accompanied by the reasons the patient received this output. This is understandably what many clinicians want, but new evidence is showing that it might nudge them to make less accurate decisions when the AI tool is incorrect.”
A good clinical decision is not only one that advances the goals of care, but it also must be legally defensible, said Dr McCradden. Explainability alone shouldn’t serve as an essential part of ethical decision making, the researchers argue.
Clinicians are required to draw conclusions from evidence and understanding, placing the patient at the centre of the process instead of AI.
“Piling more weight onto the value ascribed to the AI tool’s output further shifts the emphasis away from the patient – their wishes, their culture, their context,” said Dr Stedman.
“Historically, reasonable judgements have been made on the basis of the totality of evidence and resources available to the clinician, contextualised in light of the patient’s specific situation.”
Both Dr McCradden and Dr Stedman concluded it is highly unlikely that an AI prediction would be the sole source of information by which a clinician makes a decision, particularly as their performance is never 100% perfect.
“It will, for the foreseeable future, always be necessary to triangulate sources of evidence to point to a reasonable decision,” said Dr McCradden.
“In this sense, physicians should consider what, specifically, the AI tool’s output contributes to the overall clinical picture. But we always need to be grounded by the patient’s wishes and best interests.”
The research is published in the Future Healthcare Journal.
