A comparative study recently demonstrated the promise held by PreSize Neurovascular (Oxford Heartbeat) software in consistently and accurately making predictions about flow-diverter placement in intracranial aneurysm treatments. However, according to lead study author Ansaar Rai (West Virginia University Rockefeller Neuroscience Institute, Morgantown, USA), a potentially even more revealing aspect of this research was the discrepancy observed across a number of experienced physicians, and their attempts to estimate flow-diverter deployments manually.
“The results showed that there was significant variability amongst the physicians,” Rai tells NeuroNews. “And, there was no difference between the degrees of experience—so, for a very experienced physician and a somewhat experienced physician, there was not much of a signal that the more experienced physician would do better.”
In a study now published in the Journal of NeuroInterventional Surgery (JNIS), eight experienced neurointerventionists were provided with volumetric data on a patient’s anatomy from their preoperative angiogram, alongside images annotated with the distal landing location of a deployed Surpass Evolve (Stryker) flow diverter, from 51 previously treated cases of unruptured internal carotid artery (ICA) aneurysms. The physicians were, however, blinded to “all other procedural details”. They were then asked to perform “a planning routine reflecting their normal practice”, and estimate the point at which the flow diverter’s proximal end would land.
According to Rai, while imaging technologies themselves have progressed significantly in recent times, methods for predicting flow-diverter implantations—including the “behaviour” of the device inside a patient’s vasculature—are currently “lagging behind”. He notes that neurointerventionists rely primarily on their experience, alongside any information provided by the vendor about the device in question, to make such predictions in a “very manual way” that is also “completely based on operator intuition”.
Rai also highlights that the aneurysm not fully closing, insufficient coverage of lateral branches and poor wall apposition are among the various consequences stemming from imprecise flow-diverter deployment. Each of these can lead to subsequent devices needing to be deployed as well as an increased risk of complications.
In their JNIS paper, Rai et al state that, while operator-dependent approaches are “intrinsically prone to errors”—and it has long been suspected that physician estimates vary in precision—such variability “has never been previously documented”. They therefore set out to chart these estimations in a cohort of experienced neurointerventionists spanning the USA, UK, Australia and Finland. And, across the same 51 historical cases, they also evaluated the predictive performance of PreSize—an artificial intelligence (AI)-powered software providing real-time planning support, 3D reconstructions of patient-specific anatomies, and simulations of implantable device deployments within a safe, virtual environment.
Predictions made using both of these approaches were subsequently compared with the postprocedural observed positioning and deployed stent length—put simply, “what actually happened”—in each case, according to Rai. Bland-Altman plots were used to make these comparisons, while accordance between predictions from each of the neurointerventionists was determined via intraclass correlation coefficients (ICCs).
Rai and colleagues’ salient finding is this: the mean accuracy of neurointerventionist-estimated flow-diverter length after deployment was 81.3% (±15%), with outliers as low as 7%, versus PreSize’s accuracy of 94.9% (±4%), with its lowest evaluated accuracy across the study sample being 84%. From this, they conclude that the software demonstrated a “significantly higher accuracy” in comparison with conventional sizing methods (p<0.001). The Bland-Altman plots also showed that PreSize’s accuracy was better than the mean accuracy across the eight physicians in each and every one of the 51 cases included in the study.
As Rai notes in conversation with NeuroNews, much of the inaccuracy and case-by-case inconsistency associated with traditional sizing approaches was ultimately confirmed by this research. There was no particular case that all the included neurointerventionists performed poorly on, nor an individual neurointerventionist who performed consistently better or worse than the rest of the group. Average accuracies achieved by the individual physicians ranged from 73.1–87.6%, and an ICC of 0.64 also indicated “poor-to-moderate interrater agreement” between their predictions. Furthermore, no discernible trends were found between accuracy and a number of vasculature/morphology-related factors, including aneurysm size, vessel diameter, and tortuosity, indicating that physician predictions were similarly variable regardless of how theoretically challenging the case was.
Rai has already presented these data at multiple scientific meetings, and reports that such wide variability—between individual physicians and specific cases alike—being laid out in a quantified manner has proved to be “even more eye-opening” than the superior performance of PreSize. While manual estimations were known to be at least suboptimal in terms of accuracy, such “extreme differences” were still perhaps beyond what many would have anticipated.
However, the second major aspect of Rai et al’s study can be seen to offer a more positive outlook for the neurointerventional world. Their analyses revealed that PreSize was “significantly more consistent and quantitatively accurate” in predicting the deployed length and location of flow diverters. This rang true whether the physicians and their estimations were considered as a large, composite group, or individually.
Rai reports that variability with the technology was “much less broad” too. The mean absolute error between controls (real-world deployed lengths), and the prediction methods used in the study, was 4mm (range, 0.03–30.2mm) for the neurointerventionists compared to just 1mm (range, 0.01–3.9mm) for PreSize. And, unlike six of the eight neurointerventionists included in the study, PreSize demonstrated “no over- or underestimation bias” overall. In addition, a small, negative correlation with aneurysm size was the only trend found between the technology’s accuracy and any of the aforementioned vascular/morphological factors.
Discussing the wider significance of these results, Rai says: “This is not the first study that has evaluated a software or AI-assisted tool for predicting device behaviour; but, it is the first study that has actually validated such a tool against a large cohort of physicians via a head-to-head comparison. That validation is important because, if someone comes to me with a software claiming it can do ‘X, Y and Z’, my first question is: how reliable is it, and what is the level of evidence [supporting] those predictions?”
“Having used the PreSize technology, I can say I wish it was available and FDA [Food and Drug Administration]-approved in the USA,” he continues, going on to explain that his centre and many others are currently using the software solely under research protocols rather than in everyday practice. PreSize has, however, received a CE mark and is already being used in several European centres.
Here, Rai also cites an anecdotal advantage of PreSize—the fact it is presented on a relatively simple and easy-to-use dashboard, and does not add a significant amount of time to the procedure. And, while the technology in its current form is something of an ‘add-on’, yet to be fully integrated into the established imaging systems many neurointerventionists use today, Rai believes progress is coming on this front too. Once deployed more widely and backed by an even greater wealth of evidence, he foresees this type of simulation technology being incorporated into existing workstations by major imaging vendors like Philips and Siemens Healthineers, and usage likely becoming “part and parcel” of flow-diverter procedures.
“I am positive that will happen—and I do not think it is even that far off,” Rai adds. “Once this happens, usage will increase, because you make it easier for the operator. Assessing the cerebrovascular anatomy, planning the procedure and selecting an implant [will be] no different dependent on the acuity of the case. You do the same thing over and over again, and it just becomes a part of the workflow every single time. It will be used universally, I have no doubts about that.”
Rai also notes that simulation tools like PreSize could expand beyond elective, unruptured aneurysm cases and flow diverters, and—“eventually”—move into more urgent, acute cases like ruptured intracranial aneurysms, and potentially even the deployment of intrasaccular devices, because “the ability is definitely there”.
Returning to the present JNIS study, Rai acknowledges that “this is not definitive” and “by no means settles the question”; while its large cohort of cases and blinded setup contribute to it being a “good study”, additional research endeavours that are fully prospective and randomised in nature will be needed to bolster existing evidence. Pertinently, a first-of-its-kind study for the neurointerventional space assessing PreSize’s real-world performance in a clinical setting is currently underway in the UK and plans to enrol a total of 100 patients.
With promise already shown in this paper and others, and plenty more evaluations to come, Rai believes it is only a matter of time before simulation technologies like PreSize find their way into many neurointerventionists’ everyday practice.
“My message would be that I think this is where the future is headed,” he avers. “Tools like this will be incorporated into our cath-lab workstations. And this will be significant because, to a large extent, it will take the need for guesswork away from physicians.”
