A systematic review and meta-analysis of 1,356 studies and more than 410,000 patients have found that “methodological flaws and incomplete reporting” occur frequently in studies advocating the safety, effectiveness or durability of intracranial aneurysm treatments—forming a “potential public health hazard” for patients and, in the view of the authors, a “failure of the peer-review process”.
Writing in JAMA Network Open, Victor Volovici (Erasmus University Medical Center, Rotterdam, The Netherlands) and colleagues initially highlight an “unprecedented rise” in medical device numbers across various specialties, before also averring that the history of medicine “is laden with examples in which the mechanistic logic that underpins the development of a device does not translate to better outcomes for patients and might, instead, be harmful”.
In addition, Volovici and colleagues note that they conducted the present review in light of “the ongoing discussion within the stroke community regarding the best treatment of intracranial aneurysms and proper patient selection”, as well as “the paucity of substantial, properly generalisable randomised evidence” on these treatments.
In an effort to critically evaluate published claims of safety, effectiveness and durability with intracranial aneurysm treatments, the authors searched Embase, MEDLINE, Web of Science, and The Cochrane Central Register of Clinical Trials, for relevant studies published between 1 January 1995 and 1 October 2022, also retrieving grey literature from Google Scholar. Their key objective was to analyse for methods and completeness of reporting, and compare the results of studies with positive, uncertain and negative concluding messages.
“All studies making any kind of claims of safety, effectiveness or durability in the field of intracranial aneurysm treatment were included,” Volovici and colleagues state, also detailing that data extraction was performed by 10 independent reviewers; an academic neuro-linguist “without involvement in intracranial aneurysm research” evaluated the conclusion of each study as positive, uncertain, or negative; and that they followed Preferring reporting items for systematic reviews and meta-analyses (PRISMA) guidelines.
Upon inclusion of 1,356 studies comprising a total of 410,993 treated patients—the majority of which involved endovascular interventions (79%)—Volovici and colleagues’ analysis revealed that:
- There was no difference in the proportion of patients with a poor outcome or in-hospital mortality between studies claiming a technique was either safe, uncertain, or not safe
- There was no difference in the proportion of intracranial aneurysms completely occluded at final follow-up between studies claiming a technique was effective, uncertain, or non-effective—and a similar trend was observed in terms of durability between studies with a positive, uncertain or negative conclusion
- When limiting the results to studies on unruptured aneurysms, only the rate of in-hospital mortality was different between the studies with a positive and uncertain conclusion regarding safety
- Fewer than 2% of studies provided any definition of safety, effectiveness or durability, and only one of the 1,356 studies provided a threshold under which the technique of interest would be considered unsafe
- Incomplete reporting was found in 546 (40%) of the included studies, while “the conclusion in the abstract was more positive than the conclusion in the text” in 68 studies (5%)
- Ten of the 17 randomised controlled trial (RCT) reports included showed “a high risk of bias”, and all but one of these demonstrated “a high or concerning risk of bias in essential domains”
“Proving safety and effectiveness of a new technique is important prior to the widespread implementation of such a technique in clinical practice, and its prerequisite is common sense: a new technique should be compared with the current standard of care,” Volovici and colleagues write, discussing these findings.
They go on to assert that, “ideally”, such comparisons should be done in a multicentre RCT with predefined timepoints; blinded, uniform assessment of clinically and radiologically relevant outcomes; and predefined analyses as well as predefined safety and effectiveness outcomes. The authors add, however, that “these desiderates are currently almost entirely absent in published studies”. Speaking to NeuroNews to clarify this point, Volovici said that, “of course, those studies are expensive and, more importantly—because we are already treating patients very well on average—the numbers needed to be enrolled to prove superiority are in the thousands”.
“Comparing a new technique with current practice in a prospective manner should be considered the bare minimum for any safety or effectiveness claim to be allowed in a report,” the authors continue. “Of course, safety may also be claimed when the incidence of complications is low—but, without a control group, one cannot infer whether the technique should be implemented in practice.”
The majority of papers included in the present review were single-group and retrospective in nature and, as Volovici and colleagues state, “single-group studies lack a critical benchmark, in that the results obtained cannot be reflected against the results of current standard clinical practice, which makes any claim of safety, effectiveness or durability weak, or unjustifiable”.
Touching on the US Food and Drug Administration’s (FDA) 510(k) clearance protocol, the authors add that, while this process accelerates innovation, “there is evidence to suggest that the […] process is less robust and may lead to more recalls”, as compared to device approvals based on high-level evidence from clinical trials. They further claim that the scientific literature backing this process “may be methodologically less sound and sometimes even lacking”.
“A thorough, critical evaluation of the evidence used to back the FDA approval of these devices is necessary,” Volovici and colleagues state. “In our view, the current literature reflects a failure of the peer-review process. In a field with a very high level of technological advance, and with techniques that all carry a certain morbidity and mortality risk, reviewers and editors should be critical and hesitant to accept manuscripts with poor methods and reporting, and make sure that conclusions are supported by study results.”
Volovici and colleagues do acknowledge a number of limitations carried by their own review and analysis of the current literature—including missing study data, discrepancies between different medical definitions, and case-mix variability. However, they conclude by re-emphasising that methodological flaws and incomplete reporting occurred “often” across the studies they evaluated, adding that studies making unfounded claims about safety and effectiveness when they are not fit to do so “can hereby form a potential public health hazard” for intracranial aneurysm patients.
“The field of intracranial aneurysm treatment can be improved by applying proper methods and reporting, and performing comparative studies in which new technologies are compared with current best clinical practice,” the authors add.
Speaking to NeuroNews following the publication of these findings, Volovici tackled the “very important question” of whether or not it is realistic to expect that physicians and researchers in the aneurysm care space will, in future, make any significant strides to remedy something that appears to have become entrenched across several decades.
“For one, researchers need to better substantiate how and why they are doing these studies,” he said. “We treat these patients very well already, the results are very good, and we have good solutions for intracranial aneurysms—what is the incremental value, in terms of health economics, of a new device? And, conversely, is it at least as safe as best practice, and is it more effective; because the point of innovation is to fare better, right? If these issues cannot be addressed, and these desiderates demonstrated, then why is a new device necessary in the first place? Which health issues does it address?
“Secondly, peer reviewers and editors should be wary of accepting unedited conclusions claiming safety or effectiveness when studies cannot support these conclusions from a methodological perspective. Lastly, durability is the ‘orphan child’ of the lot. How durable are these techniques? We have no clue, and durability may be most important of all—the point of the treatment is to prevent bleeding! Only about 140 of these studies deal with durability, and most studies have a maximum of 19 months’ follow-up.”
Concluding, Volovici also highlighted a notable pattern revealed by time series analyses assessing temporal trends in intracranial aneurysm research: safety results “have been improving over the years”, but effectiveness results have not, which he feels “should also cause concern”. Volovici further reported that regression analyses in this review “seem to suggest” that the quality of research has decreased, on average, following the advent of stent-assisted coiling, with more studies claiming safety and effectiveness on “poor methodological grounds” since the wider adoption of this technique in the early 2000s.
