Testing Artificial Intelligence software used to identify stroke features

Grant Mair, the Stroke Association’s Edith Murphy Foundation Senior Clinical Lecturer, University of Edinburgh, and honorary consultant neuroradiologist, NHS Lothian, Scotland. At the ESC-WSO 2020 conference he gave a talk entitled, “Large-scale clinically representative evaluation of e-ASPECTS & e-CTA software, the RITeS study (real-world independent testing of e-ASPECTS software).”

Software designed to assist clinicians interpret brain imaging in stroke is increasingly available. Yet there are no agreed standards for assessment and no requirement for software to demonstrate tangible improvement in patient care. Published testing often lacks independence from software developers and datasets may be under-powered and poorly-defined.¹ As a major charitable funder of stroke research in the UK, the Stroke Association commissioned research to provide robust, large-scale and independent testing of one commercially available option.

Initially designed for automated Alberta stroke programme early CT score (ASPECTS) scoring² of non-enhanced CT brain scans in acute ischaemic stroke, e-ASPECTS (Brainomix Ltd – Oxford, UK) is now part of a software suite and offers additional automated features including haemorrhage detection on CT, and CT angiography scoring (e-CTA).

The Real-world Independent Testing of e-ASPECTS Software (RITeS) collaboration comprises researchers from four UK centres and one in Germany. All five centres have extensive experience conducting stroke research, with access to large numbers of imaging studies approved for research. The RITeS database comprises nine completed national/international clinical stroke trials or registries^6-14 with >6000 unique patients covering the gamut of potential diagnoses underlying acute stroke symptoms (ischaemia, haemorrhage, mimics). All imaging studies in the RITeS database were scored consistently by centralised expert readers, masked to clinical data including other imaging. Inter-rater agreement between experts for ASPECTS and angiography scoring was 0.56 and 0.70¹⁵, respectively. Scan assessment occurred before the RITeS study began.

The RITeS study sought a large number of clinically representative acute stroke cases from the RITeS database for a ‘real-world’ evaluation of e-ASPECTS and e-CTA. Primarily, we aimed to 1) compare ASPECTS results, and 2) compare CTA results (identification of obstruction) provided by software versus expert humans. Our study will also ultimately include assessment of e-ASPECTS haemorrhage detection, and a clinical impact evaluation, but these components are ongoing.

Our methods were pre-specified,¹⁶ and include intention-to-analyse testing, i.e. we attempted to process all available scans and did not exclude cases where imaging quality was poor, or where processing was unsuccessful. We aimed for the maximum number of unselected cases from those available with clinical and demographic representation similar to other large pre-specified datasets.^3-5 We manually identified the most appropriate DICOM image set per case (axial orientation, soft tissue kernel, thinnest slices available) and uploaded these to the Brainomix online platform (version 9.0p3). We recorded all scan processing outcomes. Multiple upload attempts were made if necessary and we tried alternative DICOM image sets if available.

We include 4117 patients (50.4% female, median age 78 years, median NIHSS 10, median 2.5 hours from symptom onset). Software successfully processed 89.1% (3660/4108) of CT and 81.6% (545/668) of CTA presented.

For assessment of ASPECT scoring, we include 3044 cases with ischaemic stroke and successfully processed imaging. In pairwise comparison of human and software results, nearly half (1409 scans, 46.3%) matched exactly (ASPECT score and side affected), while over two thirds (68.7%) were ±1 ASPECTS point, and over three quarters (82.3%) were ±2 ASPECTS points. For 356 scans (11.7%), humans and software scored the opposing cerebral hemisphere as abnormal. Human-software inter-rater agreement was 0.50.

For assessment of CTA, we include 545 cases with successfully processed imaging. In pairwise comparison of human and software results, nearly two thirds (344 scans, 63.1%) matched exactly (183 normal, 149 proximal/ 12 distal side-matched obstruction). Most mismatches were false negative (82 scans, 15.0%) or false positive (57 scans, 10.5%) calls by software relative to humans. Humans and software scored the same side but different abnormal vessels in 34 scans (6.2%), while for 13 scans (2.4%) the opposing side was scored abnormal. Human-software inter-rater agreement was 0.48.

RITeS provides a large, independent assessment of e-ASPECTS and e-CTA using non-selected clinically-representative cases. Our study relied on retrospective scan processing and expert opinion delivered non-acutely. Thus, our results may differ from routine practice where scans are processed prospectively, rapidly evaluated in real-time, and where readers may often be less experienced. For the primary software features of ASPECT scoring on CT and identification of CTA obstruction, e-ASPECTS and e-CTA agreed with human expert reviewers for around two thirds of cases. Software users should be aware that results may differ from expert opinion (18-31% in our analysis). e-ASPECTS and e-CTA should be used strictly within their approved indications to aid clinicians who understand stroke CT interpretation, and not by inexperienced readers.

Dr Grant Mair, on behalf of the RITeS Investigators.

REFERENCES

1. Mikhail P, Le MGD, Mair G. Computational Image Analysis of Nonenhanced Computed Tomography for Acute Ischaemic Stroke: A Systematic Review. J Stroke Cerebrovasc Dis 2020;29:104715.
2. Barber PA, Demchuk AM, Zhang J, Buchan AM. Validity and reliability of a quantitative computed tomography score in predicting outcome of hyperacute stroke before thrombolytic therapy. ASPECTS Study Group. Alberta Stroke Programme Early CT Score. Lancet 2000;355:1670-1674.
3. Sentinel Stroke National Audit Programme (SSNAP) Clinical Audit Apr-Jun 2019 Public Report. 2019. Available at: https://www.strokeaudit.org. Accessed 25th November 2020.
4. Emberson J, Lees KR, Lyden P, et al. Effect of treatment delay, age, and stroke severity on the effects of intravenous thrombolysis with alteplase for acute ischaemic stroke: a meta-analysis of individual patient data from randomised trials. Lancet 2014;384:1929-1935.
5. Roman LS, Menon BK, Blasco J, et al. Imaging features and safety and efficacy of endovascular stroke treatment: a meta-analysis of individual patient-level data. Lancet Neurol 2018;17:895-904.
6. Huang X, Cheripelli BK, Lloyd SM, et al. Alteplase versus tenecteplase for thrombolysis after ischaemic stroke (ATTEST): a phase 2, randomised, open-label, blinded endpoint study. Lancet Neurol. 2015; 14(4): 368–376.
7. van der Worp HB, Macleod MR, Bath PM, et al. EuroHYP-1: European multicenter, randomized, phase III clinical trial of therapeutic hypothermia plus best medical treatment vs. best medical treatment alone for acute ischemic stroke. Int J Stroke. 2014; 9(5): 642–645.
8. IST-3 collaborative group. The benefits and harms of intravenous thrombolysis with recombinant tissue plasminogen activator within 6 h of acute ischaemic stroke (the third international stroke trial [IST-3]): a randomised controlled trial. Lancet. 2012; 379(9834): 2352–2363.
9. Samarasekera N, Lerpiniere C, Fonville AF, et al. Consent for Brain Tissue Donation after Intracerebral Haemorrhage: A Community-Based Study. PLoS One. 2015; 10(8): e0135043.
10. Muir K, White P, Murray A, et al. Results of the Pragmatic Ischaemic Thrombectomy Evaluation (PISTE) Trial. Stroke. 2016; 47: LB9.
11. MacDougall NJ, McVerry F, Huang X, et al. Post-stroke hyperglycaemia is associated with adverse evolution of acute ischaemic injury. Cerebrovasc Dis. 2014; 37(suppl 1): 267.
12. El-Tawil S, Wardlaw J, Ford I, et al. Penumbra and re-canalization acute computed tomography in ischemic stroke evaluation: PRACTISE study protocol. Int J Stroke. 2017; 12(6): 671–678.
13. RESTART Collaboration. Effects of antiplatelet therapy after stroke due to intracerebral haemorrhage (RESTART): a randomised, open-label trial. Lancet. 2019; 393(10191): 2613–2623.
14. RIGHT-2 Investigators. Prehospital transdermal glyceryl trinitrate in patients with ultra-acute presumed stroke (RIGHT-2): an ambulance-based, randomised, sham-controlled, blinded, phase 3 trial. Lancet. 2019; 393(10175): 1009–1020.
15. Mair G, von Kummer R, Adami A, et al. Observer reliability of CT angiography in the assessment of acute ischaemic stroke: data from the Third International Stroke Trial. Neuroradiology 2015;57, 1–9.
16. Mair G, Chappell F, Martin C et al. Real-world Independent Testing of e-ASPECTS Software (RITeS): statistical analysis plan [version 1; peer review: 1 approved]. AMRC Open Res 2020, 2:20 (https://doi.org/10.12688/amrcopenres.12904.1)