Background:

As part of our ongoing systematic review of complex interventions for the primary prevention of cardiovascular diseases, we have developed and evaluated automated machine-learning classifiers for title and abstract screening. The aim was to develop a high-performing algorithm comparable to human screening.

Methods:

We followed a three-phase process to develop and test an automated machine learning-based classifier for screening potential studies on interventions for primary prevention of cardiovascular disease. We labelled a total of 16,611 articles during the first phase of the project. In the second phase, we used the labelled articles to develop a machine learning-based classifier. After that, we examined the performance of the classifiers in correctly labelling the papers. We evaluated the performance of the five deep-learning models [i.e. parallel convolutional neural network (CNN), stacked CNN, parallel-stacked CNN, recurrent neural network (RNN) and CNN–RNN]. The models were evaluated using recall, precision and work saved over sampling at no less than 95% recall.

Results:

We labelled a total of 16,611 articles, of which 676 (4.0%) were tagged as ‘relevant’ and 15,935 (96%) were tagged as ‘irrelevant’. The recall ranged from 51.9% to 96.6%. The precision ranged from 64.6% to 99.1%. The work saved over sampling ranged from 8.9% to as high as 92.1%. The best-performing model was parallel CNN, yielding a 96.4% recall, as well as 99.1% precision, and a potential workload reduction of 89.9%.

Future work and limitations:

We used words from the title and the abstract only. More work needs to be done to look into possible changes in performance, such as adding features such as full document text. The approach might also not be able to be used for other complex systematic reviews on different topics.

Conclusion:

Our study shows that machine learning has the potential to significantly aid the labour-intensive screening of abstracts in systematic reviews of complex interventions. Future research should concentrate on enhancing the classifier system and determining how it can be integrated into the systematic review workflow.

Funding:

This project was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme and will be published in Health Technology Assessment. See the NIHR Journals Library website for further project information.

Building the text collections

The data set used for this study comprises a corpus of 16,611 titles and abstracts of articles (a subset of the 133,260 articles yielded by the more sensitive search), which were collected and labelled by a pair of human reviewers (with 0 representing a ‘not relevant’ study and 1 representing a ‘relevant’ study) from an ongoing systematic review on interventions for the primary prevention of CVD.¹⁴ We evaluated each identified study against the following selection criteria.

Classifier system

We developed automated machine-learning classifiers using Uber’s Ludwig low code deep-learning toolbox.¹⁵ Ludwig is a user-friendly deep-learning toolbox that allows users to train and test deep-learning models for a variety of applications, including text classification. We developed and compared the performance of the five deep-learning models [parallel convolutional neural network (CNN), stacked CNN, parallel-stacked CNN, recurrent neural network (RNN) and CNN–RNN].^16–21 Deep learning refers to ‘neural networks with multiple layers of perceptrons inspired by the human brain’ and has been shown to bring benefits in text generation, word representation estimation, sentence classification and feature presentation.^22–36 Recently, deep-learning models have been shown to perform better than traditional machine-learning algorithms in many natural language processing text classification applications.^22–36

Deep-learning methods, in addition to their high performance, do not rely on any hand-engineered features and can instead learn the most appropriate features for a given task. As a result, deep learning is currently the most popular and successful approach to natural language processing. The most common deep-learning architectures are CNN and RNN.^22,23 CNN is a type of deep, feed-forward artificial neural network (where node connections do not form a cycle) that employs a variant of multilayer perceptrons that is designed to require minimal pre-processing.^22,23 A RNN is a type of artificial neural network in which node connections form a directed graph along a sequence. Theoretically, RNN with a sequential architecture should be better suited for sequence modelling tasks (e.g. machine translation, language modelling or speech recognition) because these tasks require the representation of complex context dependencies.^22,23 Similarly, CNN’s hierarchical architecture should be more useful for text classification, where detecting representative patterns can be critical to solving the problem.^22,23 In fact, CNN architectures have been shown to outperform state-of-the-art text classification algorithms because they can extract the most informative ngrams (i.e. a contiguous sequence of n items from a given sample of text or speech) describing a text.^22,23

Evaluating the classifiers

We used the conventional information retrieval terminology of recall and precision, which are synonymous with sensitivity and positive predictive value, respectively, to evaluate the classifiers. In the current scenario, the recall statistic is of primary concern, as a means of confirming that eligible study reports are not discarded incorrectly from the systematic review. Precision is also an interesting metric because it can be used to calculate the number of articles that require manual screening by reviewers. We were concerned with the number of irrelevant articles that were incorrectly classified as relevant by machine-learning classifiers (i.e. records with an assigned probability score greater than the identified threshold score), which had to be manually filtered out by the reviewers. Precision was calculated as the proportion of retrieved articles that genuinely report a relevant study. Recall and precision were calculated using a two-by-two table that represented ‘relevant’/’irrelevant’ articles and whether they were classified correctly or incorrectly (Table 1).

TABLE 1

The 2 × 2 table from which precision and recall were calculated

The following formulas are used to calculate precision and recall:

To be useful for systematic reviews, a document classification system must save reviewers the time and effort of reading each paper. Simultaneously, the number of missed papers containing high-quality evidence must be very small. For this study, we assumed that the system needed a recall of ≥ 0.95 to identify an adequate fraction of the positive papers.^37,38 As long as the recall is at least 0.95, precision should be as high as possible.^37,38

Furthermore, the most important aspect of the system to evaluate is how much future work the reviewers could save in the process of screening abstracts and titles.³⁸ We defined work saved as the percentage of papers that meet the original search criteria but are not required to be read by the reviewers (because they have been screened out by the classifier). With a 0.95 random sampling of the data, a recall of 0.95 can be obtained, and this process would save the reviewers 5% of the time spent reading the papers. Clearly, for the classifier system to be advantageous, the amount of work saved must be greater than the amount of work saved by simple random sampling. As a result, for a given level of recall, we measure the work saved in addition to the work saved by simple sampling.^38,39 We define work saved over sampling (WSS) as follows:

where TP represents the number of true positives identified by the classifier, TN represents the number of true negatives identified by the classifier and FN represents the number of false negatives identified by the classifier. The recall is R, and the total number of samples in the test set is N. We have fixed recall at 0.95^37,38 for the current work, so WSS at 95% recall (WSS@95%) is:

Main findings

We developed five separate artificial intelligence models to predict whether or not a paper is applicable for our systematic analysis of complex interventions. The best-performing classifier exhibited excellent performance on the testing set, yielding a recall of 96.4%, as well as a precision of 99.1%, and demonstrating the potential for a workload saving of 92%. This automated approach has the greatest potential for reducing workload in conducting systematic literature reviews of complex interventions.

Comparison with previous studies

To optimise productivity, support vector machine-based natural language processing approaches have been developed to classify important medical literature papers on a variety of topics. In 2005, Aphinyanaphongs et al.⁴⁰ created the first support vector machine tool to assist in the systematic analysis of literature by defining specific documents in the field of internal medicine. Several similar methods were subsequently suggested, including one introduced by Wallace et al.⁷ that involves active learning to minimise annotation costs.⁴¹ Wallace et al.⁴² reduced the number of papers to be checked manually by approximately 50% for a systematic review. Fiszman et al.⁴³ developed a framework that uses symbolic significance analysis to classify potentially important documents for cardiovascular risk factor guidance. The recall of their system was 56% and the accuracy was 91%.⁴⁴ While most of the current approaches concentrate on clinical literature, Miwa et al.³ expanded the reach of their approach to social science literature. CNN-based natural language processing methodologies have been introduced for short-text and sentence classification.¹⁹ However, few methodologies have been introduced and evaluated for the classification of medical literature. Using the probability of bias in text classification data sets, Zhang et al.¹⁶ developed a CNN model to test the bias of the study design in the literature on randomised clinical trials. The reduction in workload ranged from 64.0% to 75.0%.

Study limitations and strengths

There are some drawbacks to our existing methods. We used only title and abstract terms, MeSH (medical subject headings) and MEDLINE publication styles as possible classification features. Additional work needs to be carried out to examine potential changes in performance, including additional features, such as full document text. Furthermore, it has recently been reported that deep-learning methods surpass conventional machine-learning algorithms in many natural language processing applications.^39,45,46 In addition to the high-precision, deep-learning methods do not rely on hand-engineered features; rather, they are able to learn the most suitable features for a specific task. Another possible limitation is the generalisability of the approach to other complex systematic reviews of other topics. Methodological research is required to assess the validity of developing high-performing classifiers for a wide range of topics. Deep learning, therefore, is potentially the leading and most promising path to natural language processing at the moment. Our reason for using this approach in our review was to efficiently increase the comprehensiveness of our searches and save workload in our update searches. CNN has been shown to be better at classifying text, where finding representative patterns is often the key to solving a text classification problem.⁴⁷ In fact, CNN architectures have shown to be better at text classification than the state-of-art algorithms in text classifications because they can find the most informative ngrams that describe a text.^18,47 ‘Traditional’ algorithms look at data based on words or features, but they do not look at the meaning of a sentence as a whole.⁴⁸ CNN, on the other hand, figures out how words are related without using a parser or a vocabulary.⁴⁹ We are aware of other methods of potentially increasing the comprehensiveness of searches efficiently, such as forward-citation searching, but the breadth and number of interventions in this review made these less attractive. In addition, we did not conduct error analysis to check the misclassified studies by the best-performing classifier. If the recommended recall threshold was not met, error analysis could have been performed, and a second model could have been built using updated training data.³⁷

Patient and public involvement

Drawing on INVOLVE’s guidance and support for best practice, we worked closely with three dedicated patient and public involvement advisors, and we invited guidance and support from our advisors during the preparatory phase of the project.

Contributions of authors

Olalekan A Uthman (https://orcid.org/0000-0002-8567-3081) (Professor, Evidence Synthesis) contributed to the protocol, study selection, data extraction, validity assessments and synthesis of the included studies. He developed the classifiers and undertook the analyses. He also contributed to the interpretation of the results and the writing of the report and had overall responsibility for the project.

Rachel Court (https://orcid.org/0000-0002-4567-2586) (Information Specialist) contributed to the protocol development, developed the search strategies, conducted a range of searches to locate studies and wrote the sections of the report relating to the literature searches. She also contributed to the protocol and interpretation of the results and commented on drafts of the report.

Jodie Enderby (https://orcid.org/0000-0002-1446-7512) (Research Associate) contributed to the protocol, study selection, validity assessments and synthesis of the included studies. She also contributed to the interpretation of the results and the writing of the report.

Lena Al-Khudairy (https://orcid.org/0000-0003-0638-583X) (Associate Professor, Evidence Synthesis) contributed to the protocol, study selection, validity assessments and synthesis of the included studies. She also contributed to the interpretation of the results and the writing of the report.

Chidozie Nduka (https://orcid.org/0000-0001-7031-5444) (Senior Research Fellow, Evidence Synthesis) contributed to the protocol, study selection, validity assessments and synthesis of the included studies. He also contributed to the interpretation of the results and the writing of the report.

Hema Mistry (https://orcid.org/0000-0002-5023-1160) (Associate Professor, Health Economics) contributed to the protocol, study selection, validity assessments and synthesis of the included studies. She also contributed to the interpretation of the results and the writing of the report.

GJ Melendez-Torres (https://orcid.org/0000-0002-9823-4790) (Professor, Evidence Synthesis) contributed to the protocol, study selection, validity assessments and synthesis of the included studies. He also contributed to the interpretation of the results and the writing of the report.

Sian Taylor-Phillips (https://orcid.org/0000-0002-1841-4346) (Professor, Evidence Synthesis) contributed to the protocol, study selection, validity assessments and synthesis of the included studies. She also contributed to the interpretation of the results and the writing of the report.

Aileen Clarke (https://orcid.org/0000-0001-8299-3146) (Professor, Evidence Synthesis) contributed to the protocol, study selection, validity assessments and synthesis of the included studies. She also contributed to the interpretation of the results and the writing of the report.

Ethics statement

This work is a systematic review of accessing, processing and analysing data that has already been published and is available to the public. As a result, no patient data were processed, and patient consent and/or registration through human research ethics committees were therefore not relevant.

Data-sharing statement

No new data have been created in the preparation of this article and therefore there is nothing available for access and further sharing. All queries should be submitted to the corresponding author.

Funding

This project was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme and will be published in Health Technology Assessment. See the NIHR Journals Library website for further project information.

Article history

The research reported in this article was funded by the HTA programme under project number 17/148/05. The contractual start date was in February 2019. This article began editorial review in December 2021 and was accepted for publication in June 2022. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and production house have tried to ensure the accuracy of the authors’ article and would like to thank the reviewers for their constructive comments on this article document. However, they do not accept liability for damages or losses arising from material published in this article.

Disclaimer

This article presents independent research funded by the National Institute for Health and Care Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, the HTA programme or the Department of Health and Social Care.

Article history

The research reported in this article was funded by the HTA programme under project number 17/148/05. The contractual start date was in February 2019. The article began editorial review in December 2021 and was accepted for publication in June 2022. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ article and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this article.