metaextractoR: An R Package for Streamlined Data Extraction in Systematic Reviews and Meta-Analyses

👩🏻‍💻 Danyang Dai @ The University of Queensland

• danyang.dai@uq.edu.au
• https://dai.netlify.app/
• @DanyangDai
• @danyang-dai-7529b4152

8th of April SSA Di Cook Award

Hello everybody. Thank you for coming today.I am Daidai and I am a third year PhD student from the University of Queensland. I’m in the final stages of my PhD, graduating mid-next year, and I’m starting to look ahead to my next chapter.

Today, I will be showcasing my very first R package that utilise Large Language Models to extract data for systematic review and meta-analysis.

Acknowledgement of Country

I acknowledge the Traditional Owners and their custodianship of the lands on which we meet.
We pay our respects to their Ancestors and their descendants, who continue cultural and spiritual connections to Country.

Before I start my presentation, I would like to acknowledge the Traditional Owners and their custodianship of the lands on which we meet. We pay our respects to their Ancestors and their descendants, who continue cultural and spiritual connections to Country.

What is Systematic Review?

• A structured, comprehensive method for identifying and summarising all available evidence on a specific research question.

• Systematic reviews and meta-analyses are pillars of evidence-based medicine.

Systematic review and meta-analysis might not be a familiar term for many of you. They are widely regarded as the “gold standard” for generating high-quality evidence to inform clinical practice and policy decisions.

The steps includes topic selection, conceptualised and create protocol, searching for studies,screening for studies,appraise studies, data extraction, synthesis and interpret results.

This might still be quite abstract to you, so let me show you an example.

Example of a sysrematic review and meta-analysis

In this systematic review and meta-analysis, we set out to understand the true prevalence of COVID-associated acute kidney injury. Yes—you heard that right. Acute kidney injury, or AKI, is actually one of the more common complications seen in hospitalised COVID patients. It refers to a sudden loss of kidney function, typically detected through changes in serum creatinine.

What sparked this project was a simple question: why do published AKI prevalence vary so dramatically? When I looked into the literature, I found reported prevalence ranging anywhere from 1% to 70%. So, we undertook this systematic review and meta-analysis to bring clarity to the global picture—examining where and why COVID-associated AKI rates differ, and uncovering geographic and socioeconomic patterns that may explain these disparities.

In this systematic review and meta-analysis, we observed a clear pattern: COVID-associated AKI higher in high-income countries, and substantially lower in low- and lower-middle-income countries. For more information, please see the paper.

To arrive at this finding, we extracted data from over 300 published studies using Covidence. And in case you’re not familiar with Covidence, let me show you a quick example of how it works. The most important information we need was the number of patients included in each study, the number of AKI patients, clincial settings on wheather this is an ICU only cohort or not.

Data extraction - Covidence

This is a very manual process as you can see, it requires a lot of human effort.

For a more robust process, dual extraction is required sometime. This means that this process will be done twice by two individuals.

Design of metaextractoR

Because data extraction can be incredibly labour-intensive, we wanted a way to streamline the process. So we built an R package called metaextractoR, which uses large language models to help automate parts of data extraction and reduce the manual workload.

The workflow is designed around the Cochrane systematic review guidelines, and importantly, it keeps humans in the loop at every step to make sure the output stays accurate and reliable.

We’ve also created three Shiny apps that give users an easy, intuitive interface, so anyone can extract data with the help of LLMs—without much coding background.

The first step is load in the list of abstracts included for your systematic reivew and meta-analysis.

We will use the buidin function in this package to separate the list of abstracts into training and testing set.

The training set will be used in the first and second apps for prompt engineering and model selection, the testing set will be use for bath processing and validation.

Demo

Accurate Evaluation

For all LLMs, the model temperature was set to 0.

Model	No. Patients	No. AKI	ICU Only	Mean Age	Median Age	Start Date	End Date
llama	0.85	0.86	0.79	0.73	0.45	0.44	0.47
gemma	0.68	0.53	0.89	0.73	0.64	0.40	0.49
medllama	0.66	0.51	0.32	0.73	NA	0.51	0.49
mistral	0.76	0.49	0.53	0.73	0.64	0.42	0.51
nuextract	0.85	0.30	0.47	0.64	0.45	0.28	0.35
deepseek	0.42	0.51	0.37	0.73	0.55	0.51	0.53

Total Accuracy of the LLMs and Run Time

Model	Accuracy	Run time
llama3.1	0.693	9.21 mins
gemma3	0.594	9.02 mins
medllama2	0.533	9.61 mins
mistral	0.594	11.55 mins
nuextract	0.529	6.91 mins
deepseek-r1	0.483	11.48 mins

Furture work

• Expand the extraction to text for qualitative analysis

• Allow full-text data extraction

• Allow other closed-sourced LLMs

Acknowledgement

Dr. Emi Tanaka emi.tanaka@anu.edu.au

Prof. Jason Pole j.pole@uq.edu.au

Github and Installation

• https://danyangdai.github.io/metaextractoR/
• Collaborate at: https://github.com/DanyangDai/metaextractoR

Stay in touch

danyan.dai01@gmail.com

https://www.linkedin.com/in/danyang-dai-7529b4152/