An exploratory study on effectiveness of GPT-4o in conducting sub-tasks of systematic literature reviews

Context: In recent years, advanced Artificial Intelligence (AI) tools, such as Generative Pretrained Transformers (GPTs), have shown remarkable potential across diverse application areas, including academic research. One of the promising applications of using these tools for conducting Systematic Literature Reviews (SLRs), which are effort intensive and time-consuming process in software engineering research. Objective: This study investigates the use of GPT-4o to support SLR activities in the context of software engineering. Methods: We employed a mixed research method to explore the significance of GPT-4o in conducting sub-SLR activities, such as formulating research questions, generating Boolean strings, and performing quality assessments. Four benchmark SLR studies were selected and used to replicate the SLR activities using GPT-4o, which were further assessed by six domain experts. Results: In the results, GPT-4o demonstrated consistency in generating Boolean search strings, identifying themes, and quality assessment criteria across all the benchmark studies, and showed close alignment with expert evaluations. The model effectively automated key SLR activities, including literature search string formulation, validation of primary studies, quality assessment, thematic analysis, and taxonomy development. However, GPT-4o was less accurate in identifying relevant studies across databases, occasionally missing domain-specific literature. Conclusions: The findings indicate that GPT-4o can effectively support several SLR tasks in software engineering when integrated within a domain expert-in-the-loop workflow. However, the study evaluates selected activities under controlled settings rather than a full end-to-end SLR execution. Future research should extend this evaluation to additional datasets and alternative LLMs to further assess generalizability.