Do Systematic Literature Reviews Outperform Informal Literature Reviews in the Software Engineering Domain? An Initial Case Study

The systematic literature reviews (SLRs) are a formally planned approach in finding, evaluating and summarising all available evidence on a specific research question. The objective of this paper is to compare formal SLRs and informal literature reviews in software engineering. For this purpose, a SLR has been conducted to compare the results with a previously conducted informal literature review of software process improvement success factors. Previous research using an informal literature review approach found 47 articles via the snow balling technique. For formal literature reviews, the SLR was conducted by applying customised search strings covering the time to 30 June 2004 (this is the deadline of the informal literature review). In total, 38 relevant articles were identified via the formal literature review. The results show that the data extraction process in the formal literature review enabled more success factors to be extracted (i.e. 34) than the informal literature review (i.e. 18). In the formal literature review, the publication inclusion and exclusion criteria and selecting primary studies helped in identifying the right list of publications. A real challenge in the formal literature review was to define a search string. In general it was observed that the SLR methodology is better than the informal literature review with respect to the planning for literature review, the design of search string, sources to be searched, publication inclusion and exclusion criteria, publication quality assessment and the data extraction process. However, the informal literature review has identified more articles than the formal literature review. Snow balling technique should be used with the formal literature review in order to identify all the relevant articles.