A Systematic Literature Review on Static Application Security Testing (SAST) Tools: Evaluation, Benchmarks, Challenges, and Future Directions

Context Static Application Security Testing (SAST) tools play an important role in finding software vulnerabilities during the first phase of the software security testing pipeline and development. Yet, the aptness of these tools to find vulnerabilities and their incorporation into Continuous Integration/Continuous Deployment (CI/CD) pipelines is an open research area. Objective: This research provides a systematic analysis and comparison of popular SAST tools (i.e., SonarQube, Checkmarx, and Bandit) based on their capability in vulnerability detection within CI/CD pipelines. Methodology: A systematic literature review (SLR) process was performed, making use of the methodology by Kitchenham and Charters [12]. The review process consisted of six stages: 1) defining research questions, 2) developing a search strategy, 3) selecting relevant studies, 4) assessing study quality, 5) extracting data, and 6) synthesizing the findings. Three performance metrics, including recall, precision, false positive rate, and detection accuracy, were used to evaluate the tool's performance. Results: Results revealed that SonarQube was popular and showed good performance on Java applications, Checkmarx had a higher precision but also generated more false positives, and Bandit was a highly effective tool for Python security flaws. The OWASP Benchmark was most commonly used but lacked diversity due to its real-world, imbalanced nature, whereas the Juliet Test Suite offered more extensive and exhaustive coverage. In terms of reporting evaluation metrics, the True Positive Rate (TPR) and False Positive Rate (FPR) were the most commonly used, whereas the Youden Index (YI), despite being less commonly used, provided a more balanced measure of performance. Conclusion: The report highlights the necessity of employing a holistic approach to vulnerability detection, demonstrating that no one SAST solution is best at everything. Further studies are to target these tools to improve precision and recall to reduce false positives and develop their combinatorial treatment within the current DevSecOps flow to enhance software security.