Evaluating the Effectiveness of SAST Tools: A Comparative Study on Vulnerability Detection, Reporting, and Usability

Detecting security vulnerabilities early in the software development lifecycle can significantly reduce costs, maintenance, and time. Research has shown that over 75% of security breaches stem from the software application level. Despite many methods developed to detect code vulnerabilities, effectively addressing this issue remains challenging. This study aims to evaluate the effectiveness of Static Application Security Testing (SAST) tools in identifying security vulnerabilities within source code. The evaluation focuses on three key factors: performance, reporting, and usability of three widely used SAST tools - Fortify SCA, Sparrow SAST, and PVS-Studio. To conduct the evaluation, we applied these tools to 25 Java test cases, estimating the total number of vulnerabilities detected by each tool. Statistical analysis was performed to compare the performance of the tools based on the number of vulnerabilities identified. In addition, the tools were assessed for their reporting capabilities, including the diversity and customization of report types. Finally, usability was evaluated using two well-established methods: the Heuristic Walkthrough Evaluation and the System Usability Scale (SUS). The results show that Sparrow SAST had the best detection performance, identifying more vulnerabilities than Fortify SCA and PVS-Studio. Fortify SCA, however, was superior in reporting, offering diverse and customizable options, and ranked highest in usability based on heuristic walkthrough and SUS evaluations. This study offers insights into the strengths and weaknesses of SAST tools, helping organizations choose the right tools for early vulnerability detection in software development.