Generative AI in offensive security: Capabilities, challenges, and risks

Large language models (LLMs) and generative AI techniques have significantly impacted offensive cybersecurity. This chapter provides a clear overview of how attackers misuse generative AI in four key areas: LLM-powered phishing and social engineering, LLM-based malware and ransomware generation, generative adversarial networks (GAN)-based evasion, and autonomous LLM agents for exploitation and red teaming. Based on recent research, the chapter explores capabilities available to attackers, challenges posed to defenders, and associated risks. Key findings include the use of prompt-based jailbreaks to bypass AI safety measures, combined LLM and GAN pipelines for producing evasive malware, polymorphic attacks that constantly change to avoid detection, and autonomous AI agents capable of planning and executing complex cyberattacks.