Microsoft's AI Red Team Adopts Hacker Mindset to Enhance Security

Generative AI’s new capabilities come with new risks, spurring a novel approach to how Microsoft's AI Red Team works to identify and reduce potential harm, according to news.microsoft.com.

Origins of Red Teaming

The term “red teaming” was coined during the Cold War, when the U.S. Defense Department conducted simulation exercises with red teams acting as the Soviets and blue teams acting as the U.S. and its allies. The cybersecurity community adopted the language a few decades ago, creating red teams to act as adversaries trying to break, corrupt, or misuse technology — with the goal of finding and fixing potential harms before any problems emerged.

Formation of Microsoft's AI Red Team

In 2018, Siva Kumar formed Microsoft’s AI Red Team, following the traditional model of pulling together cybersecurity experts to proactively probe for weaknesses, just as the company does with all its products and services. Meanwhile, Forough Poursabzi led researchers from around the company in studies from a responsible AI lens, examining whether the generative technology could be harmful — either intentionally or due to systemic issues in models that were overlooked during training and evaluation.

Collaboration for Comprehensive Risk Assessment

The different groups quickly realized they’d be stronger together and joined forces to create a broader red team that assesses both security and societal-harm risks alongside each other. This new team includes a neuroscientist, a linguist, a national security specialist, and numerous other experts with diverse backgrounds.

Adapting to New Challenges

This collaboration marks a significant shift in how red teams operate, integrating a multidisciplinary approach to tackle the unique challenges posed by generative AI. By thinking like hackers, the team aims to identify vulnerabilities and mitigate risks before they can be exploited in real-world scenarios.

This initiative is part of Microsoft’s broader effort to deploy AI responsibly, ensuring that new capabilities do not come at the expense of safety and societal well-being.