Your AI Worked 59 Times. What About the 60th? - DevSec Station | Lyssna här

What happens when your AI assistant does exactly what you expect 59 times... and then does something completely bananas on the 60th?

While teaching a secure coding class, I watched nearly 60 students use the same AI assistant, programming language, and prompts. For 59 of them, everything went as expected.

But one student's AI-generated Python code included instructions to suppress linting checks, followed by code that leaked secrets to an external file.

The AI hadn't escaped a sandbox. It wasn't even an agent. It was simply generating code within the boundaries we'd given it.

And that's what makes the story so interesting.

Welcome to Season 2 of DevSec Station! In this episode, Tanya Janca explores why AI's unpredictable behaviour becomes a security problem when we give it access to files, credentials, tools, infrastructure, and deployment capabilities.

We'll discuss:

- Why successful AI runs don't guarantee the next run will be safe.
- Why better prompts aren't a substitute for security controls.
- Least privilege, independent security checks, and human approvals.
- How to limit the blast radius when an AI system does something unexpected.

** Your one thing to do this week **
Pick an AI assistant or agent you're using and ask yourself: What's the worst thing it's technically capable of doing with its current permissions?

Not what you've asked it to do. What it can *actually* do.

Because we don't need AI systems to be perfectly predictable. We need to engineer systems that can safely survive a surprise.

DevSec Station is a podcast by Tanya Janca (SheHacksPurple), focused on short, practical lessons that help software developers build more secure software.

Follow Tanya:

Rss