“An AI that could design novel biological pathogens. An AI that could hack into computer systems. I think these are all scary.” - Sam Altman , CEO - Open AI.
AI is the hot topic, no doubt… but beneath this boom lies a
layer of underlying security risks waiting to be hacked. Like Naval says - before
you can break the system, you gotta master it first.
As I delved deeper into AI security, I began exploring
articles and hands-on playgrounds through good old Google ways. The more I
learned, the more confident I felt - but I was still seeking a real challenge
to truly test my skills. That’s when I came across the Certified AI/ML
Pentester (C-AI/MLPen) by The SecOps Group. In this blog, I’d love to share my
journey, learnings, and overall experience with you all.
TL;DR: This exam isn’t a walk in the park. It won’t teach
you “How to?” you’re expected to come
prepared. Learn the concepts first, then challenge yourself with the exam. I’ve
included some helpful resources you can use to practice beforehand. The exam
experience itself is smooth, and you get + one attempt - so take a deep breath
and go for it. If you are prepared๐
Additionally, this blog is not intended to explain how to perform an
AI/ML pentest.
The result of all the hard work in this course looks like this! :-)
๐ง Part 1: Preparation –
What Should You Study?
Before diving into the exam, it's essential to know what you're signing up for. The exam portal provides a well-structured syllabus that outlines the key areas you’ll be tested on. This helps set your direction and gives clarity on what to expect.
The topics are spread across AI/ML attack surfaces, prompt
injection, output handling vulnerabilities, chaining attacks, and more — so
make sure to read the syllabus thoroughly. Understanding the core concepts will
go a long way in helping you connect the dots during practical challenges.
๐งช Part 2: Is There a
Playground to Practice On?
Yes! And honestly, practice is the game-changer here. You don't want to go into this exam cold.
๐น 1. Web LLM Attacks byPortSwigger
This one’s a goldmine. I spent a good amount of time going
through it thoroughly and practicing the hands-on labs. It covers a variety of
real-world LLM vulnerabilities like:
- Exploiting LLM APIs with excessive agency
- Chaining vulnerabilities in LLM APIs
- Indirect prompt injection
- Insecure output handling
You’ll learn not just the what, but the how behind these
vulnerabilities - and trust me, that’s key.
๐น 2. Gandalf Lab – Your AI Boss Fight
This one's just fun. Gandalf challenges you with increasingly clever prompt injection puzzles - like trying to trick an LLM that's actively resisting you.
⚔️ Try to solve all exercises - they sharpen your creative thinking and simulate what you'll face in a real-world AI pentest.
๐น 3. More Study Materials
(Credit to the Awesome Creators)
Don’t stop at just one or two labs. Here are some handpicked
resources that helped me strengthen my understanding:
- Prompt Injection – LearnPrompting
- Adversarial Prompting in LLMs
- Prompt injection explained, with video, slides, and a transcript by Simon Willison
- Exploring Prompt Injection Attacks – NCC Group
- OWASP Top 10 for LLM Applications (2025)
- MITRE ATLAS Matrix
- Real World LLM Exploits – Lakera
- AI Vulnerability Deep Dive – Bugcrowd
- LLM Top 10 (2023–24)
- Prompt Hacking & Misuse – Unite.AI
You can also explore this awesome GitHub repo with hands-on
vulnerable LLM apps:
๐ Vulnerable LLM Applications (OWASP)
⏳ Part 4: Exam Time!
⏱️ Duration: 4 hours and 15 minutes
๐ Mode: 100% Online & On-Demand - take it from anywhere
✅ Format: A mix of practical
scenarios where you’ll need to spot, exploit, and capture flags from various
AI/ML-based vulnerabilities
๐ฏ Part 5: What’s the Pass
Criteria?
To pass:
You need to score 60% or more / Capture 6 out of 8 total flags.
Solve multiple challenges and identify vulnerabilities
The exam is not beginner-friendly, but if you’re prepared and you’ve gone through the practice labs, it becomes a rewarding and achievable milestone.
Conclusion