Key Takeaways:
Most penetration testing hasn’t improved security outcomes because the market optimized for cost, not quality. The result is a widespread “checkbox” exercise where automated scans and templated reports are sold as adversarial testing – leaving organizations with the appearance of assurance rather than actual risk reduction.
AI is not a fix for a broken testing model. Layering AI on top of weak methodologies only accelerates low-value output. Real penetration testing still depends on human-led, creative, context-aware adversarial thinking that cannot be replicated by automation alone.
Organizations can materially improve security without bigger budgets by focusing on fundamentals: knowing what real testing looks like, prioritizing risk over volume of findings, and adopting a breach-ready mindset that assumes compromise and plans for it.
It’s no secret we feel the industry has done the customer a disservice. For years, firms have gotten away with passing off automated vulnerability scanners and AI-generated output as real penetration testing — while actual quality standards for the discipline have never materialized. The customer pays. The customer gets a polished report. And the customer is no more secure than before.
So when an opportunity came to share those views with like-minded veterans who genuinely get it, our founder and CEO Adriel Desautels jumped at it.
ESW #465: Fixing Pentesting
Adriel recently joined hosts Adrian Sanabria and Tyler Shields on Enterprise Security Weekly (ESW #465), the long-running security podcast on SC Media, for a frank and wide-ranging conversation about the state of penetration testing.
The segment is titled “Fixing Pentesting” and it pulls no punches.
The Core Problem: 20 Years of Not Moving the Needle
Penetration testing has been a fixture of the security industry for two decades. And yet, by almost any honest measure, it hasn’t fundamentally improved the security posture of most organizations. Why?
The answer lies in how the market evolved. Competitive pressure pushed pricing down. Lower pricing forced shortcuts. Shortcuts meant automated tools and templated reports wearing the costume of expert human analysis. Over time, many customers lost the ability to distinguish genuine adversarial testing from a scanner output dressed up in a binder.
The result: organizations spend real money on a ritual that too often provides the appearance of security assurance without the substance.
Why AI Won’t Save a Broken Model
The industry’s current answer to the problem is to bolt AI onto existing processes. The pitch is compelling on the surface — more speed, more scale, more findings. But if the underlying methodology is broken, AI amplifies the problem rather than solving it.
Real penetration testing requires human creativity, adversarial thinking, and deep contextual judgment. An attacker exploiting your organization doesn’t follow a script. Your pen test shouldn’t either. AI tools can play a role in supporting skilled practitioners, but they are not a substitute for the human expertise at the center of genuine adversarial assessment.
Practical Steps Organizations Can Take Without Breaking the Budget
The conversation wasn’t just a diagnosis. Adriel shared concrete, accessible recommendations for organizations looking to meaningfully strengthen their security posture — none of which require enterprise-scale budgets.
Some of the practical guidance covered:
- Understanding what a real penetration test should look like — so you can evaluate vendors honestly and ask the right questions before you sign a contract
- Risk-based prioritization — focusing limited security resources where actual exposure exists, rather than chasing a never-ending list of scanner findings
- Breach-ready thinking — building a posture that assumes compromise will happen and plans accordingly, rather than betting everything on prevention
Additional resources shared during the segment include an HBR piece on how boards are falling short on cybersecurity, and Netragard’s own guide to what penetration testing actually is (and isn’t).
Listen to the Full Episode
If you work in security — whether you’re a CISO, a practitioner, or an executive trying to make sense of your organization’s risk — this conversation is worth your time.
FAQ
What’s wrong with most penetration testing today?
Many engagements rely heavily on automated scanning and templated reporting, which creates the illusion of security without delivering meaningful insight into real-world risk.
Will AI improve penetration testing quality?
Not by itself. AI can enhance efficiency, but when applied to an already flawed methodology, it tends to scale low-value results rather than replace the need for skilled human adversarial testing.
How can organizations get more value from penetration testing?
Focus on evaluating vendors for real adversarial capabilities, prioritize risks that matter instead of chasing every finding, and adopt a breach-ready mindset that assumes attackers will succeed and plans accordingly.
