What You Need to Know About Penetration Testing Liability

What You Need to Know About Penetration Testing Liability

Penetration Testing Liability

Penetration tests are designed to identify potential gaps in an organization’s cybersecurity. With an effective penetration test comes a variety of different risks.  Before engaging a penetration test provider, it is essential to understand the risks of penetration tests, how to minimize them, and why a good penetration testing firm will not be able to accept liability for actions performed in good faith.

Pen Test Risk Reward

A Good Penetration Test Carries the Potential for Damages and/or Outages

Any reputable penetration testing firm will not provide a guarantee that their services are entirely safe.  Any provider that does so is likely either deceptive or using testing tools that are so ineffective as to be essentially worthless.

The reason why safety cannot be guaranteed is that many computing systems and programs are fairly unstable during normal operations.  How many times have you had Microsoft Office or Excel crash and cause data loss during normal use?  If these and other programs were completely reliable, Microsoft wouldn’t have bothered developing Autosave.

If these systems are so unstable during “normal” use, consider the expected impacts of the very unusual conditions that they will be subjected to during a pentesting engagement.  Penetration testing is designed to identify the bugs in software that an attacker would exploit as part of their attacks.  The best way to locate and determine the potential impact of these vulnerabilities is to use the same tools and techniques that a real attacker would.  This poking and prodding, while carried out with the best of intentions, falls outside the definition of “normal” use for this software.

While the probability that a penetration test will cause a significant failure or damage is less than 1%, it is still possible.  For this reason, when undergoing a penetration test designed to provide an accurate assessment of an organization’s systems and cyber risk, it is impossible for the testing provider to accept liability for outages and other damages caused by reasonable testing activities performed in good faith.

The Level of Risk Depends on the Services Provided

All penetration tests carry some risk of outages or other damages to the systems under test.  However, not all penetration tests are created equal, and different types of tests carry varying levels of risk.  Depending on the type of test performed, an organization may need to accept a higher level of risk and different types of risk.

Automated Tests are Higher Risk

One of the main determinants of risk in a penetration test is the type of penetration testing services provided.  Basic tests, which rely heavily upon automation, are much riskier than realistic threat penetration testing.

Some penetration test providers rely heavily upon automated tools such as scanners and exploitation frameworks to reduce the manual work required during a test.  While this may improve the speed, cost, and scalability of the test, it does so at the cost of significantly increased risk.  The scripted tests performed by these tools launch an attack if a system “appears” to be vulnerable without checking for strange or risky conditions.  This dramatically increases the probability that a system will experience a memory error or other issue that will cause the program to crash.

Realistic threat penetration testing, on the other hand, carries a lower level of risk because it realistically emulates what a skilled attacker would do when attempting to exploit the system.  Cybercriminals, when attacking a system, are trying not to be detected and use tools and techniques designed to minimize the probability that they will be detected before they achieve their objectives.  Testing driven by human talent, experience, and expertise is more likely to avoid potential damage and other outages and minimize risk than a penetration test more heavily reliant upon automation.

Basic Tests Carry Long-Term Risk

The risks associated with a penetration test are not limited to potential damages and outages.  A poorly-performed penetration test also carries the potential for long-term risks to an organization.

Companies commonly undergo penetration testing to fulfill regulatory requirements and often select the most basic test available in an attempt to “check the box” for compliance.  While these basic tests may earn a compliant rating, they do little to measure the organization’s true cybersecurity risk.

In the long term, this reliance upon basic tests carries significant cost to an organization.  Companies like Equifax, Target, Sony, Hannaford, and the Home Depot all were tested as compliant with applicable regulations yet suffered damaging data breaches.  In fact, the CEO of Target said, “Target was certified as meeting the standard for the payment card industry in September 2013. Nonetheless, we suffered a data breach.”

The ROI of good security is equal to the cost in damages of a single successful compromise.  You can add the cost of all of the security technologies and testing to the cost in damages as well because those things did not prevent the breach.

Is Your Vendor Delivering Genuine Penetration Testing Services?

Determining whether or not a penetration testing vendor is offering automated or manual testing services may seem difficult.  However, looking at how the provider calculates the cost of an assessment can be an easy way to accomplish this.

Any penetration testing provider will need to scope the size of the project before providing a quote.  The questions that they ask and actions performed during the scoping phase can help to determine the types of services that they provide.

Vendors that provide services dependent upon automation will ask questions like “how many IP addresses do you have?” or “how many pages is your application?”.  These questions are important to them since they determine the number of scans that they expect to perform.

However, these kinds of questions do not provide a realistic assessment of the complexity of the penetration testing engagement.  If a customer tells a vendor that they have 10 IP addresses and a web application with 10 pages the vendor might bill them $500.00 per IP and $1,000.00 per page totaling $15,000.00.  While that price sounds reasonable at face value, what happens if none of the 10 IP addresses hosts any connectable services?  In workload terms that is 0 man-hours.   Despite this the customer in our example would still pay $5,000.00 for 0 hours of work.  Does that sound reasonable?

The inverse is also true and is the exact reason why these types of vendors rely on automated scanning (rather than manual testing) for service delivery.  What happens if each of the 10 IP addresses requires 40 hours of work totaling 400 man-hours?  The cost would still be $5,000.00 as quoted which means that the vendor would need to work at an effective rate of $12.50 an hour!  Of course no vendor will work for that rate and so they compensate for the overage with automation but don’t tell their customer.

The same is true for web pages.  It is entirely possible to have 10 static web pages that cannot be attacked because they take no input.  It is also possible to have 10 web pages each of which takes significant input and could require even more than 40 hours to test per page.  So, as a general rule of thumb, companies looking for penetration testing services should avoid vendors that price based on the count of targets (IP addresses, pages, etc).  These vendors are generally the ones that deliver basic tests packaged as advanced tests.  They also face higher liability as was demonstrated when Trustwave was sued by banks for certifying Target.

Why a Penetration Testing Firm Can’t Accept Liability

A good penetration testing firm cannot accept liability for damages and outages caused by their services.  While a good, manual penetration test does everything that it can to minimize the potential for something to go wrong, unforeseen circumstances can cause unanticipated issues.

Some classic examples of penetration tests that went wrong in unexpected ways are the recent cases of Coalfire and Nissan.

In Coalfire’s case, penetration testers were hired to perform an assessment of the physical security of an Iowa courthouse building.  Due to a misunderstanding in the terms of the engagement, the penetration testers were arrested and faced felony charges when found testing the security of the courthouse after hours.

The Nissan case, on the other hand, is an example of a penetration test that was commissioned under false pretenses.  The VP of the company had the test performed without the knowledge of the CEO or key IT personnel.  The results of the test were used to gain access to the email account of the company chairman and access data used to bring charges against him for financial misconduct.

A penetration testing vendor cannot accept liability for potential damages caused by the test because some aspects are completely outside of their control.  This is why you shouldn’t be alarmed to see a release of liability statement in a penetration testing agreement and might have cause for concern if a vendor provides a contract that lacks such a clause.

Blog Posts

Karen Huggins

Chief Financial, HR and Admin Officer
Karen joined the Netragard team in 2017 and oversees Netragard’s financial, human resources as well as administration functions. She also provides project management support to the operations and overall strategy of Netragard.
Prior to joining Netragard, she worked at RBC Investor Services Bank in Luxembourg in the role of Financial Advisor to the Global CIO of Investor Services, as well as several years managing the Financial Risk team to develop and implement new processes in line with regulatory requirements around their supplier services/cost and to minimize the residual risk to the organization.
With over 20 years of experience in finance with global organizations, she brings new perspective that will help the organization become more efficient as a team. She received her Bachelor of Finance from The Florida State University in the US and her Master of Business Administration at ESSEC Business School in Paris, France.

Philippe Caturegli

Chief Hacking Officer
Philippe has over 20 years of experience in information security. Prior to joining Netragard, Philippe was a Senior Manager within the Information & Technology Risk practice at Deloitte Luxembourg where he led a team in charge of Security & Privacy engagements.

Philippe has over 10 years of experience in the banking and financial sector that includes security assessment of large and complex infrastructures and penetration testing of data & voice networks, operating systems, middleware and web applications in Europe, US and Middle East.

Previously, Philippe held roles within the information system security department of a global pharmaceutical company in London. While working with a heterogeneous network of over 100,000 users across the world and strict regulatory requirements, Philippe gained hands-on experience with various security technologies (VPN, Network and Application Firewalls, IDS, IPS, Host Intrusion Prevention, etc.)

Philippe actively participates in the Information Security community. He has discovered and published several security vulnerabilities in leading products such as Cisco, Symantec and Hewlett-Packard.

He is a Certified Information Systems Security Professional (CISSP), Certified Ethical Hacker (CEH), PCI Qualified Security Assessors (PCI-QSA), OSSTMM Professional Security Analyst (OPSA), OSSTMM Professional Security Tester (OPST), Certified in Risk and Information Systems Control (CRISC)and Associate Member of the Business Continuity Institute (AMBCI).

Adriel Desautels

Chief Technology Officer
Adriel T. Desautels, has over 20 years of professional experience in information security. In 1998, Adriel founded Secure Network Operations, Inc. which was home to the SNOsoft Research Team. SNOsoft gained worldwide recognition for its vulnerability research work which played a pivotal role in helping to establish today’s best practices for responsible disclosure. While running SNOsoft, Adriel created the zeroday Exploit Acquisition Program (“EAP”), which was transferred to, and continued to operate under Netragard.
In 2006, Adriel founded Netragard on the premise of delivering high-quality Realistic Threat Penetration Testing services, known today as Red Teaming. Adriel continues to act as a primary architect behind Netragard’s services, created and manages Netragard’s 0-day Exploit Acquisition Program and continues to be an advocate for ethical 0-day research, use and sales.
Adriel is frequently interviewed as a subject matter expert by media outlets that include, Forbes, The Economist, Bloomberg, Ars Technica, Gizmodo, and The Register. Adriel is often an invited keynote or panelist at events such as Blackhat USA, InfoSec World, VICELAND Cyberwar, BSides, and NAW Billion Dollar CIO Roundtable.