Penetration tests are designed to identify potential gaps in an organization’s cybersecurity. With an effective penetration test comes a variety of different risks. Before engaging a penetration test provider, it is essential to understand the risks of penetration tests, how to minimize them, and why a good penetration testing firm will not be able to accept liability for actions performed in good faith.
A Good Penetration Test Carries the Potential for Damages and/or Outages
Any reputable penetration testing firm will not provide a guarantee that their services are entirely safe. Any provider that does so is likely either deceptive or using testing tools that are so ineffective as to be essentially worthless.
The reason why safety cannot be guaranteed is that many computing systems and programs are fairly unstable during normal operations. How many times have you had Microsoft Office or Excel crash and cause data loss during normal use? If these and other programs were completely reliable, Microsoft wouldn’t have bothered developing Autosave.
If these systems are so unstable during “normal” use, consider the expected impacts of the very unusual conditions that they will be subjected to during a pentesting engagement. Penetration testing is designed to identify the bugs in software that an attacker would exploit as part of their attacks. The best way to locate and determine the potential impact of these vulnerabilities is to use the same tools and techniques that a real attacker would. This poking and prodding, while carried out with the best of intentions, falls outside the definition of “normal” use for this software.
While the probability that a penetration test will cause a significant failure or damage is less than 1%, it is still possible. For this reason, when undergoing a penetration test designed to provide an accurate assessment of an organization’s systems and cyber risk, it is impossible for the testing provider to accept liability for outages and other damages caused by reasonable testing activities performed in good faith.
The Level of Risk Depends on the Services Provided
All penetration tests carry some risk of outages or other damages to the systems under test. However, not all penetration tests are created equal, and different types of tests carry varying levels of risk. Depending on the type of test performed, an organization may need to accept a higher level of risk and different types of risk.
Automated Tests are Higher Risk
One of the main determinants of risk in a penetration test is the type of penetration testing services provided. Basic tests, which rely heavily upon automation, are much riskier than realistic threat penetration testing.
Some penetration test providers rely heavily upon automated tools such as scanners and exploitation frameworks to reduce the manual work required during a test. While this may improve the speed, cost, and scalability of the test, it does so at the cost of significantly increased risk. The scripted tests performed by these tools launch an attack if a system “appears” to be vulnerable without checking for strange or risky conditions. This dramatically increases the probability that a system will experience a memory error or other issue that will cause the program to crash.
Realistic threat penetration testing, on the other hand, carries a lower level of risk because it realistically emulates what a skilled attacker would do when attempting to exploit the system. Cybercriminals, when attacking a system, are trying not to be detected and use tools and techniques designed to minimize the probability that they will be detected before they achieve their objectives. Testing driven by human talent, experience, and expertise is more likely to avoid potential damage and other outages and minimize risk than a penetration test more heavily reliant upon automation.
Basic Tests Carry Long-Term Risk
The risks associated with a penetration test are not limited to potential damages and outages. A poorly-performed penetration test also carries the potential for long-term risks to an organization.
Companies commonly undergo penetration testing to fulfill regulatory requirements and often select the most basic test available in an attempt to “check the box” for compliance. While these basic tests may earn a compliant rating, they do little to measure the organization’s true cybersecurity risk.
In the long term, this reliance upon basic tests carries significant cost to an organization. Companies like Equifax, Target, Sony, Hannaford, and the Home Depot all were tested as compliant with applicable regulations yet suffered damaging data breaches. In fact, the CEO of Target said, “Target was certified as meeting the standard for the payment card industry in September 2013. Nonetheless, we suffered a data breach.”
The ROI of good security is equal to the cost in damages of a single successful compromise. You can add the cost of all of the security technologies and testing to the cost in damages as well because those things did not prevent the breach.
Is Your Vendor Delivering Genuine Penetration Testing Services?
Determining whether or not a penetration testing vendor is offering automated or manual testing services may seem difficult. However, looking at how the provider calculates the cost of an assessment can be an easy way to accomplish this.
Any penetration testing provider will need to scope the size of the project before providing a quote. The questions that they ask and actions performed during the scoping phase can help to determine the types of services that they provide.
Vendors that provide services dependent upon automation will ask questions like “how many IP addresses do you have?” or “how many pages is your application?”. These questions are important to them since they determine the number of scans that they expect to perform.
However, these kinds of questions do not provide a realistic assessment of the complexity of the penetration testing engagement. If a customer tells a vendor that they have 10 IP addresses and a web application with 10 pages the vendor might bill them $500.00 per IP and $1,000.00 per page totaling $15,000.00. While that price sounds reasonable at face value, what happens if none of the 10 IP addresses hosts any connectable services? In workload terms that is 0 man-hours. Despite this the customer in our example would still pay $5,000.00 for 0 hours of work. Does that sound reasonable?
The inverse is also true and is the exact reason why these types of vendors rely on automated scanning (rather than manual testing) for service delivery. What happens if each of the 10 IP addresses requires 40 hours of work totaling 400 man-hours? The cost would still be $5,000.00 as quoted which means that the vendor would need to work at an effective rate of $12.50 an hour! Of course no vendor will work for that rate and so they compensate for the overage with automation but don’t tell their customer.
The same is true for web pages. It is entirely possible to have 10 static web pages that cannot be attacked because they take no input. It is also possible to have 10 web pages each of which takes significant input and could require even more than 40 hours to test per page. So, as a general rule of thumb, companies looking for penetration testing services should avoid vendors that price based on the count of targets (IP addresses, pages, etc). These vendors are generally the ones that deliver basic tests packaged as advanced tests. They also face higher liability as was demonstrated when Trustwave was sued by banks for certifying Target.
Why a Penetration Testing Firm Can’t Accept Liability
A good penetration testing firm cannot accept liability for damages and outages caused by their services. While a good, manual penetration test does everything that it can to minimize the potential for something to go wrong, unforeseen circumstances can cause unanticipated issues.
Some classic examples of penetration tests that went wrong in unexpected ways are the recent cases of Coalfire and Nissan.
In Coalfire’s case, penetration testers were hired to perform an assessment of the physical security of an Iowa courthouse building. Due to a misunderstanding in the terms of the engagement, the penetration testers were arrested and faced felony charges when found testing the security of the courthouse after hours.
The Nissan case, on the other hand, is an example of a penetration test that was commissioned under false pretenses. The VP of the company had the test performed without the knowledge of the CEO or key IT personnel. The results of the test were used to gain access to the email account of the company chairman and access data used to bring charges against him for financial misconduct.
A penetration testing vendor cannot accept liability for potential damages caused by the test because some aspects are completely outside of their control. This is why you shouldn’t be alarmed to see a release of liability statement in a penetration testing agreement and might have cause for concern if a vendor provides a contract that lacks such a clause.
