Whistleblower Series – Don’t be naive, take the time to read and understand the proposal.

In our last whistleblower article, we showed that the vast majority of Penetration Testing vendors don’t actually sell Penetration Tests. We did this by deconstructing pricing methodologies and combining the results with common sense. We’re about to do the same thing to the industry average Penetration Testing proposal. Only this time we’re not just going to be critical of the vendors, we’re also going to be critical of the buyers.
A proposal is a written offer from seller to buyer that defines what services or products are being sold. When you take your car to the dealer, the dealer gives you a quote for work (the proposal). That proposal always contains an itemized list for parts and labor as well as details on what work needs to be done. That is the right way to build a service-based proposal.

The industry average Network Penetration Testing proposal fails to define the services being offered. Remember, the word ‘define’ means the exact meaning of something. When we read a network penetration testing proposal and we have to ask ourselves “so what is this vendor going to do for us?” then the proposal has clearly failed to define services.

For example, just recently we reviewed a proposal that talked about “Ethos” and offered optional services called “External Validation” and “External Quarterlies” but completely failed to explain what “External Validation” and “External Quarterlies” were. We also don’t really care about “Ethos” because it has nothing to do with the business offering. Moreover, this same proposal absolutely failed to define methodology and did not provide any insight into how testing would be done. The pricing section was simply a single line item with a dollar value, it wasn’t itemized. Sure the document promised to provide Penetration Testing services, but that’s all it really said (sort of).

This is problematic because Penetration Testing is a massively dynamic service that contains a potentially infinite amount of techniques (attacks and tests) for penetration attempts. Some of those techniques are higher threat than others; some are even higher risk than others. If a proposal doesn’t define the tests that will be done, how they will be done, what the risks are, etc.,  then the vendor is free to do whatever they want and call it a day. Most commonly this means doing the absolute minimum amount of work while making it look like a lot.

Here’s some food for thought…

Imagine that we are a bulletproof vest Penetration Testing Company. It’s our job to test the effectiveness of bulletproof vests for our customers so that they can guarantee the safety of their buyers. We deliver a proposal to a customer that is the same quality as the average Network Penetration Testing proposal and our customer signs the proposal.

A week later, we receive a shipment of vests for testing. We hang those vests on dummies made up of ballistics gel in our firing range. We then take our powerful squirt guns, stand ten feet down range and squirt away. After the test is complete, we evaluate the vests and determine that they were not penetrated and so passed the Penetration Test. Our customer hears the great news and begins selling the vest on the open market.

In the scenario above, both parties are to blame. The customer did not do their job because they failed to validate the proposal, to demand clear definitions, to assess the testing methodology, etc. Instead they naively trusted the vendor. The vendor failed to meet their ethical responsibilities because they offered a misleading and dishonest service that would do nothing more than promote a false sense of security. In the end, the cost in damages (loss of life) will be significantly higher than the cost of receiving genuine services. In the end, the customer will suffer as will their own customers.

Unfortunately, this is what is happening with the vast majority of Network Penetration Tests. Vendors are perceived as experts by their customers and are delivering proposals like the ones described above. Customers then naively evaluate proposals assuming that all vendors are created equal and make buying decisions based largely on cost. They receive services (usually not a genuine penetration test), put a check in the box and move onto the next task. In reality, the only thing they’ve bought is a false sense of security.

How do we avoid this?

While we can’t force Network Penetration Testing firms to hold themselves to a higher standard, their customers can. If customers took the time to truly evaluate Network Penetration Testing proposals (or any proposal for that matter) then this problem would be eradicated. The question is do customers really want high quality testing or do they just want a check in the box? In our experience, both types of customers exist but most seem to want a genuine and high-quality service.

Here are a few things that customers can do to hold their Network Penetration Testing vendor to a higher standard.

  • Make sure the engagement is properly scoped (we discussed this in our previous article)
  • Make sure the proposal uses terms that are clearly defined and make sense. For example, we saw a proposal just one week before writing this article that was for “Non-intrusive Network Penetration Testing.” Is it possible to penetrate into something without being intrusive? No.
  • Make sure that the proposal defines terms that are unique to the vendor. For example, the proposal that we mentioned previously talks about “External Quarterlies” but fails to explain what that means. Why are people signing proposals that make them pay for an undefined service? Would you sign it if it had a service called “Goofy Insurance”?
  • Make sure the vendor can explain how they came to the price points that are reflected in the proposal. Ask them to break it down for you and remember to read our first article so that you understand the differences between count based pricing (wrong) and attack surface based pricing (right).
  • (We’ll provide more points in the next article).

As the customer, it is up to you to hold a vendor’s feet to the fire (we expect it). When you purchase poor quality services that are mislabeled as “Penetration Tests” then you are enabling the snake-oil vendors to continue. This is a problem because it confuses those who want to purchase genuine and high-quality services. It makes their job exceedingly difficult and in some cases causes people to lose faith in the Network Penetration Testing industry as a whole.

If you feel that what we’ve posted here is inaccurate and can provide facts to prove the inaccuracy then please let us know. We don’t want to mislead anyone and will happily modify these entries to better reflect the truth.

How to find a genuine Penetration Testing firm

There’s been a theme of dishonesty and thievery in the Penetration Testing industry for as long as we can remember.  Much in the same way that merchants sold “snake-oil” as a cure-all for what ails you, Penetration Testing vendors sell one type of service and brand it as another thus providing little more than a false sense of security.  They do this by exploiting their customers lack of expertise about penetration testing and make off like bandits.  We’re going to change the game; we’re going to tell you the truth.

Last week we had a new financial services customer approach us.  They’d already received three proposals from three other well-known and trusted Penetration Testing vendors. When we began to scope their engagement we quickly realized that the IP addresses that they’d been providing were wrong.  Instead of belonging to them they belonged an e-commerce business that sold beer-making products!  How did we catch this when the other vendors didn’t?  Simple, we actually take the time to scope our engagements carefully because we deliver genuine Penetration Testing services.

Most other penetration testing vendors do what is called count based pricing which we think should be a major red-flag to anyone.  Count based pricing simply says that you will pay X dollars per IP address for Y IP addresses. If you tell most vendors that you have 10 IP addresses they’ll come back and quote you at around $5,000.00 for a Penetration Test ($500.00 per IP). That type of pricing is not only arbitrary but is fraught with serious problems. Moreover, it’s a solid indicator that services are going to be very poor quality.

Scenario 1: The Overcharge (Too much for too little)

If you have 10 IP addresses but none of those 10 IP addresses are running any connectable services then there’s zero seconds worth of work to be done.  Do you really want to pay $5,000.00 for zero seconds worth of work?  Moreover, is it ethical for the vendor to charge you $5,000.00 for testing targets that are not really testable? While we don’t think its ethical we’ve seen many, many vendors do this very thing.

Scenario 2: The Undercharge (Too little money for too much work)

What if those 10 IP addresses were serving up medium complexity web applications? Lets assume that each web application would take 100 hours to test totaling 1,000 hours of testing time (not including the reporting, presentation, etc.).  If you do the math then that equates to an absolutely absurd hourly rate of $5.00 per hour for the Penetration Tester!  Of course, no penetration tester is going to work for that much money so what are you really paying $5,000.00 for?
Well, lets assume that the very-low-end cost of a penetration tester is around $60.00 per hour (its actually higher than that).  In order to deliver 1,000 hours of work at $5,000.00 the test would need to be 92.7% automated resulting in an exact hourly rate of $60.24 per hour. Do you really want to pay $5,000.00 for a project that is 92.7% automated? Moreover, is that even a Penetration Test?

The terms Penetration Test and Vulnerability Scan only have one correct definition.  The definition of Penetration Test is a test that is designed to identify the presence of points where something can make its way into or through something else.  Penetration Tests are not assessments (best guesses) of any kind.  A Penetration Test either successfully penetrates or it does not not, there is no grey zone; there are no false positives.  (This is why we can guarantee the quality of our penetration testing services.)

A Vulnerability Assessment on the other hand is a best guess or an educated guess as to how susceptible something is to risk or harm.  Because it is an assessment and not  a test there is room for error (guessing wrong) and so false positives should be expected.  A Vulnerability Scan is similar to a Vulnerability Assessment only instead of a human doing the guess work a computer program (with a much higher margin of error) does the guess work.
So, if 92.7% of a service is based on Vulnerability Scanning then how is it that Penetration Testing vendors can label such a service a Penetration Test?  They should call it what it really is, which is a Vetted Automated Vulnerability Scan; and a Vetted Automated Vulnerability Scan is about as effective as Penetration Testing a bulletproof vest with a squirt gun.  We’re not sure about you but we wouldn’t want to wear that vest into battle. These types of services provide little more than a false sense of security.

Back on track…
The question becomes how should a vendor price their services and build their proposals? While we won’t disclose our methodology here because we don’t want to enable copycats, we will provide you with some insight through an analogy.
Your car breaks down and you call a random mechanic.  You tell the mechanic what sort of car you drive and how many miles are on it but never provide any details as to what happened.  The mechanic then quotes you $300.00 to fix your car (without ever really diagnosing it) and $50.00 for a tow.  Would you bring your car to that mechanic? How can he afford to fix your car for $300.00 and make a profit?  To accomplish that must have an arsenal of junk parts gnome slaves working for peanuts, right?  Would you really trust the quality of his work? That is count based pricing.

Fortunately automobile mechanics (most of them anyway) are more ethical than that (and gnome slaves don’t really exist anyway).  Most of them won’t deliver a quote until after they’ve evaluated your car and successfully diagnosed the problem.  Once diagnosed they’ll provide you with an itemized quote that includes parts, labor, taxes, and a timeframe for service delivery.  They won’t negotiate much on price because in most cases you are getting what you pay for.

Genuine Penetration testing vendors are no different than genuine mechanics. All Penetration Testing vendors should be held to the same standard (including us).  What you pay for in services should be a direct reflection the amount of work that needs to be done. The workload requirement should be determined through a careful, hands-on assessment. This means that when pricing is done right there is no room to adjust pricing other than to change workload. Any vendor that offers a lower price if you close before the end of the month is either offering arbitrary pricing or they are padding their costs.

What if a vendor truly needs to discount services? When we deliver Penetration Testing services we don’t charge for automation.  If we automate 10% then our services are discounted 10%.  If we automate 100% then our services are delivered to our customers free of charge. (Yes, that’s right, we’ll scan you for free while other vendors might charge thousands of dollars for that). Why charge for automation when it takes less than 3 minutes of our time to kick off a scan?  Just because we can charge for something doesn’t mean that it’s ethically right.

Anyway, this article is one of many to come in our whistle blower series. Please feel free to share or contact us with any questions.

If you feel that what we’ve posted here is inaccurate and can provide facts to prove the inaccuracy then please let us know.  We don’t want to mislead anyone and will happily modify these entries to better reflect the truth.