Bug Bounty companies (often called crowd sourced penetration tests) are all the hype. The primary argument for using their services is that they provide access to a large crowd of testers, which purportedly means that customers will always have a fresh set of eyes looking for bugs. They also argue that traditional penetration testing teams are finite and, as a result, tend to go stale in terms of creativity, depth, and coverage. While these arguments seem to make sense at face value, are they accurate?
The first thing to understand is that the quality of any penetration test isn’t determined by the volume of potential testers, but instead by their experience, talent, and overall capabilities. A large group of testers with average talent will never outperform a small group of highly talented testers in terms of depth and quality. A great parallel example of this is when the world’s largest orchestra played ninth symphonies of Dvorák and Beethoven. While that orchestra was made up of 7,500 members, the quality of their song was nothing compared to that which is produced by The Boston Symphony Orchestra (which is made up of 91 musicians).
Interestingly, it appears that bug hunters are incentivized to spend as little time as possible per bounty. This is because bug hunters need to maintain a profitable hourly rate while working or their work won’t be worth their time. For example, a bug hunter might spend 15 minutes to find a bug and collect a $4,000.00 bounty, which is an effective rate of $16,000.00 per hour! In other cases, a bug hunter might spend 40 hours to find a bug and collect a $500.00 bounty which is a measly $12.50 per hour in comparison. Even worse they might spend copious time finding a complex bug only to learn that it is a duplicate and collect no bounty (wasted time).
This argument is further supported when we appraise the quality of bugs disclosed by most bug bounty programs. We find that most of the bugs are rudimentary in terms of ease of discovery, general complexity, and exploitability. The bugs regularly include cross-site scripting vulnerabilities, SQL injection vulnerabilities, easily spotted configuration mistakes, and other common problems. On average they appear to be somewhat more complex than what might be discovered using industry standard automated vulnerability scanners and less complex than what we’ve seen exploited in historical breaches. To be clear, this doesn’t suggest that all bug hunters are low talent individuals, but, instead, that they are not incentivized to go deep.
In contrast to bug bounty programs, genuine penetration testing firms are incentivized to bolster their brand by delivering depth, quality, and maximal coverage to their customers. Most operate under a fixed cost agreement and are not rewarded based on volume of findings, but instead by the repeat business that is earned through the delivery of high-quality services. They also provide substantially more technical and legal safety to their customers than bug bounty programs do.
For example, we evaluated the terms and conditions for several bug bounty companies and what we learned was surprising. Unlike traditional penetration testing companies, bug bounty companies do not accept any responsibility for the damages or losses that might result from the use of their services. They explicitly state that the bug hunters are independent third parties and that any remedy with respect to loss or damages that a customer seeks to obtain is limited to a claim against that bug hunter. What’s more is that the vetting process for bug hunters is lax at best. In nearly all cases, background checks are not run and even when they are run the bug hunter could provide a false identity. The validation around who a bug hunter really is, is also lacking. To sign up to most programs you simply need to validate your email address. In simple terms, organizations that use bug bounty programs accept all risk and have no realistic legal recourse, even if a bug hunter acts in a malicious manner.
To put this into context, bug bounty programs effectively provide anyone on the internet with a legitimate excuse to attack your infrastructure. Since these attacks are expected as a part of the bug bounty program, it may impact your ability to differentiate between an actual attack and an attack from a legitimate bug hunter. This creates an ideal opportunity for bona fide malicious actors to hide behind bug bounty programs while working to steal your data. When you combine this, with the fact that it takes an average of ~200 days for most organizations to detect a breach, the risk becomes even more apparent.
There’s also the issue of GDPR. GDPR increases the value of personal data on the black market and to organizations alike. Under GDPR, if personal data of a European citizen is breached, the organization that suffered the breach can face heavy fines, penalties, and more. In article 4 of the GDPR, a personal data breach is defined as “a breach of security leading to the accidental or unlawful destruction, loss, alteration, unauthorized disclosure of, or access to personal data transmitted, stored or otherwise processed”. While bug bounty programs target configurations, systems, and implementations, they do not incentivize bug hunters to go after personal data. However, because of GDPR, a malicious bug hunter who exploits a vulnerability that discloses personal data (accidental or not), may be incentivized to ransom their finding for a higher dollar value. Likewise, organizations might be incentivized to pay the ransom and report it as a bounty to avoid having to notify the Data Protection Authorities (“DPA”) as is required by GDPR.
On a positive note, many of our customers use bug bounty programs in tandem with our Realistic Threat Penetration Testing services. Customers who use bug bounty programs have far less vulnerabilities in terms of low-hanging-fruit than ones who don’t. In fact, we are confident that bug bounty programs are pointedly more effective at finding bugs than automated vulnerability scanning could ever be. It’s also true that these programs are more effective than penetration testing vendors who deliver services based on the product of automated vulnerability scans. When compared to a research driven penetration test, however, the bug bounty programs pale in comparison.
