Followup to my last Brian Chess - Fortify Software post.

Recently I published a post about Fortify Software's Brian Chess because of some outlandish claims that he made in an article about penetration testing being "Dead by 2009". The off-line and 0n-line comments that resulted from that post were mostly in favor of what I'd written and one of those comments really caught my eye. So here is a post dedicated to Rafal in response to his comment on my article about Brian Chess.

Comment By Rafal shown below, verbatim in pink:


"If I may call a sanity timeout here folks - while I don't agree with Brian's assertions necessarily - if you combine a few factors you could conceivably come to the conclusion that penetration testing will start to dwindle (just not as quickly as 2009)."

Its only conceivable for those who do not know what Penetration Testing is, and many self-proclaimed security guru's don't. So lets start with some (partial) definitions here:

Vulnerability Assessment:
(Assessment: the act of assessing; appraisal; evaluation.)
A Vulnerability Assessment is a service that evaluates a particular target, or set of targets for the purpose of identifying points of exposure that are open to assault. A Vulnerability Assessment does not attempt to compromise or penetrate into a target once a point of exposure is identified, it only aims at assessing the target for points of risk. Vulnerability Assessments by their very nature are prone to False Positives and False Negatives as the findings are never validated via Penetration or Exploitation.

Vulnerability Assessment Tools include:

• WebInspect for Web Application Vulnerability Assessments
  
• Nessus for Network Vulnerability Scanning
• Fortify for Web Application Vulnerability Assessments
• Retina Network Vulnerability Scanning
• etc... you get the idea.
  

Penetration Test:
(Penetration: the act or power of penetrating.)
A Penetration Test is a service that evaluates a particular target, or a set of targets for the purpose of identifying points of exposure that are open to assault. A Penetration Test differs from a Vulnerability Assessment in that it attempts to penetrate into the target by exploiting any discovered points of risk and exposure. A Penetration Test when done properly will result in an accurate deliverable that contains no false positives. This is possible because exploitation of a risk or point of exposure is either successful or not. Penetration Tests can include theoretical findings but they should not be reported on as positives.

Penetration Testing Tools include (I'd recommend these):

• The Metasploit Framework
  
• CANVAS
  
• Core Impact
  
• etc...
  

You can use a Vulnerability Assessment or a Penetration Test against any type of target not just technology based targets. At Netragard we perform physical penetration tests, wireless penetration tests, network penetration tests, social engineering based penetration tests, web application penetration tests, etc. Likewise we can deliver vulnerability assessments against the same set of targets if penetration testing is too aggressive.

(I get the feeling that both Rafal and Brian Chess think that Penetration Testing is a Web Application only service)

"Here's my logic - feel free to scrutinize. For the record I work for HP (the SPIDynamics acquisition) so you guys can feel free to rip on the fact that our marketing folks I'm sure make interesting claims as well... but I digress. Here are some things to consider:

Actually we've got quite a bit of interesting history with HP, but that's a different story. With respect to SPIDynamics and the Web Inspect tool, I'm sorry that HP ever acquired SPIDynamics. WebInspect was a reasonable tool for doing preliminary reconnaissance against Web Applications during non-covert services. Once HP acquired the technology its quality went down the tubes. Not only that but the process of acquiring a license from HP is excruciatingly painful at best. What ever happened to being able to purchase the product online? /end rant

"1. When you do penetration testing, what are you really testing? Are you testing the system or the intelligence and skill of the pen tester? This is a very tough question to answer."

Why is that a difficult question to answer? If you've built your penetration testing team properly then your team will be able to expose its targets to the same or greater threat level than that which they will likely face in the real world. The fact of the matter is the more secure the infrastructure the more challenging the test and yes, its impossible to know everything but its not impossible to do a great job.

"2. Pursuant to #1 above, and the business' (living in reality land here) need to do lowest-cost vendors... what value do you suppose that the 90%+ of companies that go lowest-cost (outsourced to India, China, Mexico, etc) are getting?"

Businesses do not "need to do the low-cost vendors", they choose to because they are making uneducated decisions in most cases. Mind you the lack of education on their part is not their fault, its the fault of the poor quality vendors. Poor quality vendors advertise their services as if they are the same quality as the high quality vendors thereby causing confusion. When a business compares the two services they don't see the difference and so they choose the less expensive one.

"3. With every point-and-click testing tool there is a double-edged sword... here's why 3a. Tools make you more efficient BUT"

I only partially agree. When the tool spits out over 2,000 false positives (like WebInspect did the last time we used it) with only 3 real positives its doing very little to increase the efficiency of a team. Other tools that produce less false positives and more accurate results are very useful for time savings but their results should not be used to create an end product. Automated tools are not dynamic by nature and as such can not identify the same risks as talented penetration testers.

"3b. Tools can make you less "hands-on" when it comes to writing low-level exploits or code..."

Tools are also the root cause of the the fraudulent security experts. I'm not saying that tools don't have their place because they certainly do. But they allow people to become lazy and as such breed "experts" that are for all intents and purposes no better than script kids (which might I add are very dangerous because they don't know what they are doing).

"4. Penetration testing is an after-the-fact requirement... which is too late. You have to use tools to augment and empower your developers to write better code at the grass-roots otherwise you're hosed."

You're partially wrong. The tools that you speak of are derived from the attacks that were created by Penetration Testers (aka: hackers). With respect to the world of Web Applications, do you think that a tool discovered the first SQL Injection vulnerabilitiy and created a method for exploitation? Ofcourse not! Tools will always be a few steps behind the capabilities of a real hacker, regardless of that hackers ethical bias. The fact of the matter is that as hackers, we perform research and identify new methods for penetration that were not previously discovered and your tools can not and will not ever be able to defend against that.

"So - to summarize, penetration testing isn't going to be "dead" in this year of 2009, but it may start to dwindle down some depending on how good the marketing machines of the tools vendors are. Brian's statement is a self-fulfilling prophecy... he is making a statement that he hopes will incite people to make that statement come true."

I disagree, and again, you are working for a vendor that makes these tools. Its in your best interest to suggest that some how Penetration Testing will be less of a requirement because of the tools that you create. The reality of it is that if people drink that kool aid they will become more vulnerable, not more secure.

When our military tests the armor of its M1A2 Abrams Tank they test it against the real threat. So why aren't we pushing our customers to do the same thing, it makes perfect sense? In our case the real threat is always going to be the malicious hacker, not the software vendor making pretty and easy to use tools. The tools do have a place but they will only ever identify the low hanging fruit. It takes a professional hacker/penetration tester to actually test an infrastructure properly. Lets see your tools perform Social Engineering or drop USB sticks in parking-lots.