Sunday, January 4, 2009

Followup to my last Brian Chess - Fortify Software post.

Recently I published a post about Fortify Software's Brian Chess because of some outlandish claims that he made in an article about penetration testing being "Dead by 2009". The off-line and 0n-line comments that resulted from that post were mostly in favor of what I'd written and one of those comments really caught my eye. So here is a post dedicated to Rafal in response to his comment on my article about Brian Chess.

Comment By Rafal shown below, verbatim in pink:

"If I may call a sanity timeout here folks - while I don't agree with Brian's assertions necessarily - if you combine a few factors you could conceivably come to the conclusion that penetration testing will start to dwindle (just not as quickly as 2009)."

Its only conceivable for those who do not know what Penetration Testing is, and many self-proclaimed security guru's don't. So lets start with some (partial) definitions here:

Vulnerability Assessment:
(Assessment: the act of assessing; appraisal; evaluation.)
A Vulnerability Assessment is a service that evaluates a particular target, or set of targets for the purpose of identifying points of exposure that are open to assault. A Vulnerability Assessment does not attempt to compromise or penetrate into a target once a point of exposure is identified, it only aims at assessing the target for points of risk. Vulnerability Assessments by their very nature are prone to False Positives and False Negatives as the findings are never validated via Penetration or Exploitation.

Vulnerability Assessment Tools include:

  • WebInspect for Web Application Vulnerability Assessments
  • Nessus for Network Vulnerability Scanning
  • Fortify for Web Application Vulnerability Assessments
  • Retina Network Vulnerability Scanning
  • etc... you get the idea.

Penetration Test:
(Penetration: the act or power of penetrating.)
A Penetration Test is a service that evaluates a particular target, or a set of targets for the purpose of identifying points of exposure that are open to assault. A Penetration Test differs from a Vulnerability Assessment in that it attempts to penetrate into the target by exploiting any discovered points of risk and exposure. A Penetration Test when done properly will result in an accurate deliverable that contains no false positives. This is possible because exploitation of a risk or point of exposure is either successful or not. Penetration Tests can include theoretical findings but they should not be reported on as positives.

Penetration Testing Tools include (I'd recommend these):

You can use a Vulnerability Assessment or a Penetration Test against any type of target not just technology based targets. At Netragard we perform physical penetration tests, wireless penetration tests, network penetration tests, social engineering based penetration tests, web application penetration tests, etc. Likewise we can deliver vulnerability assessments against the same set of targets if penetration testing is too aggressive.

(I get the feeling that both Rafal and Brian Chess think that Penetration Testing is a Web Application only service)

"Here's my logic - feel free to scrutinize. For the record I work for HP (the SPIDynamics acquisition) so you guys can feel free to rip on the fact that our marketing folks I'm sure make interesting claims as well... but I digress. Here are some things to consider:

Actually we've got quite a bit of interesting history with HP, but that's a different story. With respect to SPIDynamics and the Web Inspect tool, I'm sorry that HP ever acquired SPIDynamics. WebInspect was a reasonable tool for doing preliminary reconnaissance against Web Applications during non-covert services. Once HP acquired the technology its quality went down the tubes. Not only that but the process of acquiring a license from HP is excruciatingly painful at best. What ever happened to being able to purchase the product online? /end rant

"1. When you do penetration testing, what are you really testing? Are you testing the system or the intelligence and skill of the pen tester? This is a very tough question to answer.

Why is that a difficult question to answer? If you've built your penetration testing team properly then your team will be able to expose its targets to the same or greater threat level than that which they will likely face in the real world. The fact of the matter is the more secure the infrastructure the more challenging the test and yes, its impossible to know everything but its not impossible to do a great job.

"2. Pursuant to #1 above, and the business' (living in reality land here) need to do lowest-cost vendors... what value do you suppose that the 90%+ of companies that go lowest-cost (outsourced to India, China, Mexico, etc) are getting?"

Businesses do not "need to do the low-cost vendors", they choose to because they are making uneducated decisions in most cases. Mind you the lack of education on their part is not their fault, its the fault of the poor quality vendors. Poor quality vendors advertise their services as if they are the same quality as the high quality vendors thereby causing confusion. When a business compares the two services they don't see the difference and so they choose the less expensive one.

"3. With every point-and-click testing tool there is a double-edged sword... here's why 3a. Tools make you more efficient BUT"

I only partially agree. When the tool spits out over 2,000 false positives (like WebInspect did the last time we used it) with only 3 real positives its doing very little to increase the efficiency of a team. Other tools that produce less false positives and more accurate results are very useful for time savings but their results should not be used to create an end product. Automated tools are not dynamic by nature and as such can not identify the same risks as talented penetration testers.

"3b. Tools can make you less "hands-on" when it comes to writing low-level exploits or code..."

Tools are also the root cause of the the fraudulent security experts. I'm not saying that tools don't have their place because they certainly do. But they allow people to become lazy and as such breed "experts" that are for all intents and purposes no better than script kids (which might I add are very dangerous because they don't know what they are doing).

"4. Penetration testing is an after-the-fact requirement... which is too late. You have to use tools to augment and empower your developers to write better code at the grass-roots otherwise you're hosed."

You're partially wrong. The tools that you speak of are derived from the attacks that were created by Penetration Testers (aka: hackers). With respect to the world of Web Applications, do you think that a tool discovered the first SQL Injection vulnerabilitiy and created a method for exploitation? Ofcourse not! Tools will always be a few steps behind the capabilities of a real hacker, regardless of that hackers ethical bias. The fact of the matter is that as hackers, we perform research and identify new methods for penetration that were not previously discovered and your tools can not and will not ever be able to defend against that.

"So - to summarize, penetration testing isn't going to be "dead" in this year of 2009, but it may start to dwindle down some depending on how good the marketing machines of the tools vendors are. Brian's statement is a self-fulfilling prophecy... he is making a statement that he hopes will incite people to make that statement come true."

I disagree, and again, you are working for a vendor that makes these tools. Its in your best interest to suggest that some how Penetration Testing will be less of a requirement because of the tools that you create. The reality of it is that if people drink that kool aid they will become more vulnerable, not more secure.

When our military tests the armor of its M1A2 Abrams Tank they test it against the real threat. So why aren't we pushing our customers to do the same thing, it makes perfect sense? In our case the real threat is always going to be the malicious hacker, not the software vendor making pretty and easy to use tools. The tools do have a place but they will only ever identify the low hanging fruit. It takes a professional hacker/penetration tester to actually test an infrastructure properly. Lets see your tools perform Social Engineering or drop USB sticks in parking-lots.


  1. It seems to me that this prophesy of the end of penetration testing is designed to test the resolve of potential pen-testers or hackers to go the whole route. Why bother doing the hard job of actually learning how to be a real hacker (penetration specialist) if you are pretty sure that some application is going to make your specialty obsolete?
    This clears the field of some potential competition, but puts a lot more unqualified testers out there who can operate metasploit.

  2. Wolf, thanks for the comment but I can't say that I agree. It seems to me that people have lost sight of what a real penetration test is. If that was not the case then you, like the rest would understand that penetration testing can not be made obsolete.

    With respect to your question, you are making a fatal assumption. The security technology is a product of research that has been done by hackers. Hackers will evolve, we always do, and will defeat any security technologies that are created. You must also remember that security technologies are by their very design reactive and as a result will always be a few steps behind the hackers. So your question "Why bother doing the hard job of actually learning how to be a real hacker (penetration specalist) if you are pretty sure that some application is going to make your specialty obsolete?" is factually inaccurate. No technology can make our specialty obsolete and with respect to that I've challenged Brian Chess but have yet to have my challenge accepted.

    The fact of the matter is that if Brian Chess does accept our challenge we'll break in. There are many ways to hack into networks, web applications aren't the only gateway and they are most certainly not the easiest. You're forgetting about other powerful tools that we have in our arsenal.