Monday, January 5, 2009

Finding The Quality Security Vendor (Penetration Testing, Vulnerability Assessments, Web Application Security, etc)

While I've written several detailed white-papers on the subject of identifying quality security vendors, I still feel compelled to write more about the subject. It is my opinion that choosing the right security vendor is critical to the health and safety of a business.  Choosing the wrong vendor can leave you with a false sense of security that in the end might result in significant damages. Often times those damages can't be fully measured and appreciated, especially when they involve the tarnishing of a good name.

This problem of identifying quality isn't new but it does take on a new importance when it involves the safety of your trade secrets, source code, or otherwise critically sensitive information.  When you trust a security provider to test your IT Infrastructure, your people, physical security, etc. you are relying on them to identify risks that malicious hackers might otherwise discover.  If the provider does not test you at the same threat level as the malicious hackers then their service is almost useless. 

If that doesn't compel you to want quality security services then go ahead and take the risk.  I suppose the question really is, how much is your network (and its data) worth? If its worth more than $500,000.00 then its probably worth spending money on a quality security vendor to protect it right?

So how do you know which providers are quality and which ones are frauds?

The first rule of thumb is to watch out for the vendors that produce deliverables that are the product of vulnerability scanners.  There are two reasons for this, the first being that you don't need to pay anyone to run an automated scan when you can do it yourself for much less, or for free. You can choose from a variety of free tools like nessus, or you go out and buy a license for a vulnerability scanner.  

Don't be fooled though, vulnerability scanners do not produce accurate results. In fact most vulnerability scanners produce results that contain anywhere from 40-90% false positives with an unknown rate of false negatives.  While these tools are useful for reconnaissance they should not be used as the primary method for security testing. 

Watch out for the vendor that tells you that they will run a vulnerability scan against your network and then "vet" the results.  Vetting doesn't mean that they are going to do additional discovery. Vetting only means that the vendor will check the results of the vulnerability scan and eliminate the false positives. The quality of the end product is then only as good as the accuracy of the vulnerability scanner. Would you bank on that?

When you are choosing the vendor make sure to ask them specific questions.  Questions that I find helpful are realistic but based on theoretical architectures.  For example you could ask a vendor the following question:

"Suppose you are confronted with an architecture that consisted of 10 desktops behind a single firewall.  That firewall has properly configured IPS capabilities and there are no ports forwarded from the internet to any system behind that firewall. How would you [the vendor] penetrate into that network? Once you penetrate how would you perform Distributed Metastasis?" Email me for the answer if you don't know it already. 

You can also ask the vendor how they would use a directory traversal vulnerability to penetrate into a network.  This is a bit of a trick question but if they know what they are doing then they will be able to answer it properly.  The short answer is that you need to inject code into the web-server's error log and then use the directory traversal vulnerability to render the code. (Again, if you need the complete answer email me and I'll get it to you.)
Another good rule is to only choose security vendors who also perform Vulnerability Research and Development ("R&D").  That is to say that the vendor must frequently perform security research against technology, identify vulnerabilities in that technology, create exploits for those vulnerabilities and must release formal security advisories. If they don't then chances are they don't know how to do it, but why is R&D important? 

R&D enables the vendor to keep its penetration testing skills honed (so long as the research done by the penetration testers).  Penetration Testers who do not perform this kind of research are literally Script Kids (sorry guys).  Script Kids are people who download tools and use those tools to penetrate into networks. In almost all cases they don't have any understanding of how the tools work.  If you think about it, thats like giving a loaded gun to a 3 year old. 

You can also ask the vendor how they collect their threat intelligence.  Threat intelligence is a critical aspect of delivering quality security services.  If the vendor doesn't have current threat intelligence about the threat then how will they help you to defend yourself against the threat? While I won't tell you how my team collects this intel, I will tell you that its not from the news and most certainly not all public forum. 

In closing, my recommendation to you is that you do your homework before you choose a vendor. Research the components required for delivering a quality service, then use your research to question the provider.  As an example, if you were going to get a Web Application Penetration Test ask the vendor to define the term "Penetration Test".  Ask the vendor what the difference is between a Penetration Test and a Vulnerability Assessment.  Also ask them to explain RFI, LFI, XSS, SQL Injection, Blind SQL Injection, etc.  Remember, you are going to spend money on security, might as well make it worth while.  If you don't then you're just adding that money to the damages from the hack that you'll suffer in the end. 

If you have any questions please feel free to leave me a comment or send me an email.  You might also want to check out the white papers that I've linked at the upper right hand corner of this blog.  Those papers go into more detail about how to choose a good security vendor and how to select the right service. 


  1. Are you suggesting you've never missed a potential vulnerability in a system while doing your vuln scans/pen testing? So someone missed snmp on a router, the point is that you can have 10 "professional" pen testers scan a network and all 10 will come back with something different.
    I agree some companies are better than others, but you are beginning to fight an uphill battle that you'll never win or even get close to.
    How much do you know about your car? How much did you research it before you bought it? Did you compare every stat, every model? I doubt it. This is what you're asking people in the industry looking to make a purchase for "security" to do. It's just not going to happen. Furthermore, when a person calls said security company, they talk to the sales people, on purpose. This lets the sales people make up and lie about whatever the company does and is selling while the engineers try to do what was sold.

    I'm not saying that people shouldn't try to educate themselves, but if a person making the descision isn't apt to do research, then it just isn't going to happen. You can jump up and down and scream all you want, it's not going to make a difference.

    I totally agree that poor quality vendors in any business should be shutdown and put out of business, but this is the way of the world.

    I'm not suggesting you stop jumping up and down and screaming, I just hope you don't get yourself burnt out or loose your enthusiasm when you don't see any results.

  2. Your response is awful emotional, are you one of those low quality shops that rely on vulnerability scanners?

    1-) We do not miss known vulnerabilities when we perform penetration tests or vulnerability assessments.

    2-) We do not rely on scanners for anything more than initial reconnaissance. The rest of our work is done manually via Real Time Dynamic Testing.

    3-) If you have 10 people using 10 scanners and the results come back different, you've got a pretty serious issue with your people or your scanners. Either that or you are taking systems off-line with your scans and you didn't do you homework before you started.

    4-) Yes I do research my cars before I buy them, and I do it quite extensivley.

    5-) Yes people should investigate companies before they spend money on security. Why are you so upset about that, are you afraid that you won't pass the scrutiny?

    6-) We are winning the battle and its certainly not uphil. We know what we are doing and so do the other quality providers.

    7-) Sales people do lie which is why you want to ask to speak to someone technical or ask for a sample report. Compare the report to the results of a nessus scan. Look for key words that indicate that its the product of automation. If it is, run.

    8-) You think that the people making the decision aren't apt to do the research? I think you're wrong.

    9-) I'm not sure what gives you the impression that I am jumping up and down here. Instead I'm doing what I've always done which is to help people protect themselves.

  3. I run two teams of penetration testers in New Zealand performing large amounts of penetration testing.
    I fully agree with the comments made here regarding automated scanners.

    Any decent consultant will detect 100% of any critical or urgent security finding. I have noticed some variance with lower severity findings as the findings are sometimes missed.

    Automated scanners can reliably find the lower severity bugs, which often relate to information disclosure bugs. Automated scanners are useful here.

    I am yet to find an automated scanner which is able to detect a complex application logic based flaw though.

    The ideal situation is using both manual testers and automated scanners and combining the results, this also helps to reduce false positives. Man is better than machine in this case :)

    It worries me GREATLY that MANY companies use 100% automated scanning and charge the same rate/hours as a REAL consultant.

    Not cool.

  4. Hi everyone, my $0.019999...

    1) I will agree whole-heartedly that you "get what you pay for", and if you're not getting a reputable firm to do your pen testing then you may as well assume you're getting a false-sense of [in]security unless...
    2) Your aim as a customer is not security - but rather - it is compliance. These are very distinct issues, more on this in a minute
    3) I do not accept that any team is good enough to find 100% of issues, at a rate of 100% of the time. Impossible, period. I can't imagine any reputable person making a bold claim like "we never miss anything" (sounds way too much like a used car salesman...)
    4) Vulnerability "scanners" are definitely not up to the task of finding higher-order vulnerabilities (those that require the complex human mind to find and understand them)... this is where AI would be required and we're simply not there yet

    On points #1/#2 above - some customers want to know they are secure, and they will pay for the platinum-service the A#1 top-rated pen testing outfit offers. Others simply want to comply with whatever regulation requires them to "get an external penetration test by a 3rd party" and therefore don't mind getting second/third-rate consultants who are glorified (or not) script-kidz... after all, the service is cheap, the findings a minimal, and the box is checked!
    Honestly... the compliance approach worries me greatly, but it's a sad fact of the world we live in that simply does not care enough about security to put in the effort/money. Look at the financial crisis if you want proof that the main goal of most organizations isn't to be "secure" or "actually trustworthy"... it's to check the compliance box to make the public feel better. Does this sound like the TSA? Oopps.

    Here's to a 2009 filled with more business than you can handle.

  5. This was a pleasure to read. You really hit on many key points on how to select a quality security vendor; however, I feel that you left out a few very important ones:

    1. How well the security vendor understands your specific vertical (Healthcare, Financial, Government, etc.). Performing R&D on applications I do not own/use proves you understand Pen Testing but does not mean you could perform a worth while test in my environment. Knowing the vertical also speeds up the testing process and thus should lower the overall costs (always a good thing).

    2. Their experience out side of Pen Testing. Personally, I prefer vendors that have experience working for large companies, as employees and not consultants. Far too often, I have received reports that say "Install Patch xyz" or something similar written by a consultant that has never had to worry about change process or support contracts that prevent/delay said installation. Understanding how a corporation works and all the politics that go along with it, helps security consultants deliver much more valuable reports thus making the remediation process easier, which again saves money.

    3. While I agree performing R&D is necessary, I disagree that to be a “quality” security vendor you need to publish your findings (security alerts/advisories). In my early days of computer security, and long before there were quality tools, I performed penetration testing and became quite good at it; however I never published a single document. Why? Most my testing was for the government or under tight Non-Disclosure Agreements and therefore I was prohibited from posting anything. Furthermore, to some extent, I am seeing more and more “post kiddies” – those that post articles/view points on some one else’s research, which they may or may not understand themselves, to get recognition. This of course muddies the waters even more.

    I wholeheartedly agree with your closing argument – “do your homework”. It is vital that the purchasing department (security or otherwise) educate themselves on exactly what they want (sometimes a simple VA scan is all you want – perhaps to validate the patch deployment process) and how to differentiate between the BS and quality. Your scenario questions are a great way to do just that.

    Just my 2 cents. Thanks again for the good read.

  6. Great post, the problem is that a very small part of the customers will read something like this. A lot of time the customer don't know too much about pentesting, or security assessement so they can't ask good questions :(

  7. I've known of a peer who actually organized a "capture the flag" competition to select the vendor for penetration testing contract. For organizations who do not have this capability, just ask around for referrals.

  8. thank you for this short write up. It was very interesting.
    I once worked for a VAR that sold vulnerability scanning/discovery
    as Pen Testings.
    Their processes were very bad in my opinions and left alot of gaps in
    the reports and testing and I imagine that the previous company that
    did the pen testing in your situation had similar problems.
    I am so amazed that people can get away with it. If only the customer
    took the time to really look into what was done and how it was done.
    then maybe they would have known that some shortcuts were taken or
    that something was either missing or not done correctly. I understand
    that some corp don't have the resource and time to learn or try to
    understand security, but in the end their the ones paying for it
    either way. either by paying for a crappy report or getting owned and
    paying the cost to recover. so in my opinion they should take the time
    to figure out what was done.
    just my two cents.