Wednesday, January 7, 2009

Network Vulnerability Scanning Doesn't Protect You

Vulnerability scanning can have a detrimental negative impact on the security posture of your IT infrastructure if used improperly. This negative impact is due to a perceptional issue that has been driven by the vendors who sell vulnerability scanning services or the vulnerability scanners themselves. The hard facts prove that vulnerability scanners can not protect your IT Infrastructure from malicious hackers. (My team penetrates "scanned" networks on a regular basis during customer engagements). That is not to say that vulnerability scanners are useless, but it is to say that people need to readjust their perception of what vulnerability scanning really is.

While there are various types of vulnerability scanners they suffer from the same disease that most security technologies suffer from. That disease is that they are reactive to hackers and will never be proactive. The fact is that vulnerability scanners can not detect vulnerabilities unless someone has first identified the vulnerability and created a signature for its detection. This process can take quite a while and is often not an ethical one. So here is how it works...

A hacker decides to perform research against a common technology like your firewall. That hacker might spend minutes, months or even years doing research just for the purpose of identifying an exploitable security vulnerability. Once that vulnerability is identified the hacker has an ethics based decision to make. Does he notify the vendor of his discovery and release a formal advisory or does he use his discovery to hack networks, steal information and profit.

If the hacker decides to notify the vendor and release an advisory then there is usually a wait period of 1-3 months before the vendor releases a patch. This lag time means that the vendor's customers will remain vulnerable for at least another 1-3 months, most probably longer. What's even more interesting is that this vulnerability may have been discovered previously by a different researcher that didn't notify the vendor. If that's the case then that probably means that the vulnerability has been in use as a tool to break into networks for a while. Who knows, it could have been discovered months or even years ago? That type of unpublished vulnerability is known as a 0day and is the favorite weapon of the malicious hacker.

At some point the vulnerability does become public knowledge. Its also at this point that the vendors who make the vulnerability scanning technology become aware of the new risk. When they do learn about the new risk they need to develop a signature, or script for their scanning technology so that it can detect the risk. That development process can take anywhere from a few days to a few weeks depending on the complexity risk. As a result, the customers that rely on vulnerability scanning are in the dark until the vendor can publish a working and tested signature... but the hackers don't need to wait at all. The hackers can use it almost immediately.

So in summary, there is a large risk window between the point of discovery of a vulnerability and the point at which a vulnerability scanner can detect the vulnerability. This risk and exposure window is usually never smaller than a few months, and can be as large as several years. During that time there is a very good chance that malicious hackers will be using your undiscovered risks to penetrate into your infrastructures. Whats worse is that you'll have no idea that you've been hacked because like vulnerability scanning technology, Intrusion Detection technology also can't identify threats if it doesn't know what to look for. Moreover most Intrusion Detection technologies aren't configured properly and as such don't work properly.

Unfortunately the story doesn't end there. Vulnerability scanners also suffer from significant issues with accuracy. In all cases where I've used (various) vulnerability scanners, the best results that I've ever achieved were about 30% accurate. This means that most of the vulnerabilities that were detected during my various scans weren't actually vulnerabilities but instead were false alarms, also called false positives. More frightening is the number of vulnerabilities that I discovered while performing Real Time Dynamic Testing (manual hacking) that were entirely missed by the vulnerability scanner. If you don't believe me then go download a free vulnerability scanner, test your network and verify the results yourself.

This inaccuracy is partially due to the architecture of the vulnerability scanners and the fact that no two networks are alike. Vulnerability scanners use static signatures or scripts that are only capable of checking a target for a vulnerability if their syntax is exactly accurate and if the target responds in a way that the scanner can understand. If however the target, lets say its a computer system, is configured in a custom way then it may not respond in a way that the scanner will understand (how many of you keep the default configuration?). This communication barrier is a large part of what causes false positives and false negatives.

An important note about false positives and false negatives. Some vendors claim that their vulnerability scanners have low rates of false positives. As with Intrusion Detection, if low false positive rates are true then its usually reasonable to say that the technology has high rates of false negatives. You can think of it as a sliding scale of 1 to 10 where 1 is 100% False Positives and 10 is 100% False Negatives. As you move up and down the scale you inevitably end up with more of one or the other, you can never eliminate them. With that said, its my opinion that more false positives are better than more false negatives.

If vulnerability scanners aren't the right way to protect yourself then what is? You should protect yourself by exposing your business to an accurate and controlled reproduction of the threat by using a quality security provider. It is important to remember that no single hacker, good or bad, has access to all of the 0-day's in the world. As such, it is entirely possible for a team of ethical hackers to accurately reproduce the threat that unethical hackers can create. Testing at that level enables you to identify weaknesses in your defenses that would not otherwise be detected by testing at lesser levels. What good would a penetration test or a vulnerability assessment do if the malicious hackers will test you harder?

One of the many advantages of using a team of talented hackers for security testing instead of relying on automated vulnerability scanners is that those hackers can and should perform research against unique technologies that they encounter during a security test. I practice what I preach by the way. When our team delivers an Advanced Penetration Test to a customer we always perform our own research against interesting targets. Those targets can be Web Applications, Web Services, or even custom daemons running on systems. In the end, if we find something new we'll write an exploit (proof of concept) for the customer and include that in the final deliverable.

In closing, I am not suggesting that network vulnerability scanners are bad because they do have their place and they do serve a purpose. They are particularly useful in the hands of a skilled security expert especially when performing reconnaissance against large networks. In that scenario the scanner enables the expert to save time and to rapidly collect intelligence about targets given that the engagement is non-stealth in nature. With that said, I wouldn't rely on scanners for anything more than just reconnaissance, at least not yet.

Note: (Thank you to minoo for pointing out a few mistakes in my previous revision of this entry. I hope that this entry is as clear as I intend it to be. There is no one team that is the best, but there are only a few good ones. If this isn't clear enough or if it needs more revision please comment.)


  1. This comment has been removed by a blog administrator.

  2. So you're saying that pen-testers magically know all the 0-day vulnerabilities during that signature awareness/creation time window? Or that they're doing original vuln research during their limited time engagement with company x? That's impressive. Why wouldn't the signature writers be able to do the same thing and just add the signatures as well? What if those vendors hire hackers and pen-testers to write their signatures?

  3. Minoo, awesome comment bro! I didn't realize that the message that I was trying to convey seemed so skewed until you posted that. I've augmented my post as a result of your comment (see above).

  4. This title is somewhat misleading.

    Vulnerability scanning does work (it finds a lot of known vulnerabilities on the target), but fixing those doesn't mean your system is safe then. Also when during a penetration test, executed by a good tester, no security issues were found, it does not mean your system is safe. It just means that nothing is found. And even if testing proves a system is perfectly safe, it might be unsafe before the ink of the report dried.

    So actually vulnerability scanning is no silver bullet. Neither is any other type of testing.

  5. Well, as usual with the security world, it's not really the problem with the software as it's implementation. Automated vulnerability scanners are great, but for a low-level security need.
    Hiring a full security team is quit costly and not exactly a viable choice for many companies. Running an automated vulnerability scanner (AVS from now on) can be incredibly useful in these cases. They are usually simple to use and provide easy to read output that can be answered online by several helpful communities.
    Also, I know of several security teams who run AVS as a first wave of testing. It can point you in the direction of current vulnerabilities before you go in-depth with the network(s) architecture.
    However, as this article states, it has become overhyped and turned into a problem. People have started relying on these while others are praising it's use as an all-in-one security test.
    In the end, AVS software is incredible. We just need to understand that it is to be used for low levels of security and not a full security testing solution. As I believe this article was trying to state.

  6. VS can help identifying & closing gaping holes, based on information about existing vulnerabilities and common misconfigurations... This is a good idea as such, and it may help increasing security against lame bored scriptkiddies.
    Most of the times these kids don't know what they are doing, and will not hesitate to bring down a server (that is, if they even realize that they are bringing down a server)
    I agree that VS will never stop the more advanced and experienced hacker, but hey, nothing can...
    so, imho, it is useful to us VS on a continuous basis, even if it is only used to identify configuration mistakes and validate the success of your patch processes. But I would not spend my money on very expensive tools that promise to find all holes and make your network 100% secure.. I'd rather spend my bucks on implementing a good incident response procedure. If a hacker is really targetting you, he will get in. Period.
    No VS will help you.

  7. An automated scanner can handle the tedious task of testing for all of the known vulnerabilities on every system. No pen-tester has the patience or the time to do this on a normal engagement.

    When performing manual testing, it is immediately obvious which customers have an automated vulnerability management practice in place, and which do not. As a manual tester, I have stopped tests mid-test, and recommended automated solutions to customers so they could clean up the low-hanging fruit before bothering with a manual test.

    The advantage of using an automated test is that you can check for the majority of known problems quickly and cost-effectively.

    The assumed advantage of using a manual testing team is that they can take the test to a deeper and more complex level than an automated tool.

    If a customer only has the budget for one, they will get a better return on investment by choosing the automated option, and using it in smaller time periods (monthly or weekly). They should then compliment that scanning with manual testing.

  8. Most important is not scanning, it's management. It means what are the procedures to discover, analyse and fix security issues (or vulnerabilities).

  9. Can't help but think that people aren't quite getting the point of the post. But that might just mean it is me who is missing it ;)

    Scanners, personnel, skills. They have been debated on this blog lately and some people seem quite offended by the posts.
    I don't feel this is personal type attacks on products, people or skill sets.
    I do read it as a critique of the people in charge of selecting the road before them.

    If you are a medium sized company getting yourself to a point where security becomes a larger and larger piece of your required business. Then you can go cheap and dirty and have the latest laid off employee from the Geek Squad do security. he will fire Nessus and Metasploit at your environment and tell you after resolving what these tools find that you are secure. And because you went with cheap and dirty you will barely have a clue about the upcoming breach before someone external alerts you to the fraudulent use of the account numbers in your "care".

    Or you can realize that being the caretaker of another persons account holder information, social security number etc. places a responsibility on your business to make a breach a very remote possibility.

    Option 1 is cheap up front. Option 2 is expensive up front.

    With the breaches occurring left and right I can not imagine going with option 1 and experiencing a breach isn't overall far more expensive than going option 2 from the get go when it is simpler, easier and far less of a headache.

  10. For those that missed it the word "enterprise" is no where in this post.

  11. A lot of good points have been made both here and on the SF list.

    I understand exactly where you are coming from Adriel, and I definitely agree with what you are trying to convey. Although many tools are market as such, they are never even close to an end-all solution to uncovering vulnerabilities, and a company's reliance on such a tool is an utter tragedy. I think the most important thing to keep in mind is vulnerability scanners are simply just another tool in our toolbox, and can be used at our discretion to take the tediousness out of uncovering known and even forgotten vulnerabilities. However, as you pointed out, scanners tend to create a lot of additional and unnecessary work since all the results have to be vetted.

    Keep the faith, Adriel, you're always on the right track.

  12. Manual testing can 'potentially' find more stuff. I say potentially because this depends hugely on the skills you hire.
    Regular VA gets the low hanging fruit on an 'enterprise-wide' basis. Lets face it. No company is going to hire pen-testers to test every server, desktop, printer and switch in their environment.
    Finally,, how many of the customers you work with actually have a mature and well development security strategy backed up with good procedures and the right kind of response capability? I am willing to guess they would be under 5% ! For the rest of them, VA is a huge benefit because they havent got their basics right yet. Dont undermine VA, it provides huge value in comparison with manual testing.

  13. While it is good to get things off your chest once in a while, I think, for the most part you are preaching to the converted. The "suits" couldn't be concerned less about the courage and correctness of you convictions.

    As others have already commented you are mostly right. Vulnerability Scanners also have their purpose in life, they show us a way to start doing really important stuff, hands-on. What is important is that people like you and others on this blog and SF list realise the best way to accomplish what the client requested - security of his IT infrastructure.

  14. Actually I wasn't talking about Vulnerability Assessments, I was talking about Network Vulnerability Scanners. The fact of the matter is that Vulnerability Assessments when done properly also have an intense manual component to them. If they don't then you're not really doing any work. I mean, how hard is it to configure a scanner and click go?

  15. I think that the title of this post is a bit mis-leading. I have used vulnerability scanners for years starting with Ballista and Satan (now Saint) and have seen them come and go. None are perfect but most do provide value in providing a picture of a networks security posture. That picture varies depending on the environment that the scanner is run in. However your title makes it sound like that value is nil and I have to disagree with you there.

    I do understand what you are saying; that vulnerability scanning alone doesnt cut it and I do agree with that however it's what you didnt say that could have made this article better. Vulnerability scanners are not a panacea for security. Rather organizations should adopt a comprehensive strategy for security that INCLUDES but IS NOT LIMITED TO vulnerability scanning, regular patching and pen testing. Vulnerability scanning is only one piece of the puzzle but the title of your post would have the casual reader think that this particular piece is of no value to the puzzle and in that I disagree.

  16. As a sales person for a vulnerability detection tool you might be surprised to find me agreeing with you, that "Network Vulnerability Scanning Doesn't Protect You". But not because I think that manual penetration testing is better. I suspect most people will surely agree that something done by a human is often better than something done by a machine, but the costs involved will restrict their choices.

    The bigger issue, and why I agree that "the hard facts prove that vulnerability scanners cannot protect your IT Infrastructure", is that you need to act on the information produced by a vulnerability detection tool. Not that you should dump your automated tool in preference of a team of human experts.

    Failure to act on the identified vulnerabilities and apply the required changes is what puts you at risk.

    Whether the user fails to act on recommendations from a machine or from a human is in that regard the same thing.

    The question of "better" then becomes, how much does a business risk reduction by 10-points cost me, if done by a machine, versus if done by human. Bearing in mind that the cost of detection is just the tip of the iceberg – the internal costs of applying the change is what will really cost the client.

  17. The real question is how much money is your network worth? Don't forget to consider the information that is contained in the network. People really can't afford the "cheap" solution, they just think that they can. At least they do until they get hacked.

  18. If you're interested, I have written a reply in my blog.

    The tl;dr version:
    I agree with what you've said, and disagree with how you've said it.

  19. Good article but I disagree with your comment: "People really can't afford a 'cheap' solution". The problem lies in the fact that there is never 100% secure solution. So even if I choose a great vendor, invest an appropriate amount of funds, and perform all the remmediation steps outlined - I can still be compromised the next day; therefore we must balance between the Risk/exposure vs cost. Running a vulnerability scanner does one thing for an organization- it cheaply (potentially free) gives them a base line to measure from. VAs used as a starting point and not an ending point will help an organization begin to build the proper security procedures/policies/practices/solutions they need.

  20. I really don't like how so many people are comparing the intelligence of the selected members of the pen testing teams with the automation of a tool. If you are going to put weight on intelligence in that regard, consider the intelligence of whomever is writing the software that you are using. Chances are, if you were to hire a pen testing team, you'd have a better chance at gauging their intellect than trying to hunt down some programmers at a corporation.