Security Vulnerability Penetration Assessment Test?

Our philosophy here at Netragard is that security-testing services must produce a threat that is at least equal to the threat that our customers are likely to face in the real world. If we test our customers at a lesser threat level and a higher-level threat attempts to align with their risks, then they will likely suffer a compromise. If they do suffer a compromise, then the money that they spent on testing services might as well be added to the cost in damages that result from the breach.
This is akin to how armor is tested. Armor is designed to protect something from a specific threat. In order to be effective, the armor is exposed to a level of threat that is slightly higher than what it will likely face in the real world. If the armor is penetrated during testing, it is enhanced and hardened until the threat cannot defeat the armor. If armor is penetrated in battle then there are casualties. That class of testing is called Penetration Testing and the level of threat produced has a very significant impact on test quality and results.

What is particularly scary is that many of the security vendors who offer Penetration Testing services either don't know what Penetration Testing is or don’t know the definitions for the terms. Many security vendors confuse Penetration Testing with Vulnerability Assessments and that confusion translates to the customer. The terms are not interchangeable and they do not define methodology, they only define testing class. So before we can explain service quality and threat, we must first properly define services.

Based on the English dictionary the word “Vulnerability” is best defined as susceptibility to harm or attack. Being vulnerable is the state of being exposed. The word “Assessment” is best defined as the means by which the value of something is estimated or determined usually through the process of testing. As such, a “Vulnerability Assessment” is a best estimate as to how susceptible something is to harm or attack.

Lets do the same for “Penetration Test”. The word “Penetration” is best defined as the act of entering into or through something, or the ability to make way into or through something. The word “Test” is best defined as the means by which the presence, quality or genuineness of anything is determined. As such the term “Penetration Test” means to determine the presence of points where something can make its way through or into something else.

Despite what many people think, neither term is specific to Information Technology. Penetration Tests and Vulnerability Assessments existed well before the advent of the microchip. In fact, the ancient Romans used a form of penetration testing to test their armor against various types of projectiles. Today, we perform Structural Vulnerability Assessments against things like the Eiffel Tower, and the Golden Gate Bridge. Vulnerability Assessments are chosen because Structural Penetration Tests would cause damage to, or possibly destroy the structure.

In the physical world Penetration Testing is almost always destructive (at least to a degree), but in the digital world it isn’t destructive when done properly. This is mostly because in the digital world we’re penetrating a virtual boundary and in the physical world we’re penetrating a physical boundary. When you penetrate a virtual boundary you’re not really creating a hole, you’re usually creating a process in memory that can be killed or otherwise removed.

When applied to IT Security, a Vulnerability Assessment isn't as accurate as a Penetration Test. This is because Vulnerability Assessments are best estimates and Penetration Tests either penetrate or they don’t. As such, a quality Vulnerability Assessment report will contain few false positives (false findings) while a quality Penetration Testing report should contain absolutely no false positives. (though they do sometimes contain theoretical findings).

The quality of service is determined by the talent of the team delivering services and by the methodology used for service delivery. A team of research capable ethical hackers that have a background in exploit development and system / network penetration will usually deliver higher quality services than a team of people who are not research capable. If a team claims to be research capable, ask them for example exploit code that they’ve written and ask them for advisories that they’ve published.

Service quality is also directly tied to threat capability. The threat in this case is defined by the capability of real world malicious hackers. If testing services do not produce a threat level that is at least equal to the real world threat, then the services are probably not worth buying. After all, the purpose for security testing is to identify risks so that they can be fixed / patched / eliminated before malicious hackers exploit them. But if the security testing services are less capable than the malicious hacker, then chances are the hacker will find something that the service missed.