A Quality Penetration Test

Someone on the pen-testing mailing list asked me to write an entry about the difference between vulnerability scanning (and services that rely on it) and Real Time Dynamic Testing™. This entry is a sanitized description of a real Advanced External Penetration Test that our team delivered to a customer. Many details were left out and our customer’s information was removed or augmented to protect their identity. Our customer did approve this entry.

Our team (Netragard, LLC.) was hired to perform an Advanced External Penetration Test as a follow-up engagement to a pen-test that was delivered by a different vendor. This might seem unusual, but we get these types of engagements more and more frequently. This test was no different than most of them, and we found significant exploitable vulnerabilities that the other vendor missed entirely, which unfortunately seems all too common.

When we deliver Advanced services we expose our customers to specific type of threat. Our goal is to create a threat that is a few levels higher than what they would likely face in the real world. Testing our customers at a threat level that is less than that would do nothing to help them defend against the actual threat. Our services are not the product of automated vulnerability scanners and scripts; they are the product of human talent.

During this particular engagement we were authorized to perform Distributed Metastasis, Covert Testing, Social Engineering, Malware Deployment, ARP Poisoning, etc. All targets were also authorized and included Web Servers that were hosted by third parties, Web Servers that were hosted locally, VPN end points, FTP servers, IDS systems, DNS servers, Secure Email Servers like tumbleweed and so on. We were not given a list of IP addresses to target, we had to identify them and request approval.

We began the engagement by performing covert social and technical reconnaissance. Reconnaissance is the military term for the collection of intelligence about an enemy prior to attacking the enemy; in this case our customer was the “enemy”. Our philosophy is that we cannot produce an accurate threat level without first understanding some details about our target’s political structure, social behavior, and technology infrastructure. We might not use all of the information that we collect while testing, but more times than not it provides us with a good idea of what will be effective, and what will not.

During reconnaissance we focused on two separate target groups. The first target group was the social structure of the client’s employees that we felt was of interest. As such we collected information about those employees that included office-location, telephone extensions, email address, relationships to other employees, friends outside of work, etc. Our secondary sets of targets for reconnaissance were technical targets. Those targets included the identification of servers used by the client, vendor identification, partner identification, the identification of IP addresses belonging to the client, the internal IP addressing scheme, operating system information, patch frequency information, etc.


We were able to use the information collected during reconnaissance to begin performing vulnerability identification through analysis. Because this service was an advanced service and required covert testing, vulnerability identification was mostly done with manual testing (Real Time Dynamic Testing™) and during reconnaissance. As testing progressed we increased our noise level until we received notification from the customer that we’d been detected. This enabled us to identify what level of testing was considered “flying below the radar” and what level was “tagged”. (Knowing this enables us to help our customers retune their IDS technologies so that they are more difficult to evade. In most cases IDS technologies are not tuned properly, and yes this includes IPS and Correlation Systems too.)

Once we were finished with vulnerability identification we built a target matrix that was organized by probability of penetration. This matrix is used as a guide for the team and enables us to test the most probable points of entry first, and the least probable points of entry last. In the case of this particular customer we identified three probable points of entry along with a few other basic vulnerabilities like Cross-site Scripting, etc. (While Cross-site Scripting is useful for performing Social Engineering based attacks, we won’t go into the details about how we used them here.) The other vendor even with basic scanning services should have detected most, if not all of these vulnerabilities, but they didn’t.

The first point of attack that we focused on was the customer’s corporate website. This website was being hosted by a third party and was using a Content Management System (“CMS”) that was created by vendor that we’ll call the Noname Group. This particular CMS was written entirely in PHP, was closed source and had no security functionality to speak of. There were multiple points were unchecked variables were passed directly to SQL statements or other critical internal application components. We were able to use those unchecked variables to penetrate into our Customer’s Web Server and take control of it.

Upon accessing that web server we found customer data that was stored in the database in clear text. This information contained names, addresses, account numbers, social security numbers, etc. In some cases the information was from users requesting information, in other cases it was users looking to sign up. As a proof of concept we wrote a ruby script that would automatically dump the contents of the database when executed. That script was submitted to the customer. Because this server was not hosted within our Customer’s IT Infrastructure it did not provide us with a platform from which we could perform Distributed Metastasis.

The next target lined up for testing was another Web Application, this time it was hosted from within our customer’s infrastructure. Again, the application suffered from a basic SQL Injection vulnerability that could be triggered by a back-tick. We used the vulnerability to fingerprint the application’s backend database and learned that it was a MS-SQL database. We also learned that was hosted on a separate server from the Web Server. We then tested for “xp_cmdshell” access and found that the “sa” user had no password set and as a result we could execute arbitrary commands against the database server with administrator privileges.

Once we gained control over the database server we began to examine other systems within proximity to our new point of control (Distributed Metastasis). That was when we learned that we’d compromised a key server that was deep within the customers IT Infrastructure and had clear access to other critical systems. We also noticed that the server that we were controlling contained multiple databases that contained a wide variety of highly sensitive information including customer banking information, social security numbers, etc. In addition, while performing network probes we identified a secondary database server. Ironically this second database server was running on the web server with the SQL Injection vulnerability that we’d just attacked.

When we tried to connect to the second database server from the internal server we were unable to access it because this time the “sa” password was set and we didn’t know what it was set to. We did however know which system accessed that database server as a result of the Social Engineering efforts that were mixed into our Social Reconnaissance. The system with access was also the third system in our targeting matrix and contained another vulnerable Web Application. This time, due to the configuration of the application SQL Injection capabilities were limited. We did however manage to find an arbitrary file read vulnerability and were able to use it to read the application’s configuration file that contained the “sa” password.

This enabled us to go back to the previously inaccessible database and access it using the sa password. This also gave us access to the xp_cmdshell function that in turn allowed us to execute arbitrary commands against the system. At this point in the test we’d managed to penetrate into both the DMZ and the corporate LAN which also allowed us to connect to any other system within proximity without issue. In other words, there was no internal segmentation in the form of VLAN’s or physical isolation. The networks were flat.

The server that we penetrated in the LAN contained a SAM file. We were able to crack 90% of the passwords in that SAM file with rainbow tables, including the Administrator password. Once we had that password we were able to use RDP to access the Active Directory server and it was technically game over. If we had not discovered the SAM we were prepared to perform ARP Poisoning to collect data and possibly in-transit credentials. Our penetration of the AD server concluded the penetration test.

It is important to note that this is not a complete description of all of the testing that we did for the customer. As with any engagement we produce a deliverable that outlines all discovered points of risk with their respective methods for remediation. In this particular case our report identified 47 risks and provided 47 methods for remediation. Remember that this customer just completed a penetration test from a different vendor, how is it that they missed 47 risks? Their services certainly did not protect our customer from hackers.