Tuesday, January 20, 2009

A Quality Penetration Test

Someone on the pen-testing mailing list asked me to write an entry about the difference between vulnerability scanning (and services that rely on it) and Real Time Dynamic Testing™. This entry is a sanitized description of a real Advanced External Penetration Test that our team delivered to a customer. Many details were left out and our customer’s information was removed or augmented to protect their identity. Our customer did approve this entry.

Our team (Netragard, LLC.) was hired to perform an Advanced External Penetration Test as a follow-up engagement to a pen-test that was delivered by a different vendor. This might seem unusual, but we get these types of engagements more and more frequently. This test was no different than most of them, and we found significant exploitable vulnerabilities that the other vendor missed entirely, which unfortunately seems all too common.

When we deliver Advanced services we expose our customers to specific type of threat. Our goal is to create a threat that is a few levels higher than what they would likely face in the real world. Testing our customers at a threat level that is less than that would do nothing to help them defend against the actual threat. Our services are not the product of automated vulnerability scanners and scripts; they are the product of human talent.

During this particular engagement we were authorized to perform Distributed Metastasis, Covert Testing, Social Engineering, Malware Deployment, ARP Poisoning, etc. All targets were also authorized and included Web Servers that were hosted by third parties, Web Servers that were hosted locally, VPN end points, FTP servers, IDS systems, DNS servers, Secure Email Servers like tumbleweed and so on. We were not given a list of IP addresses to target, we had to identify them and request approval.

We began the engagement by performing covert social and technical reconnaissance. Reconnaissance is the military term for the collection of intelligence about an enemy prior to attacking the enemy; in this case our customer was the “enemy”. Our philosophy is that we cannot produce an accurate threat level without first understanding some details about our target’s political structure, social behavior, and technology infrastructure. We might not use all of the information that we collect while testing, but more times than not it provides us with a good idea of what will be effective, and what will not.

During reconnaissance we focused on two separate target groups. The first target group was the social structure of the client’s employees that we felt was of interest. As such we collected information about those employees that included office-location, telephone extensions, email address, relationships to other employees, friends outside of work, etc. Our secondary sets of targets for reconnaissance were technical targets. Those targets included the identification of servers used by the client, vendor identification, partner identification, the identification of IP addresses belonging to the client, the internal IP addressing scheme, operating system information, patch frequency information, etc.

We were able to use the information collected during reconnaissance to begin performing vulnerability identification through analysis. Because this service was an advanced service and required covert testing, vulnerability identification was mostly done with manual testing (Real Time Dynamic Testing™) and during reconnaissance. As testing progressed we increased our noise level until we received notification from the customer that we’d been detected. This enabled us to identify what level of testing was considered “flying below the radar” and what level was “tagged”. (Knowing this enables us to help our customers retune their IDS technologies so that they are more difficult to evade. In most cases IDS technologies are not tuned properly, and yes this includes IPS and Correlation Systems too.)

Once we were finished with vulnerability identification we built a target matrix that was organized by probability of penetration. This matrix is used as a guide for the team and enables us to test the most probable points of entry first, and the least probable points of entry last. In the case of this particular customer we identified three probable points of entry along with a few other basic vulnerabilities like Cross-site Scripting, etc. (While Cross-site Scripting is useful for performing Social Engineering based attacks, we won’t go into the details about how we used them here.) The other vendor even with basic scanning services should have detected most, if not all of these vulnerabilities, but they didn’t.

The first point of attack that we focused on was the customer’s corporate website. This website was being hosted by a third party and was using a Content Management System (“CMS”) that was created by vendor that we’ll call the Noname Group. This particular CMS was written entirely in PHP, was closed source and had no security functionality to speak of. There were multiple points were unchecked variables were passed directly to SQL statements or other critical internal application components. We were able to use those unchecked variables to penetrate into our Customer’s Web Server and take control of it.

Upon accessing that web server we found customer data that was stored in the database in clear text. This information contained names, addresses, account numbers, social security numbers, etc. In some cases the information was from users requesting information, in other cases it was users looking to sign up. As a proof of concept we wrote a ruby script that would automatically dump the contents of the database when executed. That script was submitted to the customer. Because this server was not hosted within our Customer’s IT Infrastructure it did not provide us with a platform from which we could perform Distributed Metastasis.

The next target lined up for testing was another Web Application, this time it was hosted from within our customer’s infrastructure. Again, the application suffered from a basic SQL Injection vulnerability that could be triggered by a back-tick. We used the vulnerability to fingerprint the application’s backend database and learned that it was a MS-SQL database. We also learned that was hosted on a separate server from the Web Server. We then tested for “xp_cmdshell” access and found that the “sa” user had no password set and as a result we could execute arbitrary commands against the database server with administrator privileges.

Once we gained control over the database server we began to examine other systems within proximity to our new point of control (Distributed Metastasis). That was when we learned that we’d compromised a key server that was deep within the customers IT Infrastructure and had clear access to other critical systems. We also noticed that the server that we were controlling contained multiple databases that contained a wide variety of highly sensitive information including customer banking information, social security numbers, etc. In addition, while performing network probes we identified a secondary database server. Ironically this second database server was running on the web server with the SQL Injection vulnerability that we’d just attacked.

When we tried to connect to the second database server from the internal server we were unable to access it because this time the “sa” password was set and we didn’t know what it was set to. We did however know which system accessed that database server as a result of the Social Engineering efforts that were mixed into our Social Reconnaissance. The system with access was also the third system in our targeting matrix and contained another vulnerable Web Application. This time, due to the configuration of the application SQL Injection capabilities were limited. We did however manage to find an arbitrary file read vulnerability and were able to use it to read the application’s configuration file that contained the “sa” password.

This enabled us to go back to the previously inaccessible database and access it using the sa password. This also gave us access to the xp_cmdshell function that in turn allowed us to execute arbitrary commands against the system. At this point in the test we’d managed to penetrate into both the DMZ and the corporate LAN which also allowed us to connect to any other system within proximity without issue. In other words, there was no internal segmentation in the form of VLAN’s or physical isolation. The networks were flat.

The server that we penetrated in the LAN contained a SAM file. We were able to crack 90% of the passwords in that SAM file with rainbow tables, including the Administrator password. Once we had that password we were able to use RDP to access the Active Directory server and it was technically game over. If we had not discovered the SAM we were prepared to perform ARP Poisoning to collect data and possibly in-transit credentials. Our penetration of the AD server concluded the penetration test.

It is important to note that this is not a complete description of all of the testing that we did for the customer. As with any engagement we produce a deliverable that outlines all discovered points of risk with their respective methods for remediation. In this particular case our report identified 47 risks and provided 47 methods for remediation. Remember that this customer just completed a penetration test from a different vendor, how is it that they missed 47 risks? Their services certainly did not protect our customer from hackers.


  1. Adriel,
    Thanks for a very interesting post!
    Just to get some feel for the scale of this exercise, what can you say about the size of the team, the hours of effort required, and the duration of the exercise in this particular case.

  2. In this particular case the team included two primary players and myself. I was involved only partially as we had several other engagements going on at the same time that also required my attention. The total testing time took 8 days and report authoring took 5 days (because of analysis etc). The average total time for an engagement like this is about 3 weeks if you include issues and resolution (usually because we need to wait for authorization to launch specialized attacks).

  3. A few comments. I feel like I'm picking on you, but believe me I'm not trying to. :)

    1) You make it sound like you essentially investigated (stalked) outside of work various employees of interest ("friends outside of work"). Is your customer authorized to allow you to do something like that? If so, can I engage your investigative services on some of my more attractive peers? :)

    2) You take an extreme stance on demonizing automated tools. So does this mean you used zero automated tools in this assessment? And was the only reason you maybe didn't use them because it needed to be more covert?

    3) I like the story, but at the end of the day, another vendor could come in and find yet another 47 things you missed and create the same argument. In fact, automated tools most likely will find things you miss in a blind, manual testing. This would depend on the scope and intention of the engagement, but were you able to inform the client of their patch levels on the servers? And, I'm not sure you know this, but finding blank "sa" passwords could have been done in moments using automated tools. :)

    4) What did the customer pay you as opposed to the previous vendor who gave them less in return? Was the scope and detail of the engagement the same? I just want to eliminate the customer as being the problem as they may have not given the previous vendor as much scope as they did you, and maybe only cost a fraction what you did. I would be extremely surprised that both engagements were conducted under the same variables with only the manual vs automated testing being different.

    My point is not to say manual testing is archaic and you should use automation. I'm saying there is a time and place and value for both.

  4. Loner, great comments and I don't feel like you're picking on us. In fact, I rather appreciate your critical point of view. Here are the answers to your questions.

    1-) When we perform Social Reconnaissance we'll do whatever our customer allows us to do. We can stalk people, put GPS transponders on cars, etc. To date we've done it once, and yes we're always authorized.

    2-) It seems that my position on automated tools wasn't clear. I don't have an issue with automated tools, but I do take issue with using their product as a deliverable. Automated tools are useful for saving time but they are not effective as a standalone solution.

    3-) Actually that's just not true. We've been first in line and in almost all cases other vendors found that we missed nothing (only comparing same scope projects here).

    With that said, it is technically impossible for anyone, anywhere to find 100% of the *unknown* vulnerabilities. Who knows, we might finish a penetration test and two weeks later someone writes a nice remote for IE. There are just too many variables there... that's why everyone is hackable.

    4-) We were given the contract that the previous vendor gave them to help us scope the project. We ended up charging about 6 times more than the other vendor, but this time we tested them the way that they wanted to be tested. Based on the report (we saw that too) the other vendor ran an automated scanner, vetted the results, and produced a deliverable. Somehow they missed things like SQL Injection triggered by a "`" and very basic XSS. During service delivery we also ran some automated scanners, those scanners missed the same vulnerabilities. Hence why we don't trust scanners and we rely on our own Real Time Dynamic Testing.

    With respect to the "sa" password detection, the scanners didn't pick it up. In fact they had no way to pick it up because from the outside there was no direct access until after we'd penetrated the servers. Hope that made sense... I'm pressed for time right now.

  5. Thanks for the responses, I appreciate it! :)

  6. Great post. Thanks a lot. Has there ever been a time when you couldn't find any flaws in your customer's security at all? If so, did you still get paid for it?

    This is exactly the kind of work I want to get into after university. Is there any advice you can give me on how to get on the right track?

    You don't have to give me a massive guide or anything but I just wanted to know basic things like whether its better to take a degree in computing science as opposed to computer security or whether I'm better off getting all the experience I can in things like an apprenticeship or getting a small job as a systems administrator.

    Thanks for any help you can give.

  7. I'm going to butt in and go @Peter just because I happened to read the comment. :)

    - Exp. as a tech in a company is very valuable be it help desk, administrator, systems, network, or even web/software dev. You'll see the problems firsthand, the challenges in solving them, and likely see what is common everywhere (and why I suspect Adriel has never been on a pen-test that wasn't successful to some satisfactory degree...reality of systems is sobering).

    - pen-test everything and dig into the technology. Pen-test your system, a lab of your own, your friends' systems. If you have a job where you manage systems, take some free time (even a weekend with your boss's approval) and scan non-critical systems, evaluate them, and take every opportunity to investigate incidents and new security vuln announcements

    And so on... :)

    I love the idea of apprenticeships, but I don't think many offer them. Make the entry levels do the automated tasks and run around with kismet, maybe even make reports and manage the cracking pieces, or take notes during interview phases. They might not like it, but that might be enough to gain confidence and start innovating in their own ways, picking up tidbits, and maybe beginning to add more value to the team. Or, worst case, they leave to other waters as a better security expert...

  8. Thanks for the great advice LonerVamp.
    I was planning to set up my own test lab this summer when I have the time and the money. I haven't looked in to apprenticeships but you're probably right that there isn't many around for this kind of thing. I understand its going to take me a while to get to where I want to be and this probably isn't the kind of job that I can simply jump into after college but it's something I'm very pationate about and sure I want to do as a job.

    Adriel, this has been a very useful post to me as its helped me to be sure of what career I want. I respect anyone that can pull off a deep and thorough pen test like that.