So I've been participating in the penetration testing mailing list that is hosted by securityfocus and I can't say that I am impressed. In fact, I might even go so far as to say that I am concerned about the caliber of the people that are offering paid services, here's why.
When a customer hires a security professional to perform a Penetration Test, Web Application Security Assessment, or any other service that customer should be getting a real expert. That expert should be able to assess the customers target infrastructure, application, or whatever and should be able to determine points of vulnerability and their respective risks. But that is not what I am seeing.
The other day a self proclaimed "expert" asked how dangerous a SQL Injection vulnerability was. They apparently identified a SQL Injection vulnerability in their customer's website but didn't know what to do with it!!! They also asked about how to exploit the vulnerability and what successful exploitation might do.
Well the first thing that came to mind was "Why the hell are you offering services if you don't know what you are doing?". I actually asked that but I didn't get any response back from the original author. When someone hires a security professional to deliver security services they expect those professionals to be subject matter experts. The unfortunate thing is that in most cases the customer has no way of verifying the professional's expertise and the customer gets taken for a ride. (Take a look at our white papers!!!)
Another example is in a recent vulnerability that one of my team members found. He was researching a product's webservice and found that it was just chalk full of holes. When he contacted the vendor, they responded with "but we just had a very extensive security assessment done against our product". We certainly couldn't tell... looks like they got taken for a ride like so many others.
Why is this a problem, why do I care? Its a problem because the providers who offer these low quality services advertise the same way as the high quality providers. The difference is that their service doesn't do anything to protect the customer, and ours does. We're not the only good security company out there, but we are one of very, very few.
Adriel,
ReplyDeleteI pretty much have the same concerns as you have. I'm from Brasil and provide specialized security services over here. I face the same situation as you described.
Of course beginners must have their chance into the market, but what I see are many fresheners and curious-kind people that start selling all type of assessment, without a drop of knowledge.
The worst thing is that they charge lower price, get hired and deliver nothing. Then, we need to reconquer customers and explain a lot of thing in order to be hired and show a real good work.
I think all those tech jobs work like that.. it's part of the game.
I agree with you. Although not a "Pen Tester" myself, my skills lie on the process of security, several of my clients have previously brought into the sales/scare talk of some "IT Security Companies" In some cases 5 figure sterling somes have been handed over for poor quality, application produced reports that have not had any sort of false negative/positive cleansing carried out.
ReplyDeleteThe reports have been so technical as to be useless to the board and even the IT team.
How do we solve it? As we find companies that have had poor service, advise them of this, and let their contracts management/legal team handle it.
I am a "n00b" in regards to the security/pen test field, and subscribe to the SF list to learn. Some of the mail i've seen is just ridiculous - if people with a seemingly lower intelligence than myself can run a business and be (financially) successful in this field, well... I don't know why *I* haven't started my own company yet! Of course I wont do that, I value knowledge, but it seems like there are a lot of people out there who are bullsh***ng their way through this area of expertise with relative ease. It is worrying.
ReplyDeleteWell, at least in the future, if I keep up the learning process, I'll be seen as a relative genius compared to the majority :P
As not a n00b and probably not that smart; I do see your point, as pen testers grow up and find themselves contracted to go beyond layer 3. Many if not most do not have the expertise to deal with nuances of the application in question. Network Pen Testers do not have have agood understanding of why things work above layer three. The point of not knowing how to proceed just enforces my belief that there is a lack of security good practice maturity not from a technical as this is a specific skill set, but from a security administrative view. Having a test plan specific and a action set based on finding is as important as being the bits and bytes protocol expert. The other issue that is driving this proliferation of inexperience is testers running pen scripts and tools that have no idea what that script or tool is going to do, as most likely they did not write it. Knowing the right people and being armed with scripts DOES NOT MAKE YOU A SECURITY PERSON. If you have the ability to bull**** your way in and then don’t care if your due care integrity is not up to par P.T. Barnum once said “ There is a sucker born every minute” If the client doesn’t understand Due Diligence then they have a high probability of getting what they paid for.
ReplyDeleteCertainly very true. However, customers get what they are paying for. You later mention arming customers against those so-called experts. Yes, but if they don't want to. That's a entire matter to explore...
ReplyDeleteI mean security services is an open, unregulated market. Certifications don't mean anything when it comes to technical services, typically pentesting. Clients do not have any comparison scale or any valuable metric to begin with. Is it going to be solved one day ? Well, don't think so.
Therefore, the only way customers can get out of this situation is by integrating this as part of their security policy. They have to continuously assess and challenge security practitioners. As being part of the way they build their security. Such as ordering one or two additional pentests a year from different companies and look at the difference. As Richard Bejtlich just pointed out, "You don’t get what you expect, you get what you inspect"
To get a broader picture, what you raise is a typical issue of information asymmetry. Markets where there is a fairly important asymmetry between customers and suppliers tend to fall down to the lowest quality because customers can't tell the difference between actors, and pricing being the very lats and only information they get. You can find a very good analysis of this in a paper called "The Market for Lemons". George Akerlof got Economic Sciences Nobel prize in 2001 with two fellow researchers for their work on that particular topic, ending up with a complete theory.
The main idea behind this paper is that lower quality products tend to eventually evict high quality products because customers can't tell the difference between the two, and tend to choose lower pricing as a safe posture: "I have a good chance to get screwed, so I don't put big money on it". This kind of situation is referred as Adverse selection.
This does not happen all the time though, and the main factor that decides whether a market will fall into that is the ability for the best quality bidders to give valuable signals to the customers so they can better understand the difference between offers and compare them for what they truly are.
And to say the least, the whole security industry is a gigantic information asymmetry driven market, where the proportion of people trying to push crappy products or services outnumbers good quality services. And you know the best thing about that ? Laws and regulations are actually just making situation worse. Regulations because they can be satisfied with dumb practises (look, we had a pentest, everything is fine). Laws because they prevent people from actually being able to openly have a deep look into security products without getting sued and spend the next ten years in courts...
Just my 0,02EUR :)
Considering the low rates that companies seek for IT security people these days, I'm not surprised that a lot of people still have their butts flapping in the breeze after a brief engagement with an "expert" who agreed to such paltry compensation.
ReplyDeleteI can't tell you how many companies these days want IT security professionals who'll do everything up to CISO level work for as little as $35 an hour. With those wages in mind, it's no wonder these people are getting rank amateurs filling their openings. For my own part, I've turned down so many offers from groups who won't pay more than $75K annually that it's beyond obscene.
Personally, I have come to take some sordid enjoyment from watching these pretenders fall on their faces. Maybe after enough security meltdowns, companies will start paying real professionals what they're worth (for a change).
You have to remember, in this economy people know security is still a hot market and when they need a job will lie about what they can do.
ReplyDeletesad
The frustration is shared although I am not in the penetration testing field or within a reasonable reach of having a clue if the job was presented to me.
ReplyDeleteI do work as a sysadmin in a business dealing with credit card information where the differences in how to do business between myself and the upper management crew are significant.
The latest in a long line of bad decisions has been to send our very non-technical Director of Information Security to get the Certified Ethical Hacker certificate, with the goal of having him perform our internal penetration test as now required by PCI. So, the days before sending him to hacker school *sigh*, us more technical competent individuals were giving crash courses in port scanning and packet sniffing. The whole situation has my self respect hiding in a corner out of sight.
I blame two for this.
One is the auditors deeming our course of action appropriate.
The second is the industry involved with processing this type of personal information. If we as security architects were given the chance to implement the controls we all preach then perhaps the forced regulation would be less of a financial burden on the company. If we had the "freedom" to design according to least privilege etc. then maybe PCI would never have been thought of or needed? So by compromising on today's solutions to save a buck, you end up having to pay what you initially saved plus interest at a later date.
I could go on but I'd just talk in circles. Just believe stress and frustration is every day at the moment.
We need this type of paper as ammunition. I am not suited to argue against their decisions when I am not in the field myself.
When I say a person is incompetent I am asked how I would penetrate our defenses if he can't do it. If I had a clue I wouldn't be working there is my response.
We at Immunity experience these problems often, especially when prospective clients have limited knowledge/experience receiving assessment work. We encourage these clients to conduct background research into the technical capabilities of the consultants themselves (papers, presentations, classes etc). We state that a good vendor will be able to elaborate on what the consultant brings to the table in addition to the tools they use. If the vendor's not got a budget for the consultant behind the tool, they may as well just buy the tool themselves, etc etc. Doesn't mean we always beat out the guy repackaging a vuln scan as a pen test, but it does help get the point across!
ReplyDelete