All posts by iamit

The Ian Amit Spectrum of Pentesting Efficacy

It’s been a while since I posted (duh), but recently I’ve had something brewing in my mind that appeared to not have been clearly discussed before, so here goes.

I’ve been seeing some discussions and ambiguity around pentesting, vulnerability assessment, and red teaming (again – no huge shocker for those of us in the industry). However, as much as the “look at our shiny new red team” marketing BS coming from big companies (read as “we do pentesting, we have a new name for it so you can pay more”), pisses me off, what bugged me even more is the lack of clarity as to where and when pentesting can/should be used, and through which means.

I offer you this – my simplified spectrum of pentesting efficacy.

In short, here’s how this works: first identify the actual need for the test. There should be three categories as follows:

  1. Testing because you have to (i.e. compliance). PCI is a good example here. It’s something you can’t avoid, and doesn’t really provide any real value to you (because of the way it is structured, and as we all know, compliance/regulation has nothing to do with security so you might as well ignore it.
  2. Testing because you want to make sure that your controls are effective, and that your applications are built properly. This is where the “meat” of your pentesting should come into play. This is where you see direct value in identifying gaps and fixing them to adjust your risk exposure and tolerance (based on your threat model and risk management, which you should have, or if you don’t, just go ahead and quit your job).
  3. Testing to see how you fare up against an actual adversary. Bucket 2 above was fairly technical in its scope and nature. This bucket is threat focused. More specifically – threat actor focused. Your risk management strategy should identify the adversaries you are concerned about, and here you want to see how you fare up against them in a “live fire” scenario.

Once you have these, you are almost done. Here’s how you approach “solving” for each category:

  1. Compliance: find the cheapest vendor that’ll check off the box for you. Don’t be tempted for cheap marketing tricks (we red team, we thoroughly test your controls, we bring in the heavy hitters who have spoken at BlackHat and DEFCON and the super-underground conferences). Remember – you are getting no security value from here, so shop around and see who will tick the box for you. Remember to be heavy handed on the reporting and remediation, as if you are doing things correctly, the scope should be very minimal (remember – just enough to cover compliance requirements) and you should easily have these covered as part of your standard security practice.
    Also – no point of putting any internal resources into here since it won’t challenge them and is menial work that should be outsourced.
  2. Controls and Applications: this is where you should invest in your own resources. Send your security engineers to training. Build up an SDLC that includes security champions and involves the dev teams (DevSecOps anyone?). This is where you should see the most value out of your investment and where your own resources are better equipped to operate as they are more familiar with your operating environment, threats, and overall risk prioritization. In this category you’ll also sometimes include testing of 3rd parties – from supply chain to potential M&As. Use your discretion in choosing whether to execute internally or engage with a trusted pentest company (make sure you utilize the Penetration Testing Execution Standard when you do).
  3. Adversarial Simulation: This is where you shift from pentesting to red teaming. The distinction is clear: pentesting focuses on one domain (technical), and sometimes claims to be a red team when phishing is involved (social). A red team engagement covers three (technical, social, physical), and more importantly, the convergence of two or three domains (social/physical, technical/social, technical/physical, or all three). This is where you should engage with a company that can actually deliver on a red team (again – use the PTES sparingly in helping you scope and set expectations for the engagement), and can work with you to identify what kind of adversary they should be simulating, how open or closed does the intelligence gathering works, how engaged will they get with targets, and to which degree are you ok with potential disruptions. This is where you’ll see value in affirming your security maturity across several domains, and how these fare up against your threat communities. This will also enable you to more clearly align your investments in reducing exposure and increasing your controls, monitoring, and mitigations for actual loss scenarios.

I did mention vulnerability assessments initially, and if you made it through here you noticed it’s not on the spectrum. Well, it kind of is – it should be part of all of the engagement types, and is not considered an independent member of an engagement. Hint – never outsource this. It’s extremely cheap to engage in VA by yourself, and there are plenty of tools/programs/services that are highly effective in continuously performing VA for you that range from free to very cheap. Don’t be tempted to pay any premium for it.

That’s all for now – hope this made some sense, and helped you in prioritizing and understanding how pentesting (and red teaming) are best applied through your security program!

Dumpster fires and security incidents

Full disclosure: this post isn’t about security per-se. It’s here because of recent conversations I’ve had with people from outside the immediate security “industry” who wondered about Equifax from a technical perspective, but mostly from a “WTF are these guys smoking” one ;-). I’m also happily not selling any of this (although I did in the past, and ran crisis management for a few major incidents – ones that happily did not end up like Equifax – although had every opportunity to…)

A lot was said and written on the Equifax hack (check out Brian’s coverage for most of it). Mostly about how badly Equifax handled their security by blaming Apache Struts, and having 3 executives dumping stock before the public announcement. How they handled the incident response by working with a 3rd party that leaked the event by registering equihax.com without proper OPSEC. And obvious security issues while working on communicating with their customers such as pointing people to use a newly registered domain name, hosting a fairly static site on WordPress, leaving trails of user config from said WordPress, etc…

But I’m not here to talk about this (again – enough have been said). I’m here to talk about how all these came out to be in the first place. Well not the original breach (that took years of neglect IMHO), but how does a major company like this gets to commit to a string of poor decisions in time of crisis.

And the simple answer is crisis management. They don’t have any. I’m not talking about a “cyber incident plan” that includes communication strategies (which I’m sure all the vendors are hawking these days in light of the major fail that Equifax are exhibiting). I’m talking about properly handling a crisis from a company management perspective.

Companies this size that go through a major breach and do not have a very strong leadership internally, tend to fall into mode of operation where all the executives are taking care of themselves, and have zero to negative cooperation. Everyone is trying to CYA, and rush to the closest “action” they can hide behind while saying “I did my job”. That’s how you end up with registering a new domain to handle the incident and running it on WordPress (instead of using your established “credible” domain).

Crisis management is often done best by bringing an outsider. It can be someone from the advisory board who doesn’t have a direct stake with the company, or just someone who’s JOB is to manage crisis scenarios. Much like the security consultants (that Equifax probably didn’t use as much as they should have), crisis management people come in, and represent the company’s best interest.

Unlike security consultants, the crisis manager is responsible for making decisions, and vetting any action the company does. Everything goes through them. From communications, through legal, to technical remediation. This assures that there is a clear line in how the company operates, there’s an owner to these actions, and the owner can report back to the board with accountability and represent the company’s best interests. Clearly, Equifax did not have any of this. I’m sure they were advised by the incident responders they hired, and potentially by other security consultants. But these don’t have crisis management experience. They lack the perspective, and the breadth of thinking about all the implications, and the solutions they propose are usually scoped to a technical element.

That’s how security incidents turn into dumpster fires. Even when you have the best security professionals working for you on the incident. Companies need to learn, that regardless of their size, for situations that exceed the typical “shit’s broken”, they need professional crisis management help. Just like they get for performing incident response (because they don’t have the skill-set) or forensics (because the don’t have the skill-set). See the trend?

When great ideas go to the wrong places

Or: why attribution is not a technical problem.

TL;DR: hacking is an art and a science, computer attacks (cyber these days) are only one manifestation of an aggressor, which has very limited traits that can trace it to its origin. Relying on technical evidence without additional aspects is not enough to apply attribution, and when done so, attackers can use it to deflect attribution to other actors.

Context: Experts, Microsoft push for global NGO to expose hackers

So, apparently, some really smart people at RAND corporation and Microsoft have decided that they are going to solver the world’s computer Bourne attack problems by creating a new global NGO to unmask and apply attribution to hacking incidents. They claim the organization will be responsible to authoritatively publish the identities of attackers behind major cyber attacks.

Which is really cute when you think about it – a bunch of brainiacs (and Microsoft people) sit around and analyze network, storage and memory dumps to trace back attacks to their origins. Sounds like a really great service, which can be used by companies and governments to trace back who attacked them, and act on it (either by suing, or means of diplomatic recourse).

The only problem is that the attribution game is not won on technical merit only. And guess what? Attackers know that very well. Even the US government knows that (or at least the organization responsible for launching such attacks) and have been trained to study different attacker’s traits and tactics so that they can replicate them in their own attack – hence throwing off attribution if/when the attacks are detected.

The reality of it is that companies are often hired to provide incident response and forensics, and in a rush/pressure to give value to their clients, come up with attribution claims based on technical merits. Cyrillic words will point to Eastern European blame (RUSSIA!). Chinese character in a binary will lead to claiming Chinese hackers are behind an attack. An Iranian IP address linked to a command and control server that trojans connect to will point to an Iranian government operation. Which is all a big steaming pile of horse feces because everyone who’s been on the offense in the last couple of decades (probably more – I can only attest to my experience) also knows that. And can easily create such traces in their attack. Furthermore, for the ones following at home thinking “oh, they know that I know…” – yes, we play that game too, and attackers are also “nesting” their red herrings to trace back to several different blamed parties, and it all depends on how deep the forensic analyst wants to dive in.

The bottom line, is that the technical artifacts of a computer attack are ALL FULLY CONTROLLED BY THE ATTACKER. Almost all forensic evidence that can be found is controlled by a knowledgeable attacker, and should be considered tainted.

Now consider an NGO who have no “skin in the game”, and relies on technical artifacts to come up with attribution. No financial evidence, no political ties, no social and physical artifacts or profiling of suspected targets or persons of interest in the victim organization. Anyone who’s been somewhat involved in the intelligence community can tell you that without these, an investigation is not worth the paper or the bits that are produced during it.

So, sorry to burst another bubble, and actually, if you read the article, you’ll see that I’m not alone, and at the Cycon conference at which this initiative was announced, several others have expressed pretty firm opinions on the futility of this initiative. So as much as I appreciate the initiative and willingness to act and “fix the problem”, perhaps it’s best to actually step out of the fluorescent light and really understand how things work in the real world 😉

PTES, remaining impartial, and insisting on high standards

The PTES (Penetration Testing Execution Standard) is standard that a small group of highly motivated and passionate practitioners have created (and yours truly). As such, it is designed to define how a penetration test should be executed – from start to finish. We tried not to skip a single element. We worked tirelessly to make sure that the standard does not reference any particular vendor or product, as we all believe that a proper penetration test is not about the tools, but more about the content and delivery.
The standard has survived several years of scrutiny and a few rounds of editing and improvements, and have never ever leaned to a specific industry player.
It has been by now adopted by the PCI council as a reference to what a penetration test is, it has been acknowledged by the British Standards Institute and placed in the same class as other standards, often receiving higher praise for its impartiality, practicality and coverage.
For some reason, in the past week or so I was approached by two different vendors, in attempts to either use their platform or writeup about how their suite of products provide “the best coverage for the PTES”. I’m sure that I’m not alone in this.
Just to be clear, I’m including the (slightly modified) answer below, which by now is also the “official” line of the core PTES founding group.

Hi [vendor],
Thanks for reaching out. Unfortunately, the PTES as a standard is not going to endorse any specific product or service. We have a guide section that offers approaches to the execution of the standard (http://www.pentest-standard.org/index.php/PTES_Technical_Guidelines) and I’m sure that your products can fit into _SOME_ of these areas.
I’d suggest avoiding an attempt to portray any product suite as “filling the needs of penetration testing standard” as it would be bound to be criticized and proven otherwise. Additionally, as an impartial member of the standard founders, we are all committed to avoiding any endorsement or participation in a product or vendor specific writeup.
Thanks, Ian

The standard is written for us. Practitioners, customers, organization, anyone and everyone who’s dealing with penetration testing. It is not about a specific product, or even a specific approach or methodology for testing. It’s about defining expectations, and delivery bars. It’s about setting up and insisting on a high standard. Not even a “minimal bar” for delivery. It is designed so that when it is adhered to, the delivery will be well above a “minimal standard”. We do not settle for minimums.
Please don’t settle on your end either.

Infosec conferences/talks redux

Don’t mind me, just poking my head in here to make sure the cobwebs haven’t taken over this place yet 😛
So yes – I’m going to be blogging waaay less then before because of, well, life? But I recently saw a post from Daniel Meissler who discussed how (in)effective are modern security talks at conferences are.
He’s bringing up a couple of great points, and talks about what a good talk in his mind would be. Figured I’d share my 2c on this based on a couple of conferences and talks I’ve been to and delivered.

So, neither approach is useful IMHO (i.e. essay, nor entertainment).
A Dan Geer style essay-reading has zero added value for the participants. Go read it yourself in your own pace and you’ll be better equipped to take something from it.

A handwaving “look at my marketing schtick” presentation has no value without any insights to the thought process behind it. Neither is a talk focused solely on the entertainment value. Even if it seems to veil itself as “but through which you’ll get awareness/education”. Especially if it’s mostly self-serving and designed to make you look good. Go away.

Slides that are visually appealing (cat pics), but that support the narrative of what the speaker is saying would be the best experience for me personally (given that there is actual content, and not just the same regurgitated BS that a lot of talks “innovate/research” with).

So, first – get something new in place.

Ok – go and google that shit. Double time. Because most of what’s been out there recently – from “unveiling” cyber criminal tools and forums, to “new” ways to avoid data exfiltration mitigations, is OLD FUCKING NEWS. You are supposed to be this OSINT Google-foo master. Prove it by not embarrassing yourself with a re-branding of old research.

Now, realizing that you may have no idea how to present this new thing, do two things:

  1. Write a paper that describes said new thing. Keep it fairly academic or white-paper style. This is the “essay” style you keep hearing about. DO NOT TRY TO PRESENT IT. It’ll be boring as fuck, and people will go into hibernation in the crowd.
  2. Start writing the story of how you found said new thing. Take note of the following:
    1. Why did you go out to invent/find said new thing? What was the motivation? What gap does this fill?
    2. How did you go about researching and finding the new thing? What challenges did you face doing so? What didn’t work through your process (much more interesting and relevant than what did work)?
    3. How do you use this new thing? How can I use it (assuming I don’t have to sell a kidney to do so. If so, pass this along to your marketing guys so they can get ready for RSA)?
    4. Show relevant data on how this new thing improved your life (professional life included). Show the situation before, and after new thing was applied. Data is cool, and you can’t argue with it (as opposed to “hey, look at me doing this thing one time with no context and no goal and how badass I am”).
    5. Give credit. Understanding that you are probably not alone researching new thing in complete void – give some props to the people/projects who have inspired you, helped you move along your research, or have done similar things, and you have build on their things to get to your new thing. (i.e. don’t be an asshole).
  3. Take this story now, and tell it. This is your talk. Find visuals that support the narrative of this story. These don’t have to be the text verbatim of what you are saying (please, for the love of god, stop it with the bullet wars). They can be cat pictures, then can be graphs, or funny graphics. Make sure there’s some context between your slides and your story narrative.
  4. Practice going through your talk and telling your story. After a couple of tries, try turning off the slides. Can you still make it work? Do you keep trying to read out from the slides (of course not, because they should only have minimal text on them).
  5. Go talk. It’s going to be great. You are going to stumble on your words sometimes, utter an “Ummm”, and an “Ahhh” from time to time. Nobody really cares. Because they are listening to your story, which is awesome, and interesting, and not reading out of your slides before you can recite them.
    1. (oh, and of course – don’t memorize the thing. You need to be able to tell that story again and again, and never sound the same. Otherwise you could have just sent a pre-recorded and edited copy of you doing this).

I guess it’s easier to say this from where I’m standing (here’s my bias declaration: I’ve done this many times, including bad presentations, and am about to deliver my last talks by the end of the month). But trust me – do yourself a favor and think about what you’d want to see/hear at a conference. It’s that simple. Don’t think about some “rock star” researcher and look up their presentation (they might suck at public speaking), just put yourself in the crowd and think “this is what would have worked for me if I’d want to learn about something”.