The Ian Amit Spectrum of Pentesting Efficacy

It’s been a while since I posted (duh), but recently I’ve had something brewing in my mind that appeared to not have been clearly discussed before, so here goes.

I’ve been seeing some discussions and ambiguity around pentesting, vulnerability assessment, and red teaming (again – no huge shocker for those of us in the industry). However, as much as the “look at our shiny new red team” marketing BS coming from big companies (read as “we do pentesting, we have a new name for it so you can pay more”), pisses me off, what bugged me even more is the lack of clarity as to where and when pentesting can/should be used, and through which means.

I offer you this – my simplified spectrum of pentesting efficacy.

In short, here’s how this works: first identify the actual need for the test. There should be three categories as follows:

1. Testing because you have to (i.e. compliance). PCI is a good example here. It’s something you can’t avoid, and doesn’t really provide any real value to you (because of the way it is structured, and as we all know, compliance/regulation has nothing to do with security so you might as well ignore it.
2. Testing because you want to make sure that your controls are effective, and that your applications are built properly. This is where the “meat” of your pentesting should come into play. This is where you see direct value in identifying gaps and fixing them to adjust your risk exposure and tolerance (based on your threat model and risk management, which you should have, or if you don’t, just go ahead and quit your job).
3. Testing to see how you fare up against an actual adversary. Bucket 2 above was fairly technical in its scope and nature. This bucket is threat focused. More specifically – threat actor focused. Your risk management strategy should identify the adversaries you are concerned about, and here you want to see how you fare up against them in a “live fire” scenario.

Once you have these, you are almost done. Here’s how you approach “solving” for each category:

1. Compliance: find the cheapest vendor that’ll check off the box for you. Don’t be tempted for cheap marketing tricks (we red team, we thoroughly test your controls, we bring in the heavy hitters who have spoken at BlackHat and DEFCON and the super-underground conferences). Remember – you are getting no security value from here, so shop around and see who will tick the box for you. Remember to be heavy handed on the reporting and remediation, as if you are doing things correctly, the scope should be very minimal (remember – just enough to cover compliance requirements) and you should easily have these covered as part of your standard security practice.
   Also – no point of putting any internal resources into here since it won’t challenge them and is menial work that should be outsourced.
2. Controls and Applications: this is where you should invest in your own resources. Send your security engineers to training. Build up an SDLC that includes security champions and involves the dev teams (DevSecOps anyone?). This is where you should see the most value out of your investment and where your own resources are better equipped to operate as they are more familiar with your operating environment, threats, and overall risk prioritization. In this category you’ll also sometimes include testing of 3rd parties – from supply chain to potential M&As. Use your discretion in choosing whether to execute internally or engage with a trusted pentest company (make sure you utilize the Penetration Testing Execution Standard when you do).
3. Adversarial Simulation: This is where you shift from pentesting to red teaming. The distinction is clear: pentesting focuses on one domain (technical), and sometimes claims to be a red team when phishing is involved (social). A red team engagement covers three (technical, social, physical), and more importantly, the convergence of two or three domains (social/physical, technical/social, technical/physical, or all three). This is where you should engage with a company that can actually deliver on a red team (again – use the PTES sparingly in helping you scope and set expectations for the engagement), and can work with you to identify what kind of adversary they should be simulating, how open or closed does the intelligence gathering works, how engaged will they get with targets, and to which degree are you ok with potential disruptions. This is where you’ll see value in affirming your security maturity across several domains, and how these fare up against your threat communities. This will also enable you to more clearly align your investments in reducing exposure and increasing your controls, monitoring, and mitigations for actual loss scenarios.

I did mention vulnerability assessments initially, and if you made it through here you noticed it’s not on the spectrum. Well, it kind of is – it should be part of all of the engagement types, and is not considered an independent member of an engagement. Hint – never outsource this. It’s extremely cheap to engage in VA by yourself, and there are plenty of tools/programs/services that are highly effective in continuously performing VA for you that range from free to very cheap. Don’t be tempted to pay any premium for it.

That’s all for now – hope this made some sense, and helped you in prioritizing and understanding how pentesting (and red teaming) are best applied through your security program!