Penetration Testing

Penetration testing does not reduce risk.

(Pause for dramatic effect.) Expect the objection “but doesn’t a penetration test discover vulnerabilities?”

Penetration testing discovers vulnerabilities. Discovering vulnerabilities doesn’t address the vulnerabilities. Addressing the vulnerabilities reduces risk. If the name of the game is discovering vulnerabilities, is a penetration test the most productive approach?

Penetration testing (pentest) receives a disproportionate amount of attention. Penetration testing does not reduce risk or increase security. Penetration testing actually increases the chance of information disclosure and application failure. Penetration testing introduces risk. Penetration testing hits the charts at number 20 (sharing that slot with Red Team Exercises) in the SANS/CIS Top Twenty Critical Security Controls.

Attempting to punch holes in a bucket does not harden the bucket. Failing to punch holes gives confidence about the bucket’s integrity.

So why do penetration testing at all? There are roles for penetration testing. Penetration testing provides some confidence that you are effectively implementing the other security controls. A penetration test may reveal gaps in your coverage. Remediating these gaps (working on the other security controls) would reduce risk and make you more secure. A penetration test is not a rigorous examination of the other security controls. You should not expect a penetration test to reveal all weaknesses.

A penetration test will be required for many regulations and contracts. Agreements, such as PCI, require penetration tests.  An agreement could not say “implement standard security controls well,” but an agreement could say “implement standard security controls in a industry-approved manner” and here a penetration test is a mechanism to verify an implementation. A penetration test is a more reasonable requirement than a system audit. If, for example, the Payment Card Industry required a system audit of any organization which would accept credit cards, the time and expense of the system audit would discourage organizations from accepting credit cards. The penetration test is a reasonable measure of establishing confidence.

Penetration tests provide some confidence that you have implemented other security controls well. Familiarity with penetration testing procedures can drive home the need to focus upon the other critical security controls.

A more rigorous approach to identifying gaps and weaknesses is to review the other security controls to determine their completeness. In other words, check your work. Do your own system audit. Like other complex issues, break the security problem into its constituent parts and address those parts.

See “Could we have prepared for this? Attack Simulations for Blue Team Hardening,” a SANS webcast presented by SANS instructor Alissa Torres. Alissa describes an Attack Simulation approach that seems to be more “agile” than penetration testing or red team exercises. There are things in the presentation that I world disagree with. Most importantly, an Attack Simulation should begin early in an organization’s maturity along with tabletop scenarios and developing an incident response procedure.

Web application vulnerability testing is often confused with penetration testing. Application software security, including web application security, is Security Control 18. Black box web application testing may be a step in a penetration test. When someone’s job title is “web application penetration tester,” there is confusion about what the job requires but the job has a flashy title.

The more important, the lower numbered, security controls should receive the majority of attention. Vulnerability scanning and remediation (Security Control 4) should occur on the scale of once a month. Vulnerability scanning should consist of configuration verification (Security Control 3) and software version (including patch level) verification. Check your work. If configurations seem weak, examine how the configuration came to exist. If changes failed, investigate why changes failed. Investigate which environments were missed (Security Controls 1 and 2). Alert when administrative privileges are used and follow up on those alerts (Security Control 5). Continue with the remaining of the SANS/CIS Top Twenty Critical Security Controls.

Implement the basics well and check your work. A penetration test will fail to reveal an exposure.

The boundaries of penetration testing must be clearly defined. A penetration test is what the contract says it is.

Case in point: the episode 229 podcast of PaulDotCom, in which the customer acknowledges that their ID card readers have a known vulnerability, tells the pentester “you can’t test that” … which evokes shameless derisive laughter. But why should it? Isn’t the customer trying to find out what they don’t know? Aren’t they trying to identify where their other critical controls are incomplete? Replacing an ID card reader system is an expensive process; knowing that it has a vulnerability doesn’t make a budget appear. Exclude that known vulnerability from the penetration test. A penetration test is what the contract says it is.

See also:

This entry was posted on Wednesday, February 9th, 2011 at 9:50 PM and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

Comments are closed.