Security Patch Management

A standardized patch management process reduces the incremental costs of patch deployment. You can afford to deploy patches with hard-to-quantify risks when your patch management process is well-designed.

Recognize that there is a reluctance to deploy patches. Fight that reluctance by determining why it exists. The view may be “our testing wasn’t rigorous enough.” Then improve the acceptance test scripts. The view may be “it ain’t broke.” That’s simply denial, the belief that because no one has actively complained it doesn’t need attention. It is broke, it needs fixing, get on it.

There is a risk which increases when security patches are released. When security patches are released, expect them to be reverse engineered. If an exploit of the vulnerability is not already available, it soon will be. When reviewing the importance of a security patch, the question “is exploit code publicly available?” is not an interesting question. Act as if exploit code is available.

“Is this vulnerability wormable?” is not an interesting question, either.  Worms were dramatic and news worthy, but worms are rare. It makes very little sense to develop a worm; there’s no money in it. Worms make news, worms get addressed, and worms make no money.

Information gathering makes money. Unauthorized access to information is a significant risk. Disclosed information has no disaster recovery mechanism; it cannot be unseen. Information disclosure often goes undetected. Prevention is your most important consideration when faced with the possibility of information disclosure.

You want to know if your mission critical systems are in jeopardy. Can your data be stolen or corrupted? Where is your data? What would be required to reach it?

Know your environment, which includes knowing your vendors and your application owners and your system owners and your change control process.

Get notified of patch availability (vulnerabilities as well). For Microsoft, the patch notification process is straightforward. Subscribe to each vendor’s vulnerability and patch announcement mechanisms. Subscribe to the Secunia notifications as well.

Check the Open Source Vulnerability Database for additional information about the vulnerabilities being mitigated and their potential risks. Any software vulnerability should be addressed quickly.

Encourage your vendors to adopt an announcement cycle. Don’t accept “when required” or “as needed” as responses. While vendors are caught by surprise when vulnerabilities are found, that unpredictability can be managed for their customers for all but the most troublesome of vulnerabilities. You want to be able to coordinate the availability of test teams.

Plan for the spikes that your support team will go through. For example, you may not know how many patches Microsoft will release on the second Tuesday of the month  but you know that patches will be released. Have support team resources ready.

When updating Java, be sure to uninstall old versions as well.

Pre-patch announcement tasks:

1. Identify your mission critical applications.
2. Baseline resource consumption of test environment.
3. Assign a patch testing team to each mission critical application.
4. Each patch testing team needs their own acceptance test script.

Patch announcement tasks:

1. Does the patch effect resource consumption?
   1. Make sure your baseline of pre-installation resource consumption is current.
   2. Install patch.
   3. Review post-patch resource consumption.
2. Set a deadline for a response; perhaps Friday 2:00pm of that week. For each patch testing team:
   1. Install the patch(es) on a test environment. The test environment should resemble the production environment. The test environment should not be the development environment.
   2. Step through the acceptance test script to determine application compatibility. Do expected results match observed results?
   3. Report pass or fail; either “no compatibility issue found” or “compatibility issue exists.” There is no “sort of” answer. There are no small or minor compatibility issues.

   The application acceptance testing task can be performed concurrent with resource consumption testing.

3. When all patch testing teams report “no compatibility issue found,” follow your change control process. Your change control process includes:
   • Identify affected systems.
   • Is a restart required?
   • What is the roll-off, recovery or uninstall plan?
   • What is the time frame for this change, and has this time frame been communicated to affected parties? Identify any schedule conflicts.
4. Deploy tested patches in stages. Deploy to a representative sample before broad deployment. While your acceptance testing should catch any problems, acceptance tests often need improvement.
5. Report deployment status.Is your patch deployment process a success or failure? Use an independent tool.It is best to not rely upon reports from your deployment tool to evaluate your patch deployment status. You are measuring how effective your deployment is. Don’t rely upon the deployment tool to tell you how well it has done its job. If the deployment tool missed a target environment for deployment, expect it to miss the target environment for reporting as well.

SearchMidmarketSecurity.com has a series of articles by Felicia Nicastro describing the security patch process:

Step by Step: Best practices for security patch testing and management

1. Introduction
2. How to prepare for security patch testing
3. Security patch testing and deployment phase
4. Security patch validation and verification

I prefer to begin testing before scheduling the deployment. You want to know that this mitigation measure is not detrimental to mission critical applications before committing to it as a mitigation measure. Testing in a test environment should be part of the patch evaluation.

Review your exposure. There will always be some exposure when mitigation would be more expensive. Patch installation should never be your only mitigation measure, but patch installation is always an important remediation measure.

Environments, platforms

How does your deployment mechanism scale? A slow distribution is not always a bad idea. Avoid saturating any network pipe. You don’t want to be taking the company out of business in an effort to prevent someone else from taking the company out of business.

Are there any organizational constraints which can effect your deployment mechanism design? For example, are there regional teams which want to do their own scheduling or want to be able to halt a distribution.

VMware Update Manager to patch ESX, VM Operating Systems, and many applications. VMware Update Manager takes a snapshot of the system before the patch is applied, providing you with a roll back mechanism.

A virtual host is as secure as its least secure guest environment. Do not think that patching the host environment replaces patching of guest environments.

The following is not an exhaustive list.

Linux tools:

• Use the Linux distribution’s patches for packages such as OpenSSL and OpenSSH. Since Linux distributions will backport patches, your patch detection mechanism must be familiar with the version numbers implemented by the Linux distribution.
• Cfengine, bcfg2, Puppet, Spacewalk for Fedora and CentOS.
• Red Hat Satellite Server, Up2Date for Red Hat Enterprise Linux.
• APT (Advanced Package Tool) for Debian and Ubuntu.
• YaST for Novell SUSE.
• YUM for AspLinux, Fedora, Yellow Dog, CentOS.

“ScaleXtreme offers the first unified patch management solution for physical, virtual and public cloud servers. Your server can be behind a firewall or on Amazon EC2, our product can patch it and manage it.” For Windows and Linux servers.

Microsoft WSUS or Microsoft SCCM for Windows environments, but you’ll still need to address the non-Microsoft applications (such as Sun Java and the Adobe products). SCCM can be used to update any Windows application, with sufficient effort. Bigfix may be more appropriate. Secunia (and its Corporate Software Inspector (CSI) software) offers patching of third-party applications through WSUS. Microsoft Attack Surface Analyzer can be part of your patch resource consumption testing.

Do not neglect your network equipment. Cisco Router Assessment Tool (RAT) or nipper.

Are there any embedded systems in scope? Embedded systems are often selected because they will require no update. Often they will require security patches. The organization will have no maintenance plan, so this becomes much more than a tool selection problem.

When budgeting, don’t neglect to include the database costs. It is not unusual for Windows solutions to include MSDE at no cost, but recommend SQL Server at a significant cost.