Using Behavioral Analysis To Discover Undetected Malware

This post exists to flesh-out an outline in “Is Anti-Virus Dead?” The outline:

Preventative and defensive measures:

• Patch vulnerabilities.
• Use anti-virus software with pattern matching technology to detect known exploits of vulnerabilities, even those you have patched. Prevent the exploits from executing, even those that will fail because the vulnerability has been patched.
• Block access to known malware distributors.
• Remove unnecessary services and ports.

Supplement these preventative measures with reactive discovery measures.

• Use behavioral analysis technology (you are here)
• Use analysis technologies that do not require behavioral analysis, such as those described in “What’s Different About This Approach?” to detect unknown exploits of unpatched vulnerabilities.

The “analysis technologies that do not require behavioral analysis” is meant to be novel and suggestive. The other sections exist as reminders that a single focus is insufficient.

Egress filtering. That is, watch the nature of the traffic sent out and watch where the traffic goes.

Snort, from Sourcefire. Inbound, implement rules (another set of signatures) that detect the use of exploits. Outbound, implement rules that alert when connections are made to addresses known to host malware (or other prohibited addresses).

See CS 646: Manual Intrusion Detection for a PowerPoint presentation of IP and TCP headers and the effects of invalid header contents (what a smurf attack looks like, for example).

Get baseline statistics about machines and ports that you can expect to have large volumes of network activity (using, for example, Cisco IOS Netflow or Netwitness). Investigate aberrations.

Nessus, from Tenable (for commercial use); otherwise from Tenable. You’ll get a vulnerability scanner you can use to audit your patch deployment mechanism. You’ll have a way to audit for peer-to-peer file sharing.

Monitor attempts to connect to the master servers of botnets (Command and Control or C&C servers). The University of Washington maintains a list of IP addresses used by botnets. An outbound request indicates a compromised (bot-infected, pwned) machine. Appropriate measures would be to block outbound access to these addresses at the proxy servers and monitor attempts to use them (at routers and at proxy servers).

See Busy Firewall Administrators Note for quick tips.

Use known bad site lists and your DNS to watch for attempts to resolve the names of domains that are known to be malicious. F-Secure reports that in addition to suspicious-looking domain names (such as weloveusa.3322.org), malware may “phone home” to typo-squatting domain names (domain names that resemble legitimate domain names) such as ip2.kabsersky.com.

Watch for responses to DNS requests, where the response returns more than two resource records. This would normally indicate a zone transfer, or the DNS request was to resolve an IRC host, and could indicate that the requester is bot-infected.

Vendor suggestions: Lancope. I have seen only presentations. StealthWatch uses LanFlow messages to make that correlation between the attention-grabbing event and the source machines and users. That’s a lot of LanFlow messages.