Log Files

“A small daily task if it really be daily, will beat the labors of a spasmodic Hercules” – Anthony Trollope

Investing in an Intrusion Detection System (IDS) is pointless if you have no plans for the log files.

If you already have log files and you are not reviewing them, then your investment in an IDS to create additional log files is less important. The IDS is there to make it easier to detect anomalies. If you already have a mechanism which detects anomalies, such as log file review, but are not using it, then what are you attempting to accomplish with your IDS investment? If you don’t review your IDS log files, then you have not accomplished your IDS tasks event though you have invested in an IDS.

• NIST Special Publication 800- 92, “Guide to Computer Security Log Management” [pdf]
• Syslog Skeet Shooting – Targetting (sic) Real Problems in Event Log
• In general, get Splunk. See the Splunk tutorial at Ethical Hacker Network. See Cooking With Splunk by John Strand at PaulDotCom. Splunk interprets many log files, consolidating their information based upon date and time.
• AlienVault offers a free open source solution.
• An enterprise-size deployment of Hortonworks, ELK, Zepplin, Kibana, X-Pack, Oozie and Spark for log files and visualization.
• logalyze
• Graylog for authentication,  alerting and so forth. real time monitoring
• SIEMonster
• SANS SOF-ELK (Elastic Stack, ELK) for visualization and analytics, ad hoc analysis.
• File beat
• Win log beat
• Logstash to parse and some filtering
• Kibana to search and visualize
• Log Parser is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows® operating system such as the Event Log, the Registry, the file system, and Active Directory®. See Log Parser.
• In specific cases, use Perl and regular expressions (regex). Use Regex Coach, Expresso or RegExr (or RegExr Desktop), which are interactive tools for creating regular expressions. Adopt tested modules from the Comprehensive Perl Archive Network (CPAN) and other published sources, such as Perl-Fu: Regexp log file processing. For example, to find most frequently occurring errors in the web log:
  1. Keep only the error lines
  2. Chop the date off the front
  3. Sort them
  4. Aggregate and count the duplicates

  Sort by number of duplicates

  grep 'on line' /var/log/httpd/error_log | cut -d' ' -f6- | sort | uniq -c | sort -nr

• Swatch Event log monitoring
• EventID.net for Windows event log interpretation (and other respources)
• WhatsUp Log Management Suite
• Sagan Log Monitor 0.2.1 is a multi-threaded, real-time system and event-log monitoring software. It uses a “Snort” like rule set for detecting nefarious events happening on your network or computer systems. If Sagan detects a “bad thing” happening, it can do a number of things with that information. Sagan can also correlate the events with your Intrusion Detection/Intrusion Prevention (IDS/IPS) system and basically acts like an SIEM (Security Information and Log Management) system.

How does log file management differ from Security Information and Event Management (SIEM)? Both use log files, both use the same log files. SIEM gives a security focus to viewing log files. As mentioned elsewhere, information security is information availability, information confidentiality and information integrity (availability + confidentiality + integrity = security). Your log file reviews should look for implications which affect information availability, confidentiality and integrity. Other (not directly security) characteristics you may be looking for: response time and usage patterns. Anomalous situations may indicate security concerns; trends may indicate operational and performance concerns.

Does your SIEM product capture information from all devices which may be attacked? Can all of your devices which may be attacked produce log files to feed your SIEM product?

See whitepapers at NitroSecurity (a Security Information and Event Management (SIEM) vendor).

Archive log files to assist with incident response. The incident response team may need to go back 90 days.