Network Forensics Puzzle #2 Solution

November 2, 2009

From Puzzle #2 – Ann Skips Bail

Answer written October 27, 2009. Not to be published before contest ends November 22, 2009.

Tools used: Wireshark, a Base 64 decoder, Xplico or base64.exe, fsum.exe and Word 2007.

Open the packet capture file (evidence02.pcap) in Wireshark.

Find the SMTP packet with the Info “334 VXNlcm5hbWU6“. This is the prompt for an email address. The response (c25lYWt5ZzMza0Bhb2wuY29t) requires a Base 64 decoder (

Find the SMTP packet with the Info “334 UGFzc3dvcmQ6“; this is the prompt for a password. The response (NTU4cjAwbHo=) requires a Base 64 decoder (558r00lz).

Selecting part of a multi-part SMTP message within Wireshark causes Wireshark to reassemble the data. This produces the email message and its header, but this will not decode MIME.

Selected the SMTP packet at 140, selected its data in the data window, double-clicked “reassembled DATA in frame: 557” and was able to view the text of the message. (That is, found that it was addressed to, “Hi sweetheart” and so forth, learned name of attached file.)

Found MIME data in the data frame; double-clicked to select it. Used File-> Export-> Selected Packet Bytes to an arbitrary file name: wireshark.raw. Used base64.exe to recreate secretrendezvous.docx.

base64 -d wireshark.raw secretrendezvous.docx

An alternate approach to carving out the email messages and their attachments would be to use Xplico Xplico (“the Internet Traffic Decoder”) can display the Internet traffic found in a pcap file. The Carlos Gacimartín image of Debian 5.0 with Xplico 0.5.2 installed and running worked fine.

Computed md5sum using fsum

fsum -md5 secretrendezvous.docx

Opened secretrendezvous.docx in Word 2007 and saved a copy as html. This produced image001.png; and I computed the MD5sum of this file.

1. What is Ann’s email address?
2. What is Ann’s email password?
3. What is Ann’s secret lover’s email address?
4. What two items did Ann tell her secret lover to bring?
fake passport and bathing suit
5. What is the NAME of the attachment Ann sent to her secret lover?
6. What is the MD5sum of the attachment Ann sent to her secret lover?
7. In what CITY and COUNTRY is their rendez-vous point?
Playa del Carmen, Mexico
8. What is the MD5sum of the image embedded in the document?

Note: Chaosreader quickly parsed the evidence02.pcap file into a set of sessions, but the results were inaccurate. Chaosreader would be a way to get a quick overview of the sessions.

C:\perl\bin\perl.exe chaosreader -v ..\evidence02.pcap


May 26, 2009

Start with Creating a Computer Security Incident Response Team: A Process for Getting Started. Review DataLossDB, the research project aimed at documenting known and reported data loss incidents world-wide. You want an Incident Response procedure and an Incident Response team created and tested before you need it. Members of the team should have copies of the procedure, and Lenny Zeltser’s cheat sheets. Test your plan regularly. Learn the gaps and revise. Search for “Incident Response Plan” and you will find many examples of published Incident Response Plans. See ENISA’s Support For CERTs/CIRTs for examples and practice exercises.

However, which came first: the incident or the declaration of the incident? The incident. If an incident occurs in a forest, and there’s no one there to notice it, does it still occur? Yes.

From the U.S. Army Information Assurance Security Officer (IASO) Training

Signs of an incident fall into one of two categories: indications and precursors. A precursor is a sign that an incident may occur in the future. An indication is a sign that an incident may have occurred or may be occurring now. Too many types of indications exist to exhaustively list them, but some examples are listed below:

  • The network intrusion detection sensor alerts when a buffer overflow attempt occurs against an FTP server.
  • The antivirus software alerts when it detects that a host is infected with a worm.
  • The Web server crashes.
  • Users complain of slow access to hosts on the Internet.
  • The system administrator sees a filename with unusual characters.
  • The user calls the help desk to report a threatening e-mail message.
  • The host records an auditing configuration change in its log.
  • The application logs multiple failed login attempts from an unfamiliar remote system.
  • The e-mail administrator sees a large number of bounced e-mails with suspicious content.
  • The network administrator notices an unusual deviation from typical network traffic flows.

One should not think of incident detection as being strictly reactive. In some cases, the organization can detect activities that are likely to precede an incident. For example, a network IDS sensor may record unusual port scan activity targeted at a group of hosts, which occurs shortly before a DoS attack is launched against one of the same hosts. The intrusion detection alerts regarding the scanning activity serve as a precursor of the subsequent DoS incident. Other examples of precursors are as follows:

  • Web server log entries that show the usage of a Web vulnerability scanner
  • An announcement of a new exploit that targets a vulnerability of the organization’s mail server
  • Information stating that the Unit will receive a cyber attack

Not every attack can be detected through precursors. Some attacks have no precursors, whereas other attacks generate precursors that the organization fails to detect. If precursors are detected, the organization may have an opportunity to prevent the incident by altering its security posture through automated or manual means to save a target from attack. In the most serious cases, the organization may decide to act as if an incident is already occurring, so that the risk is mitigated quickly. At a minimum, the organization can monitor certain activity more closely—perhaps connection attempts to a particular host or a certain type of network traffic.

A Mandiant presentation at Black Hat in 2006 reported that organizations detected attacks:

  • Sometimes through anti-virus alerts, although most alerts are uninvestigated. (Note that this is my point as well; investigation of anti-virus alerts yields results.)
  • Sometimes through an Intrusion Detection System (IDS), although attacks through SSH, HTTPS and VPN escape IDS detection. (Note that with a largely mobile or remote workforce, relying upon VPN and public network availability, IDS does not involved.)
  • More often through client reports, outside the company.
  • Sometimes through end user problem reports.

End user problem report symptoms that may be indicators of an incident:

  • Continual termination of anti-virus software.
  • Installing new applications does not work.
  • Commonly used applications will not run.
  • You cannot “Save As”.
  • Task Manager closes immediately after you open it.

You do not know if an incident should be declared until you are well into an investigation (the Identification stage of Incident Response). An unexpected restart in an event log might be an incident. A port scan might be an incident. You won’t know if the incident response procedure should be invoked until you have learned more about the specific restart or more about the specific port scan. (In the military? Unexpected restarts and port scans are ALWAYS incidents. Report them.)

Many sources will help make the determination that an incident has occurred. Get LennyZeltser’s Security Incident Survey Cheat Sheet for Server Administrators now, and be comfortable with it. Read John D. Howard & Thomas A. Longstaff’s A Common Language for Computer Security Incidents.

Once you have determined that an incident has occurred, your (previously established and tested) incident response procedure must be invoked.

Grab a bound notebook (not spiral bound, you don’t want to be accused of tearing out pages), preferably with numbered pages and write down the events which have occurred so far. Use details. Imagine that you will be reading this seven years from now and need to reconstruct the events. That is, mentally rewinding and replaying the steps prior to declaring an incident needs to be the first task of your incident response procedure. This is particularly problematic if a criminal investigation may be involved. When that is the case, the forensic process, including establishing a chain of custody, must be followed. There will be steps you wish you had done differently prior to the declaration of an incident. That happens; that’s prudent. Once you recognize an incident, however, your actions must adapt.

The steps of incident response. Most lists include only the PICERL steps. The full team events of notification and post-incident review should also be included.

  1. Preparation
  2. Identification or Assessment
  3. Notification
  4. Containment
  5. Eradication or Remediation
  6. Recovery
  7. Post-Incident Review
  8. Lessons Learned (or Follow Up)

Determining if an incident should be declared is part of the Identification step in incident response.

Note that this list assumes that preparation and declaration of an incident are not within the scope of the incident response. Once a specific incident has been declared, the specific incident life cycle begins. General preparation is outside the scope of specific response.

Regarding Notification: there’s internal notification and external notification. Internal notification is to the Incident Response team. A member of the Incident Response team should be responsible for external notification. There are incidents which require external notification, which is why a lawyer is on the Incident Response team. They may not be the person who makes the notification, but they would review the incident for notification requirements.

If you find yourself in the extremely unusual situation of knowing that a notification is required but is not being done, you may believe that your choice is between making the notification (and losing your job) or not making the notification. Get legal advice. Consider the long term possibilities. Possible outcome if you make the notification: lose job. Possible outcome if you choose to not make the notification: lose job, face prosecution, lose savings, serve time, and become unemployable.

That notebook … is it for all incidents, or only security incidents? A burst pipe in the building or a natural disaster are also incidents. Without the notebook, Lessons Learned becomes difficult.

Test: City of Norfolk hit with code that takes out nearly 800 PCs. What lessons do you take away?

Cluff’s team noticed that computers were taking longer than normal to shut down around 4:30 p.m. on Feb. 9. Those machines could not then be restarted. After investigating, his team discovered that a virtual print server was pushing out malicious code. The team pulled the virtual server offline, scrubbed it and reverted it to a previous instance of the print server software, he said.

Regrets: Did not preserve the virtual print server for further investigation. How did this happen? What could have prevented this? Evidence is lost.

Online Trust Alliance Data Breach and Incident Readiness Planning Guide

Privacy Rights  Clearinghouse including actual reported breaches

Mozilla InvestiGator (MIG) is a platform to perform investigative surgery on remote endpoints. It enables investigators to obtain information from large numbers of systems in parallel, thus accelerating investigation of incidents and day-to-day operations security. Masche adds memory forensics capabilities to MIG.


Hidden Files

April 8, 2009

You might suspect that malicious code exists on a machine because …

  • It accessed a web site or IP address known to have hosted malicious code.
  • Anti-virus software detected malicious code. That’s a good reason to suspect additional malicious code escaped the anti-virus software. Anti-virus software could be making a false report, of course. Before you remediate (which usually means “reimage”), you should learn if there is any additional malicious code that has gone undetected.

What steps you can take to confirm (or partially deny) your suspicions? You look at running processes, but after you’ve looked at running processes and you still think there’s something malicious, look for the files that someone has made an effort to hide. “Hidden” is not synonymous with “invisible;” to go unnoticed will do.

Where could I find hidden files?

    • The “hide in plan sight” strategy is at least as old as Poe’s “The Purloined Letter.” Its longevity reflects its effectiveness. Finding files that don’t belong amongst the hundreds of files that do is a challenge. Having a table of known good files and their hash codes can help eliminate suspects. Using a utility to find unsigned executables and confirming that the signatures that are found are authentic will produce a long list that includes many benign conditions. See, for example, sigcheck from Sysinternals.

sigcheck -s -v c:\ >result.csv

    • Suspect recently created files in C:\Winnt\System32 (or C:\Windows\System32). The date stamp is often unmodified. Similarly, suspect recently created files in C:\Winnt (or C:\Windows) and in the user’s temporary files (C:\Documents and Settings\Local Settings\Temp). These files often have names like svchost.exe, spoolsv.eve, symwsc.exe, swupdtmr.exe or winservices.exe, names which resemble Windows components but are not.
    • Suspect files that do not sort “correctly”. In an attempt to “hide in plain sight,” malware may use extended ASCII characters to create innocent looking filenames. Use the Character Map application to review the available ASCII characters, such as U+0441. While it is a Cyrillic small letter Es, it strongly resembles a Latin small letter c. Using the Cyrillic characters, a file named “C:\Program Files\Common Files\Οracle\wіnlogon.exe” can be created; but the “i” in winlogon.exe is not a Latin “i” and the “O” in Oracle is not a Latin “O”. The Cyrillic characters will not sort as their Latin counterparts do.
    • The non-breaking space character (decimal 160, hexadecimal A0) can be used as a file name or folder name character. (Alt+160 on the numeric keypad types a non-breaking space character.) A file and a folder has an icon as well, but the Properties Customize tab, Change Icon button reveals many clear icons. The result is an easily missed file or folder name.
    • Hide in a system folder, such as “C:\Windows\Downloaded Program Files” (or “C:\Winnt\Downloaded Program Files”) or “C:\$RECYCLE.BIN”. There is a real “Downloaded Program Files” folder, but you won’t see its contents using Windows Explorer. Use a command window instead. Expect hidden, system files and search subdirectories.

dir "C:\Windows\Downloaded Program Files" /ah /s
dir "C:\Windows\Downloaded Program Files" /s

    • Hide using the Directory and System attributes. Foundstone’s hfind utility hunts for files with the hidden attribute, directories with the hidden attribute and directories with the system attribute. There are a lot of hidden files and folders, a lot of benevolent conditions to ignore.

hfind C: >> local.txt
hfind \\remote\c$ >> remote.txt

streams -s *.*

    • Rootkit technologies employ techniques to hide files from standard file system utilities. There are many utilities to find these technoolgies and what they are hiding. Sysinternals’ Rootkit Revealer reports some benevolent conditions. When used in conjunction with psexec (also a Sysinternals utility), it can scan remote systems.

psexec \\remote -c rootkitrevealer.exe -a c:\windows\system32\rootkit.log

See Invoke-PsExec when invoking psexec on multiple targets.

Note that the above are aggressive but not exhaustive measures. To search rigorously for rootkits, for example, boot from an alternate drive and to search the suspect drive. The measures given show how you can search for undetected malware at little cost.

Hidden data is a larger subject. There are many more places to hide data, within files, within slack space and within space no file is using (including sectors that have been marked as “bad”). If malware was hidden using these approaches, then non-standard file system drivers would be required to execute it. Find those files, the driver files, in order to find the additional hidden malware.

Read the rest of this entry »

Root Cause Inspection

April 8, 2009

Take, for example, the following virus detection alert:

From: servername [mailto:servername]
Sent: (Date and time)
To: AV Alerts – HQ
Subject: EXPL_EXECOD.A on machine(user)

Virus alert.
EXPL_EXECOD.A is detected on machine(user).
Infected file: C:\Documents and Settings\user\Local Settings\Temporary Internet Files\Content.IE5\NXXZUEMP\exp4[1].htm
Detection date: (date and time)
Action: Virus successfully detected, cannot perform the Clean action (Cannot perform the Delete action)

Antivirus software has detected a virus in exp4.htm and prevented it from running.

Case closed?

A look at the index to the browser cache shows that exp4.htm is among the many web pages from (EXPL_EXECOD.A) (EXPL_EXECOD.A) (EXPL_EXECOD.A)

There was a different exploit at each exp?.htm page. We were detecting the use of one of the five exploits, but only one of the five.

A web site in the Russian Federation.

Domains registered to this address:

Samples of the files were submitted to for verification and to the specific anti-virus vendor for analysis.

The  URL was submitted to a public malware block list ( and to the specific anti-virus vendor for inclusion in their web filtering product.

The IP address was blacklisted in the client firewall.

The vulnerabilities that these malware samples attempted to exploit had already been remediated (software updates and patches installed).

I have provided links to utilities that make reviewing web browser history easier. When malware arrives through a web browser, you want to learn where it came from.

  1. What else came from the same source? Is it also malicious, but undetected?
  2. Do you want to blacklist that source? If detected malware was delivered once, are you betting that the next malware will also be detected? Why take that chance?
  3. Submit suspicious links to a central reporting site, such as Malware Block List.
  4. Keep a log (spreadsheet, table, database) of what you have detected. The URL (http://), its IP address, the reason it caught your attention, what you did with the information, and the date seen are basic fields. Tip: convert the dotted decimal IP address to a decimal IP address. = 112+(128*256)+(53*256*256)+(58*256*256*256) = 976584816. You will find a large IP address number easier to sort than a set of octets.

hpHosts may be helpful. You can use it to learn if this IP address or URL has been reported as malicious, or if other malicious sites are at that IP address. For example, to see if there are malicious hosts whose IP address starts with 63.246.20, use

hpHosts uses the following abbreviations to categorize their reasons for including IP addresses in the malicious list:
ATS: ad/tracking server
GRM: grass roots marketing (astroturfing)
EMD: malware distributor (adware, spyware, viruses etc). (Classification: )
HJK: hijacking
EXP: exploits and social engineering
FSA: fraudulent security (and non-security) applications
WRZ: Warez and keygens
PSH: Phishing
HFS: spammed the hpHosts forums

Targeted Forensics: Mapping a Process to a Malicious Command and Control describes how to determine which process is connecting to a malicious command and control center, using Volatility and a memory dump.

The Targeted Forensics Series: Confirming Remote Desktop Connections (Part 1 of 2) (Part 2 of 2) describes finding evidence of a remote desktop connection to or from a Windows device, using the registry and log parser.