WHAT TO DO IF HACKED

"I've Been Hacked! What Should I Do?" Part 1

Where forensics is the application of scientific knowledge to legal problems, computer forensics applies to the collection and analysis of digital evidence. Computer forensics play a weighty role in answering the question, "What should I do if I suspect my network has been hacked?"

You might perceive computer forensics as a task as distant from your duties as traveling to Mars. But if it isn't in your job definition now, it soon might be. For example, the state of California has passed SB 1386, a law effective July 1, 2003, requiring companies that do business in California to report all computer security breaches that might reveal customer data--even if the system breached is not in California. While your organization might have a policy that encourages keeping quiet about security incidents, the new California law could not only force your organization to change that policy, it also might thrust any future network security incident into the view of law enforcement.

For the long-term health of your company (not to mention your career), you should know at least the basics of forensics. In this four-part series, I will explain some of what you need to know about computer forensics. Part 1 focuses on the basics. In Part 2, I'll discuss what you should do before shutting down a system that may contain evidence of an incident. Parts 3 and 4 discuss tools that you can use to examine binary files and disk partitions for evidence.

Ready? Let's dig in.

The Basics
When performing any investigation, you need to record all the information you collect in a notebook. Paper works best, even in this electronic age. You want to take good notes, because turning evidence over to law enforcement might occur weeks or months after the incident. And if the case goes to trial, you will want really complete notes to help with your memories about what you did as much as a year or two before. So include in your notes everything you do, such as:

time and date when you performed each step in your investigation

who you were talking to

phone numbers used

what was said

what commands you entered on what system

summary of results.

The actual output of commands can be captured to a file on another system--but I will discuss how to do that next time.

You probably already know something about the proper way to handle a crime scene from watching movies or television. And "handle" is exactly the wrong word. When a crime occurs, it is important not to disturb the crime scene in any way, even when the victim is a computer. Changing or cleaning up after a crime may disturb or destroy evidence, and this is especially true of digital evidence.

Now you're faced with a real quandary. How can you investigate a victim system without touching it? The answer is, "As carefully as possible." In the unlikely event that fingerprints on the system might be important, wear cotton gloves. But you also must be just as careful not to damage the digital evidence stored on the victim.

Colonel Mustard, in the Kitchen, with a Keyboard
Collecting evidence is not that dissimilar to the old board game, "Clue." You want to learn what was done, when it was done, who did it, and how. Most of your investigation should be done with the system shut down, and working from copies of the hard drive. But before you can even begin, you must be certain that there really is an incident to investigate, and that the system you are looking at is the only one (locally) involved. And this means sitting down at the console and taking a look around.

First, consider the physical environment, that is, where the victim system is located. You need to record who (if anyone) was using the system when the incident that sparked this investigation was reported, as well as who might have physical access to this system. In many cases, the system might be located in an office cubical, so anyone in the office has access to it. Or the system could be in a server room that only a limited number of people have access to -- a more ideal situation for an investigator.

Everything you do on a computer changes the digital state of that computer. Accurate notes are essential, so that later you can distinguish the changes you caused from changes that might have been made during the incident. In Part 2, I suggest techniques for information gathering that produce a minimal footprint. For now, since you are just looking around, limit your activities to viewing files, network and process status, and registry keys. Viewing these resources will change the access (read) time associated with files, directories and registry keys, so keep an accurate list of what you look at.

You are looking for evidence that supports the initial incident report. Suppose you became suspicious that a system had been broken into because you noticed DENY messages in your firewall logs indicating unusual outgoing network connection attempts from the suspect system. Or, an IDS noticed an attack directed at a particular IP address. Note that in each case, you might need to look at your DHCP server logs to determine which system was using that IP address at that time. You might use the netstat -an command to display network connections, or check to see if the system fingered by the IDS system even has the required vulnerability (for example, the attack you think might've happened could exploit a service that turns out not to be installed on this machine; or perhaps the service is present, but patched recently enough that it's not vulnerable).

Common changes intruders make after a successful attack include creating new user accounts, changing privilege associated with user accounts, executing new and unusual processes, adding new files to -- and changing -- startup files or registry keys used to start programs. You can examine all of these things without modifying the hard drive, except that their "time last read" will update.

The Difference between Policy and Procedures
Part of being prepared for an attack involves getting your management involved in the creation of policies and procedures. Think of policies as the set of guidelines for the proper use of computers and networks within your organization, and procedures as the step-by-step instructions for performing certain security-related tasks. For a short course on forensics that discusses policy, read this paper or this Ernst and Young paper.

Policy and procedure helps you, the investigator, ensure you don't some day wind up with the equivalent of legal egg all over your face. With policies and procedures in place, everyone in your organization should know what his or her role is, and what to do (or, what not to do) so no evidence gets destroyed.

 April 4, 2003

"I've Been Hacked! What Should I Do?"
Part 2: How to shut down

In Part 1 of this series, you learned how to establish whether a system actually had or had not been a victim of an incident. Assuming your investigation concludes that an intrusion has occurred, in Part 2 we consider what you should do before you turn off the victim system.

You should regard a computer that has been attacked as a crime scene, and that means you must handle that system and all its resources with care. As soon as you turn that system off, all of its currently running processes, as well as any open files and network connections, will be lost. It is as if the victim system were a house of cards, and turning off the power will flatten it.

You might ask, "Why not leave the system turned on, but disconnect it from the network?" There are two big problems with this idea. First, most systems have a purpose other than maintaining the state of that system when attacked. The victim was somebody's desktop, or perhaps an important server. The odds are, someone needs that machine back simply to do his or her normal job; you might not have the luxury of leaving it disconnected indefinitely. The second problem is that the attacker might have booby trapped the system. Disconnecting the network can be detected by a script or program that responds by deleting all files on the system, then shutting it down. (This really has happened.)

The middle road involves collecting as much of the transient information as possible before shutting off the system. By transient information, I mean everything about the system that will be lost by shutting it down: local and network logons, processes, open files, working directories, and network connections.

You want to know about opened files because Trojans and other malware often store configuration information in files, and finding these files provides you with leads to the perpetrator or to finding other interesting things hidden in your system. A backdoored version of the Secure Shell, for example, would include an open file used to capture passwords entered when using the Secure Shell server.

Network connections and listening services get represented as network sockets. You can use the netstat -a command to view these connections and services, but there are better commands that will link processes and the connections or services associated with them. I explain them later in this article.

Collecting evidence that courts should accept
We want to have minimum impact on the victim system, while finding out (and making a record of) what is going on. To do so, we will:

Download (and possibly build) a tool to copy data across the network
Download tools for collecting data

Burn these tools onto a CD

All of the tools that I am recommending produce text output. We don't want full screen, GUI displays, because text is much easier to work with as evidence. You really can't search a screen capture for a particular filename, can you? But searching text is a snap. And for storing the text output, you can store it remotely using a tool called cryptcat.

Cryptcat is based on netcat (nc), a versatile network tool available in both UNIX and Windows versions. Netcat can act as a network client or a server, and can be used to transfer data between two systems. Cryptcat adds encryption, so not only can you transfer data, but you can also use encryption to protect that data. If the keys used with cryptcat don't match, cryptcat exits without copying anything. By using cryptcat, you could assert in court that you are sure your network copy worked, and that no one else could have modified the data while it was copied. Note that if you are familiar with SSH, you might elect to use it instead.

Obtain cryptcat from Farm9, a security consulting company in Oakland, California. For Windows systems, you just download and unzip the Windows version, and you are ready to go. UNIX and Linux users have to download, uncompress, untar, and build the distribution, but the process is not difficult. The only even slightly tricky thing you must do is select your OS type and supply it on the "make" command line used to build cryptcat.

Once you have cryptcat ready, you can familiarize yourself with its networking capabilities by trying the following examples. First, you can copy the output of a command from a Windows 2000 system to a Linux system by doing the following.

On a Linux system named linux, first start a server:

cryptcat -k your_secret_key_here -l -p 33333 > pstree.out

On Windows:

pstree | cryptcat -k your_secret_key_here linux 33333

Cryptcat uses the same option letters as does netcat, with one exception. By default, cryptcat uses the letters "metallica" as its encryption key, and you want to change that by supplying your own key with the -k flag. Your key can be up to 32 characters (more are ignored), and if you include spaces, surround the entire key with "quotes." The server side, linux in this case, tells cryptcat that it will 'listen' (-l) on port 33333 (-p 33333). The port number is not important, but you should pick a high numbered port that is unlikely to be in use, and use the same port (and key) at the client end. The client end specifies the server name (linux in this example) or IP address and port to use.

Cryptcat and netcat don't notice the end-of-file from the pipe, so must be killed when you finish with them. When your cryptcat client finishes, enter Control + C to tell it to exit. If you aren't sure the client has finished, just give it ten seconds.

Who is doing what on this system?
For UNIX systems, the only tool you need besides cryptcat is lsof. You may need to build lsof, or you can search for pre-built RPMs for Linux systems or binaries for other versions of UNIX.

Lsof stands for LiSt Open Files. Remember that I mentioned that "running processes" include lists of open files and network connections and listening servers? Lsof executed with no flags lists all processes, along with their Current Working Directory (CWD), open files, and network info. Once you have used lsof, you will want to run it anytime you find an unusual process running on a system, because it will show you which files and network connections the process is using.

Windows systems require several tools to duplicate the abilities of lsof. I recommend fetching a toolkit called PsTools from Sysinternals. The PsTools zip file contains PsList and PsLoglist, which you will need. Also fetch handle.exe from Sysinternals, for listing open files associated with processes.

To list network connections and listening services, grab a tool from Foundstone called fport.

Next, you build a CD that contains all the tools you need. You can actually create a single, ISO 9660 format CD, and have different directories for each of the operating systems you want to support. Only Macs need a specially formatted CD (and if you use a Mac to create an ISO 9660 CD, the CD can work with all systems). For example, if you have downloaded or built tools for Windows and Linux, create a Windows and a Linux directory on your CD, and copy the appropriate tools into each directory.

When you need to capture data from a live system, insert your custom-built CD, mount it (if you are using a UNIX system that does not do this automatically), and change to the appropriate directory for the system you are investigating. Now you can execute commands, pipe the output to cryptcat, and save the results on another system.

Capture the perpetrator in chains
You can chain multiple commands together by enclosing the commands in parentheses. For example, when working with Windows systems, this string will list running processes and CPU information, and send the output to another machine, encrypted:

(pslist && psloglist security && handle && fport) | cryptcat ...

On UNIX/Linux, a roughly equivalent command string tells you who is logged in, what processes are running, and what drives are mounted, then sends the output to another machine, encrypted:

(who; lastlog; ps -auxwww; mount; lsof) | cryptcat ...

In Windows, if you want to execute commands interactively and save the output, you can use the -e (execute) flag in cryptcat on the victim:

cryptcat -k big_secret_key -l -p 33333 -e cmd

and use the script command on a UNIX system to capture the output to a file. The example below names the resultant file "live.output.win2k":

script live.output.win2k
cryptcat -k big_secret_key win2k 33333

If the victim is a UNIX system, replace cmd with the sh.

The Grand Finale: Pull the plug
Once you believe you have gathered all the evidence you might need from the live system, pull the plug--literally. You are taking a risk in doing this, as some file system corruption may occur, but this is better than having a booby trap delete all files when you do a proper shutdown.

On UNIX/Linux systems, enter:

sync; sync

and wait until the next command prompt appears before turning off the system, as using sync twice will flush disk buffers and not return the command prompt until it has finished.

Now you have gathered all the transient information you can, recorded it using a method that assures integrity, and you've safely shut down the victim machine. Next week, we'll start examining the hard drive.

Web Page Designed By  ADAM
Copyright © 1981 - 2008 MINDPRIDE CONSULTING All rights reserved.
Revised: November 21, 2007