Category: Security exploits

In this series of articles I am trying to help server admins and owners of VPS or Dedicated servers to find viruses or malware on their servers. Part of the diagnosis of your system is to see what emails are being sent out and from which accounts. Since spammers like to use compromised servers, I believe that it makes sense to check regularly that the emails being sent out roughly match what you would expect to see.

I have servers that I host client websites on. If a client who usually sends out 20 emails a month suddenly sends out 500 then this is cause for concern and I would immediately investigate the server for malware.

On linux systems, Exim (the mail transfer agent) already logs the working directory of messages sent to the queue by a script. Here’s an example of what you would expect to see in an exim_mainlog file:

2015-08-10 13:52:28 cwd=/home/fredb/public_html 3 args: /usr/sbin/sendmail -t -i
2015-08-10 13:52:28 1ZOmZ2-0004XN-GK <= [email protected] U=fredbloggs P=local S=133267 [email protected] T="Site Database Backup Monday, August 10th, 2015 at 1:52 pm" for [email protected]
2015-08-10 13:52:28 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1ZEmZ2-0004XE-EK
2015-08-10 13:52:29 1ZEmZ2-0004XE-EK SMTP connection outbound 123459211149 1ZEmZ2-0004XE-GE fredbloggs.co.uk [email protected]

Note: I like to use Notepad++ to analyze these large text files within Windows as other editors aren’t quite up to the task.

So it looks like there’s some function of the ‘fredbloggs’ website that auto-backs up the database, then sends a related email notice to whatever email address the webmaster provides, in this case, [email protected]. The working directory for the generation of that message was “/home/fredbloggs/public_html”. Nothing suspicious here as we have an auto-backup program installed on this WordPress-powered website. Nothing to see here, move along please…

Here’s another example:

2015-08-03 12:00:15 cwd=/home/janedoe/public_html/wp-admin 3 args: /usr/sbin/sendmail -t -i
2015-08-03 12:00:15 1ZQDTb-0005QW-5k <= [email protected] U=janedoe P=local S=989 [email protected] T="[https://janedoe.co.uk] WordPress Login Address Changed" for [email protected]

Again, possibly normal but I’d raise the question whether Jane changed her email address on WordPress. If not, this is cause for concern.  It’s a kind of detective work where you need to step back and look at all of the evidence to compile a big picture.

So, let’s run this beauty of a command against the exim_mainlog to give us an idea from which working directories our server gets messages sent to the mail queue:

zgrep "cwd=" /var/log/exim_mainlog*|awk '{print $3}'|sort|uniq -c|sort -n|sed 's/cwd=//g'

The exim_mainlog records the arrival and delivery of all emails. It explains where the mail came from, to which address it was delivered, the hostname of the server and more. Additional details can be added to this log file by using extended logging in exim. Your output would be something like this on most systems:

8 /home/janedoe/public_html/wp-content/plugins/cforms

So within the last 30 days, the /cforms directory has sent 8 messages to the queue. Cforms is a defunct WordPress plugin and now, as such, unsupported by the developer against exploits. Would you expect that Jane’s website should do that? A result like this isn’t necessarily suspicious as this is normal contact form use. Something like this, however, would be VERY suspicious:

815 /home/janedoe/public_html/images

I can’t think of a valid reason why an ‘images’ directory should be sending mail, so alarm bells would trigger and that’s definitely something I would look into further.

So, presuming we saw strange usage numbers or a bizarre path, let’s dig even deeper and look at what the Subject of Jane’s emails actually were, as this gives us an indication of spam activity. Change directory into /var/log

cd /var/log

Now run this:

zgrep -A 1 "/home/jane" exim_mainlog* |grep T= |awk -F T= '{print $2}' |sort | uniq -c |sort -n |awk -F " for " '{print $1}'

Nice, it returns a list like this which tells us all we want to know:

1 "Akismet: Spam - Jane Doe Books Contact Form: Pay only when you get results"
1 "Jane Doe Books Contact Form: Help with my book club "
1 "Site Database Backup Friday, July 17th, 2015 at 10:02 am"
1 "Site Database Backup Friday, July 27th, 2015 at 1:02 pm"
1 "Site Database Backup Friday, July 31st, 2015 at 10:36 pm"
1 "[Jane Doe Books] Your site has updated to WordPress 4.2.3"
1 "[Jane Doe Books] Your site has updated to WordPress 4.2.4"

Again, no cause for concern and the only spammy one there would be the first one, already marked as such by Akismet.

If you have lots of adverts for cheap meds or blue pills in there then you need to find the offending code that’s pushing spam through your email system. Start with a virus scan on your Linux server

Hope this helps and feel free to drop me a comment below.