BigNoseBird.Com- home Small Logo
The 508 compliant Guide to 
       Big Nose Bird
RETURN TO THE NOTEBOOK INDEX

 
Blackhole Spider Stopper

Ever check your logs, only to discover that some runaway spider or a spider pretending to be a browser is eating your server for lunch? Thousands of nonsense requests clogging up the works and chewing on bandwidth.

Here is a script that you can run from cron that will check your error_log to see if a single IP address has made more than a certain amount of bogus requests.

How the script works is simple. It reads the webserver's error_log every so often to see if a host has created more errors than your preset threshold. If a host has done this, the script uses the route command to route the traffic to an unused IP address on your local network, and the script e-mails you to tell you about the problem.

Important: The unused IP or blackhole address must be not respond to ping or you will create serious problems for yourself!!! Be sure to set the permissions on the script to 700 (read/write/execute for owner only).

In order to install this script, you must have root permission on your server. The script must run as root in order to reroute the spider's traffic to a dead IP address.

In this example, the script is named trap and is installed in the /var/log/httpd directory.

Below is a sample crontab entry to run the script.

0,5,10,15,20,25,30,35,40,45,50,55 * * * * root /var/log/httpd/trap >/dev/null 2>/dev/null

(all on one line)

If you cut and paste the script that follows, do not do it from this page's source code since the < & > characters are represented with special characters.

    Drawbacks
  • Requires you to manually remove the route table entries.
  • Since it just reads from the start of the log, you must also rotate or clear your error log to prevent the IP from forever being blocked.
  • Does not adapt to other log formats automatically. This script uses the standard Apache Error_log format.
  • The script only can deal with one log file.
To manually remove a blackholed IP after clearing your error_log, issue the command (under linux anyway):
  /sbin/route -n del -host 123.123.123.123 gw 192.192.192.192
where the 123 address is the banished IP and the 192 address is your blackhole IP.
#------------SCRIPT-STARTS-BELOW-THIS-LINE-----------#
#!/usr/bin/perl

#copyright 2000 bignosebird.com

#Path to sendmail on your server. the -t option must be used.
 $SENDMAIL="/usr/sbin/sendmail -t";
#Path to name of your error_log
 $ERROR_LOG="/var/log/httpd/error_log";
#Path to name of the list of banished IP's
 $BLACKHOLE_LIST="/var/log/httpd/blackhole";
#The Dead IP address on your LAN.
 $BLACKHOLE_IP="192.192.192.192";
#The maximum number of errors you wish to tolerate
 $MAX_ERRORS=500;
#The e-mail address to send the notice to
 $NOTIFY="me\@myaddress.com";
#The return address for the notice
 $MAILFROM="root\@myaddress.com";
#The path to your route command
 $ROUTE="/sbin/route";
  
  @blackholed=();
  &read_list;
  &read_errorlog;
  &process_file;
  &rewrite_list;

sub read_errorlog {
  open (IX,"<$ERROR_LOG");
  while (my $line=<IX>) {
    if ($line=~/Spin Client/i || $line=~/trace/) {
     next;
    } 
    chop $line;
    $line=~s/\[//g; 
    $line=~s/\]//g; 
    my @parts=split(/\s+/,$line);
    my $ip=$parts[7];
    if ($ip !~/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]/){
      next;
    }
    if ($master{$ip} eq "") {
      push(@iplist,$ip);
      $master{$ip} = 1;
    }
     else { 
      $master{$ip} = $master{$ip} + 1;
     }
  }
  close (IX);
}
  
sub process_file {
  foreach my $ip (@iplist) {
     if ($master{$ip} > $MAX_ERRORS && &holed($ip) == 0) {
      # this means we have a problemo 
       push(@blackholed,$ip);
       my $timestamp = time;
       my $stamp="$ip\|$timestamp";
       push(@outlist,$stamp);
       &notify_webmaster($ip,$master{$ip});
       &kill_ip($ip);
     }
  }
}

sub read_list {
  open(BLKHOLE,"<$BLACKHOLE_LIST");
  while (my $bh=<BLKHOLE>) {
    chop $bh;
    my @subs=split(/\|/,$bh);
    my $badip=$subs[0];
    push(@blackholed,$badip);
    push(@outlist,$bh);
  }
  close(BLKHOLE);
}

sub rewrite_list {
  open(BLKHOLE,">$BLACKHOLE_LIST");
  foreach my $towrite (@outlist)
   {
    print BLKHOLE "$towrite\n";
   }
  close(BLKHOLE); 
}

sub notify_webmaster {
  my ($ipaddy,$count)=@_;
  my $MSG=<<__END_OF_MAIL__;
To: $NOTIFY
From: $MAILFROM
Subject: $ipaddy blackholed

The IP address $ipaddy was blackholed after $count errors.

__END_OF_MAIL__
  open (SM,"|$SENDMAIL");
  print SM "$MSG\n";
  close(SM);
}

sub kill_ip {
  my ($ip)=@_;
  system("$ROUTE -n add -host $ip gw $BLACKHOLE_IP");
}

sub holed
{
  my ($item)=@_;
  foreach my $tocheck (@blackholed) {
    if ($item eq $tocheck) {
     return 1;
    }
  }
  return 0;
}


#------------SCRIPT-ENDS-HERE------------------------#



Find or Give Help on the BBS
 
Home Top E-Mail
If it looks great, it's by Christine
Some Fine Print
© 1997-2003 BigNoseBird.Com®, Inc. All rights reserved. All other trademarks are the sole property of their respective owners. The products that we recommend are only ones that we use. We have no relationship with any of the authors or their companies. We cannot assume responsibility for their ultimate performance or lack of same. We also cannot assume responsibility for either any programs provided here, or for any advice that is given since we have no control over what happens after our code or words leave this site. Always use prudent judgment in implementing any program- and always make a backup first! For further information, please read our Privacy Statement. We can be contacted at webmaster@bignosebird.com.


<reallybig.com>
Web Builder Network Portal
Advertise
on the
Reallybig.com
Network
BigNoseBird Newsletter
Subscribe
Un-Subscribe


Sign up today to receive our low volume newsletter. Tips, tricks, news, and whatever else crosses our minds.
Back Issues
Privacy Statement