Here is a script that you can run from cron that will check your error_log to see if a single IP address has made more than a certain amount of bogus requests.

How the script works is simple. It reads the webserver's error_log every so often to see if a host has created more errors than your preset threshold. If a host has done this, the script uses the route command to route the traffic to an unused IP address on your local network, and the script e-mails you to tell you about the problem.

Important: The unused IP or blackhole address must be not respond to ping or you will create serious problems for yourself!!! Be sure to set the permissions on the script to 700 (read/write/execute for owner only).

In order to install this script, you must have root permission on your server. The script must run as root in order to reroute the spider's traffic to a dead IP address.

In this example, the script is named trap and is installed in the /var/log/httpd directory.

Below is a sample crontab entry to run the script.

0,5,10,15,20,25,30,35,40,45,50,55 * * * * root /var/log/httpd/trap >/dev/null 2>/dev/null

(all on one line)

If you cut and paste the script that follows, do not do it from this page's source code since the < & > characters are represented with special characters.

Drawbacks
• Requires you to manually remove the route table entries.
• Since it just reads from the start of the log, you must also rotate or clear your error log to prevent the IP from forever being blocked.
• Does not adapt to other log formats automatically. This script uses the standard Apache Error_log format.
• The script only can deal with one log file.

  /sbin/route -n del -host 123.123.123.123 gw 192.192.192.192

#------------SCRIPT-STARTS-BELOW-THIS-LINE-----------#
#!/usr/bin/perl

#copyright 2000 bignosebird.com

#Path to sendmail on your server. the -t option must be used.
 $SENDMAIL="/usr/sbin/sendmail -t";
#Path to name of your error_log
 $ERROR_LOG="/var/log/httpd/error_log";
#Path to name of the list of banished IP's
 $BLACKHOLE_LIST="/var/log/httpd/blackhole";
#The Dead IP address on your LAN.
 $BLACKHOLE_IP="192.192.192.192";
#The maximum number of errors you wish to tolerate
 $MAX_ERRORS=500;
#The e-mail address to send the notice to
 $NOTIFY="me\@myaddress.com";
#The return address for the notice
 $MAILFROM="root\@myaddress.com";
#The path to your route command
 $ROUTE="/sbin/route";
  
  @blackholed=();
  &read_list;
  &read_errorlog;
  &process_file;
  &rewrite_list;

sub read_errorlog {
  open (IX,"<$ERROR_LOG");
  while (my $line=<IX>) {
    if ($line=~/Spin Client/i || $line=~/trace/) {
     next;
    } 
    chop $line;
    $line=~s/\[//g; 
    $line=~s/\]//g; 
    my @parts=split(/\s+/,$line);
    my $ip=$parts[7];
    if ($ip !~/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]/){
      next;
    }
    if ($master{$ip} eq "") {
      push(@iplist,$ip);
      $master{$ip} = 1;
    }
     else { 
      $master{$ip} = $master{$ip} + 1;
     }
  }
  close (IX);
}
  
sub process_file {
  foreach my $ip (@iplist) {
     if ($master{$ip} > $MAX_ERRORS && &holed($ip) == 0) {
      # this means we have a problemo 
       push(@blackholed,$ip);
       my $timestamp = time;
       my $stamp="$ip\|$timestamp";
       push(@outlist,$stamp);
       &notify_webmaster($ip,$master{$ip});
       &kill_ip($ip);
     }
  }
}

sub read_list {
  open(BLKHOLE,"<$BLACKHOLE_LIST");
  while (my $bh=<BLKHOLE>) {
    chop $bh;
    my @subs=split(/\|/,$bh);
    my $badip=$subs[0];
    push(@blackholed,$badip);
    push(@outlist,$bh);
  }
  close(BLKHOLE);
}

sub rewrite_list {
  open(BLKHOLE,">$BLACKHOLE_LIST");
  foreach my $towrite (@outlist)
   {
    print BLKHOLE "$towrite\n";
   }
  close(BLKHOLE); 
}

sub notify_webmaster {
  my ($ipaddy,$count)=@_;
  my $MSG=<<__END_OF_MAIL__;
To: $NOTIFY
From: $MAILFROM
Subject: $ipaddy blackholed

The IP address $ipaddy was blackholed after $count errors.

__END_OF_MAIL__
  open (SM,"|$SENDMAIL");
  print SM "$MSG\n";
  close(SM);
}

sub kill_ip {
  my ($ip)=@_;
  system("$ROUTE -n add -host $ip gw $BLACKHOLE_IP");
}

sub holed
{
  my ($item)=@_;
  foreach my $tocheck (@blackholed) {
    if ($item eq $tocheck) {
     return 1;
    }
  }
  return 0;
}


#------------SCRIPT-ENDS-HERE------------------------#