Cloaking is showing search engines other content then human visitors. To use cloaking effectively you need to detect whether your visitor is a searchbot.

In this post I will try to explain…
the process of implementing a cloaking script.

User agent or IP-based detection?
I will always recommend IP-based search engine detection. It is too easy for search engines to detect user agent based cloaking. It is very important!! to have an up to date IP database! I only work with the best IP database providers and always re-synchronise to keep up to date.

Search engine IP databases
Depending on your programming knowledge you can use several IP database providers that always have an up to date database. I’ve only listed the ones I can recommend. These list every engine by name and bot purpose.

• Fantomaster
  Fantomaster provides several ways to help you detect search engine spiders. The SpiderSpy service provides both database access and scripting help to make it easy for you to keep your database up to date.
  price $258 per year
• IP-delivery.com
  A quicker script then Fantomaster and  a guarantee on being up to date. But for almost four times the price I’ve never tried it. Some friends swear this is the best.
  price $995 per year
• Reverse DNS lookup
  You can also check every IP address visiting you and see if they belong to the Google domain. This article provides the Google explanation how to detect their crawling spiders.

Using default cloaking scripts
Fantomaster and IP-delivery.com provide free scripts with their service and you can easily make a function that gives back a simple yes or no result. You can check if the visitor is a search engine and you can also detect each search engine separately. Then you can program the different content you want to show them.

Htaccess example
My favourite way to use Fantomaster SpiderSpy is by making a RewriteMap for mod_rewrite on Apache servers. You implement this by doing the following.

Use the following instructions to generate the RewriteMap:

• Generate an empty text file with write rights for PHP (for instance rewritemap.txt).
• Customize the following php script:

  <?php
   
  $fantomaster_user = “”;
  $fantomaster_pass = “”;
  $rewritemap_url = “/home/vhosts/yourdomain.com/www/rewritemap.txt”;
   
  function doRequest($user_pass) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, ‘http://fantomaster.com/dardanelles/registerdb/fabotbasexml.cgi’);
    curl_setopt($ch, CURLOPT_USERPWD, $user_pass);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_CRLF, true);
    $data = curl_exec($ch);
    curl_close($ch);
   
    if ($data) {
        return $data;
    } else {
        return curl_error($ch);
    }
  }
   
  $rawxml = trim(doRequest($fantomaster_user.’:’.$fantomaster_pass));
  $matches_host = array();
  $matches_ip   = array();
  preg_match_all(‘%([^<]*)%’, $rawxml, $matches_host);
  preg_match_all(‘%([^<]*)%’, $rawxml, $matches_ip);
   
  $matches_host = array_unique($matches_host[1]);
  $matches_ip   = array_unique($matches_ip[1]);
   
  $spidertxt=”;
  foreach($matches_host as $host){
   $spidertxt.=$host.’ spider’.”\n”;
  }
  foreach($matches_ip as $ip){
   $spidertxt.=$ip.’ spider’.”\n”;
  }
   
  if($f = fopen($rewritemap_url, ‘w’)){
   fwrite($f, $spidertxt);
   fclose($f);
  }else{
   echo ‘failed to write’;
  }
   
  ?>

• After running this script you should have a textfile with IP adresses and hostnames followed by “spider”. This textfile must be specified in your httpd.conf (server wide) or vhost.conf (just for this domain) file to be able to use it in a RewriteCond. The problem with shared hosting is that you can probably not edit it yourself.The conf file should contain the following line:

  RewriteMap allBots txt:/home/vhosts/yourdomain.com/www/rewritemap.txt

• Then you can address allBots from a rewrite condition in your .htaccess file.RewriteEngine On

  RewriteBase /
  RewriteCond   ${allBots:%{REMOTE_HOST}} =spider [OR]
  RewriteCond   ${allBots:%{REMOTE_ADDR}} =spider
  RewriteRule    ^(.*)$  otherurl.php  [L]

  More on mod_rewrite and RewriteRules can be found here.

  The rule above redirects any request from a searchbot to otherurl.php.

• Update the list every 12 hours by running a cronjob on the PHP file you generated.
• And you’re done!

Cloaking examples
You can probably think of many ways to use cloaking to your advantage. Here are some examples.

Whatever you use cloaking for, use it wisely. My tip of the day: Always keep your IP database up to date! If less decent ways of cloaking are detected they will be punished and your site will probably be removed from the index.

This entry was posted on Saturday, November 4th, 2006 at 10:59 am and is filed under Cloaking, Tutorials. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.