How does it work?
With the new canonical tag you indicate that there might be more than one URL within your website presenting this specific information and if search engines want to choose between those when presenting it to visitors, you would prefer the given URL. I for instance use wordpress to rewrite to nice URLs but this post is available under http://www.vdgraaf.info/?p=120 as well.

If I would place <link rel="canonical" href="http://www.vdgraaf.info/canonical-tag-creates-new-legal-cloaking-possibilities.html" /> in the head of this page, Google would only show that one to visitors. But just like a 301-redirect Google would attribute all linkjuice to the remaining page.

The canonical tag is intended for sites that cannot do a proper 301 redirect because of several reasons. For instance if there still are slight differences between the pages under different URLs, but too little for Google to see them as unique. The same article might be available under different categories, an affilliate ID in the URL just changes some form values or you use almost the same text for the French speaking population of Canada as you do for people in France. I can think of many more, but all these legitimate reasons allow for slight changes to the page served. And it is logical that search engines now allow you to indicate which version to choose.

One page, different audiences
Your website is created for several audiences and in the ideal situation you could change nuances depending on the visitor type. The most important audiences we want to differentiate right now are “linkers”, “navigators” and “searchers”.

• People linking to your website
  A clean webpage without much commercial intent is far more likely to recieve inbound links than a page with a lot of branding and call-to-actions. A page that shows your good side (for instance with a reference to your altruism to good causes, or a reference to the specific link partner) will give you a far higher success rate on your link building effort. Use a URL like http://www.vdgraaf.info/i-am-a-good-boy.html for them.
• People navigating from within your website
  People that navigate through your website follow a certain path and you can offer more specific content depending on where they came from. http://www.vdgraaf.info/so-you-clicked-the-banner.html would be good for this audience.
• People from search engines
  A good landing page for search engines gives the answer to what you searched for and shows a clear call-to-action to the most logical next step. Besides changing the appearance of the page that you would like search engines to serve, you could also use slightly different code to be more search engine friendly. You should make this version the canonical one by adding <link rel="canonical" href="http://www.vdgraaf.info/this-one-is-for-searchers.html" /> to the head section of all your versions.

Does the tag give real cloaking abilities
It is logical that Google will or already has implemented a check on simularity between different versions. Because the tag is onpage and not in an external file like the robots.txt, the page has to be spidered first. It is a small extra step to do some text calculation as well. The canonical instruction will probably be followed when the page would normally trigger duplicate content filter. It will however assign all accumulated links to that one remaining occurrence.

So use this tag with care, but with the same content you can still create many different versions. Have fun!

In this post I will try to explain…
the process of implementing a cloaking script.

User agent or IP-based detection?
I will always recommend IP-based search engine detection. It is too easy for search engines to detect user agent based cloaking. It is very important!! to have an up to date IP database! I only work with the best IP database providers and always re-synchronise to keep up to date.

Search engine IP databases
Depending on your programming knowledge you can use several IP database providers that always have an up to date database. I’ve only listed the ones I can recommend. These list every engine by name and bot purpose.

• Fantomaster
  Fantomaster provides several ways to help you detect search engine spiders. The SpiderSpy service provides both database access and scripting help to make it easy for you to keep your database up to date.
  price $258 per year
• IP-delivery.com
  A quicker script then Fantomaster and  a guarantee on being up to date. But for almost four times the price I’ve never tried it. Some friends swear this is the best.
  price $995 per year
• Reverse DNS lookup
  You can also check every IP address visiting you and see if they belong to the Google domain. This article provides the Google explanation how to detect their crawling spiders.

Using default cloaking scripts
Fantomaster and IP-delivery.com provide free scripts with their service and you can easily make a function that gives back a simple yes or no result. You can check if the visitor is a search engine and you can also detect each search engine separately. Then you can program the different content you want to show them.

Htaccess example
My favourite way to use Fantomaster SpiderSpy is by making a RewriteMap for mod_rewrite on Apache servers. You implement this by doing the following.

Use the following instructions to generate the RewriteMap:

• Generate an empty text file with write rights for PHP (for instance rewritemap.txt).
• Customize the following php script:

  <?php
   
  $fantomaster_user = “”;
  $fantomaster_pass = “”;
  $rewritemap_url = “/home/vhosts/yourdomain.com/www/rewritemap.txt”;
   
  function doRequest($user_pass) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, ‘http://fantomaster.com/dardanelles/registerdb/fabotbasexml.cgi’);
    curl_setopt($ch, CURLOPT_USERPWD, $user_pass);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_CRLF, true);
    $data = curl_exec($ch);
    curl_close($ch);
   
    if ($data) {
        return $data;
    } else {
        return curl_error($ch);
    }
  }
   
  $rawxml = trim(doRequest($fantomaster_user.’:’.$fantomaster_pass));
  $matches_host = array();
  $matches_ip   = array();
  preg_match_all(’%([^<]*)%’, $rawxml, $matches_host);
  preg_match_all(’%([^<]*)%’, $rawxml, $matches_ip);
   
  $matches_host = array_unique($matches_host[1]);
  $matches_ip   = array_unique($matches_ip[1]);
   
  $spidertxt=”;
  foreach($matches_host as $host){
   $spidertxt.=$host.’ spider’.”\n”;
  }
  foreach($matches_ip as $ip){
   $spidertxt.=$ip.’ spider’.”\n”;
  }
   
  if($f = fopen($rewritemap_url, ‘w’)){
   fwrite($f, $spidertxt);
   fclose($f);
  }else{
   echo ‘failed to write’;
  }
   
  ?>

• After running this script you should have a textfile with IP adresses and hostnames followed by “spider”. This textfile must be specified in your httpd.conf (server wide) or vhost.conf (just for this domain) file to be able to use it in a RewriteCond. The problem with shared hosting is that you can probably not edit it yourself.The conf file should contain the following line:

  RewriteMap allBots txt:/home/vhosts/yourdomain.com/www/rewritemap.txt

• Then you can address allBots from a rewrite condition in your .htaccess file.RewriteEngine On

  RewriteBase /
  RewriteCond   ${allBots:%{REMOTE_HOST}} =spider [OR]
  RewriteCond   ${allBots:%{REMOTE_ADDR}} =spider
  RewriteRule    ^(.*)$  otherurl.php  [L]

  More on mod_rewrite and RewriteRules can be found here.

  The rule above redirects any request from a searchbot to otherurl.php.

• Update the list every 12 hours by running a cronjob on the PHP file you generated.
• And you’re done!

Cloaking examples
You can probably think of many ways to use cloaking to your advantage. Here are some examples.

Whatever you use cloaking for, use it wisely. My tip of the day: Always keep your IP database up to date! If less decent ways of cloaking are detected they will be punished and your site will probably be removed from the index.

When should you cloak?

• Cloaking javascript and CSS
• Cloaking to get your competitors links

First some basic questions answered:

What is Cloaking?
Cloaking is showing a search engine other content then normal visitors.

What does Google find plausible reasons to cloak?
When you have a website that is hard to index by a script, you can have an alternative version with the same content available.
Or when you have a password protected area you want Google to index, you can let him in without a password. Just add “noarchive” to your robots metatag, otherwise people just look at the Google cache.
There are more reasons, but the rule is: “If it improves the searchbot crawlability of the same content a user sees”.

Why should I cloak?
Normally you shouldn’t! Make good sites with good code and both users and Google should see the same. If your code is unfriendly for search engines, rewrite the code and don’t cloak.

There are situations where you want to draw a users attention to one place and Googles attention to another. Normally I’d use images to draw a users attention to my “call to action” and headers to draw Googles attention to the most important text. So in most situations you don’t need cloaking for this.

If you’re a good boy, don’t use cloaking! If you’re a bad boy (or girl), do it!

But there are good spammy tactics?
Of course there are many fun ways to use cloaking to your advantage. Most of the time you wan’t both linkable content and search engine friendly content. This could become a compromise and you should always want the best of both. Here are some fun tactics with cloaking.

I hope Matt Cutts doesn’t read this post because there are ways to detect the following tactics in an algorithm, they just don’t detect them yet. My blog is too small to be detected by Matt Cutts (the Google spamcop), but when he does: “Matt please leave a comment!”

Javascript and Stylesheet tactics
With external javascript and stylesheets you can change the appearance and visibility of your content. Google sees this and devalues hidden content.

I tried to disalow my .JS and .CSS files from being indexed in the robots.txt, but Google disobeys this and still reads them. Then I tried IP cloaking (detecting if an IP belonged to a search engine and showed different javascript and stylesheet information) and it worked.

You can hide a block of content, you can show an H1 header as normal inline text, you can hide links and much more all by the use of javascript or css. Currently the detection of cloaking isn’t very sophisticated and algorithmicly not many spammers are caught. When you get caught on cloaking it is mostly because someone ratted on you to Google. If you just cloak the .JS and .CSS files people don’t see any difference between the search engine cache and normal file. The cached version still uses the users version of external files.

Cloaking for linkbuilding
This way is more dangerous because people can detect it more easily and tell Google about it. But many people ask me: “How do I get links from my competitors to my commercial website?” Here is a possible answer to this question!

 
• Make a non-commercial website about a subject your competitors might be willing to link to and point them to it. Don’t let it have any link to you or your commercial activities.
  For instance: Results of the top 5 SEO companies competition website.
  Email: Congratulations you made 1st place!
  Result: They will proudly link to it.
• When they start linking you want to divert the linklove to your commercial activities without loosing the links.
• Either cloak and place good links to your commercial website or use a 301-redirect to divert all linkpoints.
  The 301 redirect shows no cache in Google, thus no trackback to where it is going.
  Competitors just don’t see it show up in Google. From their IP the domain shows the normal site, so no reason not to link to it.
• The links will remain and could even grow, but the love is diverted!

I hope these tips were helpfull. Cloaking is seen as illegal by most search engines and you should try this at your own risk. Everything they don’t detect at this moment, can be detected in the future.