You can now view keyword volume estimates in the Adwords keyword tool!

So far Google has never been “close to” open about search volumes. Google trends only shows charts without numbers and it has a threshold of about 1000 searches per day. Previously the Adwords keyword tool showed only monthly graphs without any reference field to numbers. But at last they changed their ways and show “rough estimates” with real numbers.

In European countries where Google has a marketshare of more than 90% this is close to a revolution. Yahoo data, Trellian, HitWise, Wordtracker and Alexa are all useless because of the limited coverage in those countries. Hopefully Google will improve even more by showing “real numbers” in stead of the current “rough estimates”. But even the current improvement is a welcome change.

In this part of the article I will explain how you get server-side information into client-side statistics and vice versa.

• Part 1: What is the difference between server-side and client-side
• Part 2: How to combine server- and client-side webanalytics (this article)

Part 2: How to combine server- and client-side webanalytics

Google Analytics
For an explanation how client-side web analytics works, I will use the most common one “Google Analytics” as an example. Tools like Google Analytics work with a Javascript that requests an image. The image request is interpreted by a server-side script that creates the log files (and it always returns the same image to the visitor). The log files are then interpreted by a program that creates all kinds of fun statistics, for people to interpret into the visitor behavior they want to research.

Because the Javascript and the image are requested from Google servers with Javascript (their browser), the visitor comes in direct contact with those servers and leaves the much needed extra information that javascript cannot detect. This primarily includes visitor IP address and Google cookie. The Google cookie also links to language preferences and other data not available in other web analytics tools.

Because the Javascript is located on google.com that may be listed as a trusted domain in the browser, it could fall under lower safety restrictions and recieve more information. Governmental agencies sometimes have very strict Javascript settings and Google is often one of the exceptions (I just can’t imagine why).

Communication from server-side to client-side
PHP and Javascript have several common grounds where they can exchange data.

• Javascript and PHP can both read and write the same cookies.
• Javascript and PHP can both generate and read URLs and extract GET variables.
• Javascript and PHP can both generate and read POST variables.
• Javascript can read and PHP can read/generate text files or XML files.
• Javascript can be generated by PHP.

Server-side information in Google Analytics
Just a few examples of server-side data that you might want in Google Analytics or other client-side web analytics programs.

• Your internal client ID.
• Real e-commerce sales.
• Files and visitors without Javascript support.

So how do you get it there?

Your internal client ID
If someone is logged in to your system you can identify that visitor, but how do you get that data in your Analytics report? … Lets say you have a variable $strClientID. You can enter it as the Google Analytics “user defined” variable by doing the following:

<script src="http://www.google-analytics.com/urchin.js" type="text/javascript"></script>
<script type="text/javascript">
_uacct = "UA-000000-0";
__utmSetVar(‘<?php echo $strClientID;?>’);
</script>

Real e-commerce sales
When you enable e-commerce tracking in Analytics you can communicate every product and every transaction to Google.
Look at this text file to see how it is done.

For more information look at
http://www.google.com/support/googleanalytics/bin/answer.py?answer=55528

Files and visitors without Javascript support
Look at this post about Google Analytics without Javascript. This requires some PHP skill, but you can probably add your own functionality to the script I’ve made. I’ve made 20 different versions for different functionality, so I won’t bother you with all of them.

The only drawback is that I’d recommend using a separate profile because all visits are logged on the IP address of your hosting provider. In analytics software “Webtrends” I did the same and then you can enter the IP and data of the client provided by your own PHP script.

Summing it up
There are several ways to communicate server-side data to javascript and thereby to Google analytics and other client-side web analytics.

I’d recommend on using client-side web analytics tools and server-side support for extra data. This is the most trustworthy to get the data you need to make sound assumptions about your visitors.

• Part 1: What is the difference between server-side and client-side (this article)
• Part 2: How to combine server- and client-side webanalytics

Part 1: What is the difference between server-side and client-side

What does client-/server-side mean?
To explain the difference between server-side en client-side analytics, I first need to explain the difference between server-side and client-side scripting..

• Client-side scripting means all calculations are done by the computer of the visitor. Most of the time the script is interpreted by their internet browser (Javascript) or a specific browser plugin (Flash actionscript).
• Server-side scripting means all calculations are done by the server your website is hosted on. The script is interpreted by the supported language parser, like PHP or ASP.

What can be detected?
Many factors like security issues have led to restrictions in both server- and client-side possibilities. Javascript isn’t even made to be able to communicate back to a webserver, but certain tricks have led to client-side webanalytics and Ajax. Therefore some types or information are only available in Javascript and some information can only be retrieved by PHP. The following table shows information available in both types and the main restrictions.

Client-side summary
Languages like Javascript can detect what browser you are using, your screen size, plugin support and much more. But that may be restricted by the security level you set for your browser. It can also detect and change any property of the page (like the width and background-color of a table) you are watching and change it on the fly, based on any type of detected interaction. This interaction can even be a slight mouse movement or the press of a key.

The javascript source code can be read with any text editor, so don’t place passwords within them. But information a user enters remains client-side (minimizes server load) and normally isn’t send to any server. But there are ways to send information to the server-side.

Depending on the security settings in your browser, javascript can request and read external files like XML feeds. To circumvent stricter security settings, javascript may also request images; And that is what is exploited by many client-side web analytics programs like Google Analytics. The requested file/image URL can be stuffed with information, so the server can retrieve these GET variables from there.
p.s. Javascript can also vacate forms and send them and it can also read/write cookies to communicate with other files.

Javascript web analytics only logs visitors that support this language, in files that allow javascript execution.

Server-side summary
Languages like PHP can request (and change) any property of the webpage it is on and any variable used to while compiling the resulting HTML code. It can also see properties of the connection between visitor and server and unlike Javascript it can detect the IP address. Any change requires a reload or new page request, which requires more loading and a client-side action (can be from both a user or client-side script).

Server-side scripting code can’t be read by visitors, so no one can steal your script and no client support is needed. The calculations are done by the server, so more visitors means more server-load. Different server-side languages, plugins and external data can supplement eachother and the only thing that counts is the end result.

Languages like PHP offer numerous ways to communicate with other servers and files across the web and no one will ever know what was required to compile the data. When a website is hosted on a shared server, the possibilities of server-side can be somewhat limited, but on a dedicated server the only programming limitations are posed by server-load, software licenses and your own technical capabilities.

Standard server-side web analytics is based on server logs. This is a very limiting way of measuring visits and you need many filters to drop search engine robot requests and other non-human visits. But Javascript can’t detect non-human visitors at all.

So how do you combine both in your analytics reports?
How do you get customer information in a javascript web analytics tool? How do you log commerce sales? How do you log search engine robot visits? How do you track requests of images, PDF files or your RSS feed?

All these questions are answered in:
Part 2: How to combine server- and client-side webanalytics

Goedkope vliegtickets

Analytics without javascript?
When you look at the analytics javascript code you see that it combines several sets of data into an image request. This image request sends the right data to Google (not the javascript). When you know what url you should use for the image, you can call the image directly and send the same data. Of course you need to be able to request the image url and that isn’t easy from another image, rss feed or pdf. This is why we request it “server side”.

Server side image requests
Javascript is “client side”; this means the browser does all the calculations and actions that the script requires. PHP, ASP, Perl and many others are server side scripting languages; which means the server that hosts the website does all required calculations. This means javascript knows more about the action of the user and PHP can easily request data from a database or other source.

PHP (I’ll focus on PHP in this example) can also request a file from the internet without executing or showing it. Just requesting the analytics image is enough to trigger a hit in your report and it will show up in your account. The downside of requesting the image from your server is: You lose the IP address and other data that Google records at the moment the image is requested. The server IP of your web server will be recorded.

Server side requests in a client side report will obscure your data. Create a separate Analytics profile for everything you log server side.

Who requests my RSS feed?
RSS feeds don’t execute javascript, but in some feed readers you can execute javascript once a post is read. This is why you need to log requests of your RSS XML file differently. You can either use a service like Feedburner or send an Analytics image request from PHP or ASP. The following example shows how you send specific data to Google Analytics from your RSS in PHP.

1. Create a new analytics profile with its own tracking code (not a copy of an existing profile). Activate the profile by temporarily changing the code on your homepage. After activating you return it to the original one.
2. Add the following PHP code to the file that produces your RSS (preferably at the bottom): urchin-image.txt
3. Update the urchin code, domain, user defined variable (if desired) and the fake page request you’d like to track.
4. I left all the other data intact because it works with them and they don’t impact the report, but when I have time I can probably clean it up some more.

Tracking PDF or Image requests
The script that produces your RSS feed is already executing PHP code. Images and PDF files don’t execute anything, so you need to add something extra to track these in a similar way. The following code shows you how you can request an image, PDF, SWF or for instance downloads and request the Google Analytics (Urchin) code at the same time.

1. Find out if your server supports mod_rewrite. If it does, you can produce cleaner URLs. As a RewriteRule in your htaccess you add something like: “RewriteRule ^tracker/(.*).gif$ /tracker.php?url=$1&filetype=gif [L]“. This requests the file tracker.php whenever a gif image from the (non-existing) directory tracker is requested. You can do the same with all other filetypes that don’t execute PHP code themselves.
2. Create tracker.php and add something like the following code: tracker.txt. Customize the script where needed and add content type headers for everything you’ve added to the htaccess.
3. Now you can open the extra profile you should have created and under “Content Optimization > Content Performance > Content Drilldown” you find the directory “tracker” with all image/file requests logged in it.

Here are all values you can fill: image-url-explained.txt

Most of the information needed for the statistics is gathered by the javascript file (and you can influence that data), but the other half is collected when the image is requested from the Google server. This means that that information belongs to your server and not to the user that did the real request. Converting server-side to client-side analytics is very usefull, but not flawless. I haven’t seen this type of Analytics tweak anywhere else, so please let me know if you would like me to post more similar tweaks or more information in the future.

Google Analytics is very good once you implement some of the following tweaks.

• Extra tracking features in Google Analytics
• Using filters in Google Analytics
• And of course the tweak shown above!

What are Analytics filters?
Google Analytics filters allow you to select specific visits or visitors on your website and either show only them, exclude them from your reports, or change the way they show up in your reports.

You can enter filters in the “Analytics Settings > Filter Manager” section of your account when you have administrator rights. Here you can enter filters that “Exclude all traffic from a domain”, “Exclude all traffic from an IP address”, “Include only traffic to a subdirectory” or make your own filter under “Custom filter”.

What can I do with Analytics filters?
The first two filters mentioned before help you to exclude visitors that cloud your reports with unrelevant visitors, for instance visits from your companies own IP address. The subdirectory filter allows you to separate subsections of your website into separate reports. More on separate website profiles later, but first let me explain how you can use the “Custom filter”.

Custom filter
In the custom filter section you can include, exclude or edit information based on many variables before they are simplified and combined in your analytics reports. The most important pieces of information to use in your filters are: Request URI, Visitor IP address and Referrer, but others like country and language can always come in handy. A full list can be found here.

Custom filters can use POSIX regular expressions. This means you define what format or sequence the filter should look for and you can extract information from it. I like using the “Advanced filter” from the “Custom filter” section. There you can use one or two fields and combine extracts from them into another field.

Full referrer URLs
Under “Marketing Optimization > Visitor Segment Performance > Referring Source” or “Marketing Optimization > Marketing Campaign Results > Referral Conversion” Analytics shows only partially where your visitors have come from. For instance the get variables (for instance ?name=analytics-filters) are excluded and it isn’t easy to just enter the referrer URL and visit it. To get the entire referral URL into your report you can use a custom-advanced filter. Here’s how:

1. Enter the Filter manager
2. Click “+ Add Filter”
3. Enter the filter name (like “Full Referral”) and select “Custom filter” from the drop-down box
4. From the radio buttons you select “Advanced”
5. For “Field A -> Extract A” you select “Referral” from the drop-down box and in the text field you enter “(.*)” (to select the entire URL)
6. Leave Field B empty and set “Output To -> Constructor” to “User Defined”. In the text field you enter “$A1″ to refer to the first selection of field A.

It should end up looking something like this:

Within a day full referral information will end up under “Marketing Optimization > Visitor Segment Performance > User-defined”

What have Adwords searchers really typed?
Another flaw in Google Analytics is the report on Google Adwords. Analytics only shows what matching option was triggered by the users search query, not what was exactly typed. In my earlier article on Analytics I showed how to fix this with javascript, but you can also use filters to extract this data. Here’s how that goes:

Copy the following screen to get referral data of cpc clicks into “Custom field 1″ (this field cannot be seen in your reports)

Copy the following screen to combine the data of “Custom field 1″ and the original data of “Campaign term” into one

From now the all reports that include Adwords search terms will also include the used query.

Using extra profiles

You can create extra profiles of the same website and filter specific traffic to only appear in all reports of that profile. I use this option to see if visitors that order something show other behaviour then other visitors, but you can also use it to differentiate Google traffic from Yahoo traffic, or new from returning visitors. This works in all your reports and you don’t need to use any cross segmentation within your reports.

Google Analytics can do almost anything expansive web analytics software can do. You just need to know how to retrieve the data.

1. Your first ever search engine will be either the default one on your computer or the one someone advices you to use. If you remain satisfied with the results you’re not likely to switch. This is why all search engines want to become the default on new computers.
2. Unexperienced searchers are more likely to type full sentences in stead of only keywords. As people become more acquainted with search they will use only key-words and even start using phrase and negative query options.
3. Unexperienced searchers won’t know the difference between sponsored and organic search results and are more likely to click ads. As experience grows they will sometimes create an automatic ad filter and they won’t even notice the ads anymore.
4. Websites that sell something are more likely to use sponsored listings. And because people aren’t always searching to buy something the sponsored listings are deemed less relevant in most cases. Only when someone is looking to buy, they are more likely to look at and click on ads.
5. The decision phase someone is in determines what keywords they use and what they click. The Search Funnel (pdf) can be used to determine what keywords belong in which phase and shows you which steps need to be taken to end up with a buying customer. Down the AIDA funnel (Attention, Interest, Desire and Action). In which Action can be buying something.

   
   (Click for PDF)

   To get a buying customer someone needs to go through the entire decision making process. If you get your website visitor in the “attention” phase you need to offer him all the following steps and information without loosing him to a search engine which might be able to give him more information then your site does. The further down the search funnel you get your visitor, the more likely someone is willing to buy or take any other desired action. Maybe this article can help: Best landing page layout.

   Target keywords from all phases, but make sure you offer the desired information for that phase. Guide them to the desired action on your site, but in these steps offer answers to possible questions in every decision phase.

6. Most websites are written in only one writing style. They can for instance use commercial language, corporate language or informal language. They’re also written from a specific knowledge level with the presumption that the visitor has that same knowledge. Write your text from all different standpoints, all knowledge levels and every possible writing style that is suitable for your product or service. Just figure out a way to do it all in one website and target all possible search queries.
7. Organic and Sponsored search results are almost always placed in the format Title, Description and URL. Eye-tracking studies on the Google result pages show that it is not only important to be amongst the first five search results, but it is even more important to be have a relevant and attractive result. The second result can easily convert better then the first as long as it is more relevant to what the searcher was looking for or even more then he was looking for.

   
   (Click for full page)

   The different colors show how much time was focussed on a specific region of the page.
   The X-es show where the user clicked.
   The red lines show how far someone scrolled.

   As you can see people don’t scroll much, but that depends on the first five results. If they are exactly what the searcher was looking for, they will click those. If it was nothing like what they were looking for, they will click the search box to refine the search. If it was somewhat what they were looking for, they will scroll to see if there is something better in those results.

   People scan the results for what they were looking for. Having the keyword in your title and description makes them bold. Something bold ate the beginning or end of the line attracts extra attention.

   Make sure your result stands out of the crowd. Titles in search engines are your page titles, the descriptions (snippets) are either: 1. Your DMOZ or Yahoo directory description. 2. Your description metatag. 3. A specific piece of text from your page where the search phrase is situated. Test titles and descriptoins in payed search and reflect the successfull ones in your webpages.

8. The future of search is in personal search, custom result page layouts and vertical search. All of these result types make it harder to predict where and how your result will be shown. Make sure you end up in every relevant vertical search type. In a future post I will focus on influencing personal and vertical searches. It isn’t very important yet, but just realize that this is coming!

These are just a few of my findings. The article would become too long if I continued. Maybe I’ll focus more future posts on this subject.

Script in action
You recently visited:

How does it work?
The trick of history detection is looking at the color of visited links. Therefore you need to use a separate color in your stylesheet for visited links. Thanks to Jeremiah Grossman an Robert Cabri for this easy to use script. The script uses javascript and should work in almost all browsers. You need to know the hexadecimal and RGB values of your visited color. The security settings in browsers prevent you from looking at the entire history file, but if you have a link to a visited website on your webpage it will have the a:visited properties attached by it by the browser. Javascript can then look for these properties and communicate them back to for instance Google Analytics or any other javascript command.

The script
Fill in your own a:visited color (make sure it is different from a:link and a:hover). Use this hexadecimal to rgb converter to find out the RGB value for Firefox. Then fill in every URL (each separate webpage) you want the script to check for in the visitor’s browser history.

Just look at my own javascript http://www.vdgraaf.info/js/javascripts.js or the code below:

addEvent( window, ‘load’, stealHistory );
 
var IEVisitedColor = ‘#cd0018′;
var W3CVisitedColor = ‘rgb(205, 0, 24)’;
 
var websites = [
  "http://ajaxian.com/",
  "http://www.dicabrio.com",
  "more of your sites...."
];
 
function stealHistory()
{
  if(document.getElementById(‘site-list’))
  {
    var List = document.getElementById(‘site-list’);
 
    for( var i = 0; i < websites.length; i++ )
    {
      var bRemove = false;
      var ListItem = document.createElement('li');
      var Link = document.createElement( 'a' );
      Link.href = websites[i];
      Link.id = i;
 
      Link.appendChild(document.¶
createTextNode(websites[i]));
      ListItem.appendChild(Link);
      List.appendChild(ListItem);
 
      if( Link.currentStyle )
      {
        var color = Link.currentStyle['color'];
 
        if( color == IEVisitedColor )
        {
          bRemove = true;
        }
      }
      else if( document.defaultView.¶
getComputedStyle( Link, null ) )
      {
        var color = document.defaultView.¶
getComputedStyle( Link, null ).color;
       
        if( color == W3CVisitedColor )
        {
          bRemove = true;
        }
      }
 
      if( bRemove == true )
      {
        List.removeChild(ListItem);
      }
      else
      {
        urchinTracker( '/visited/' + websites[i] );
      }
    }
  }

And then
In Google Analytics you can now find the visited URLs under “Content Optimization” -> “Content Performance” -> “Content Drilldown” under the fake directory “/visited”. Feel free to customize this script to do whatever you like with your visitors history. It can also be used to offer specific content directly. Like “I see you have visited … You probably don’t know that our service is much …”

This script tracks every javascript enabled visitor. Not just the users of MyBlogLog.

]]> http://www.vdgraaf.info/hey-you-where-have-you-been.html/feed 0 http://www.vdgraaf.info/google-analytics-tweaks.html http://www.vdgraaf.info/google-analytics-tweaks.html#comments Wed, 03 Jan 2007 22:22:04 +0000 Peter van der Graaf http://www.vdgraaf.info/google-analytics-tweaks.html Google Analytics is the best free web analytics software. It has just a few flaws and some of them are easily fixed. In this article I will show you a few tweaks to get better reports from Google Analytics.

AdWords keyword tracking
One of the biggest flaws in Google Analytics is probably in the reports on AdWords usage. When you visit “Marketing Optimization -> Search Engine Marketing -> AdWords Analyses (or “CPC vs Organic Conversion -> google[cpc]“)” Analytics only returns the matching options that were triggered, but not the exact typed in search phrase.

When you need to tweak your keywords in AdWords you really need to have good keyword statistics. The following script looks at the refering url to your webpage and looks for the word “google”. If that exists it looks for the query and pushes it as a fake page request to Analytics. This means both AdWord and organic Google results will be logged, but it is unwise to use a special URL to differentiate them from each other. You can use the Adwords autotagging option and edit the script to look for “gclid=…” in the URL. This is an indicator of adwords that is added to your URL after your landing page quality score is calculated.

if( document.referrer )
{
  if( document.referrer.indexOf( ‘google’ ) != -1 )
  {
    var urlPattern = /(\?|&)q=([^&]*)/;
    var aMatches = urlPattern.exec( document.referrer );    if( aMatches != null )
    {
      urchinTracker( ‘/query/’ + aMatches[2] );
    }
    else
    {
      urchinTracker();
    }
  }
  else
  {
    urchinTracker();
  }
}
else
{
  urchinTracker();
}


To see what search phrases have been logged you go to “Content Optimization -> Content Performance > Content Drilldown” in Google Analytics and look for the (fake) directory “query”. Here you find every search phrase from Google. When you compare these figures to “Marketing Optimization -> Search Engine Marketing -> CPC vs Organic Conversion -> google[organic]” you see what is left for AdWords.

Target tracking
Even when you’re not selling anything directly from your website, you can always give certain goals a financial value. This virtual value gives you a good indication which marketing campaign is producing the highest value. For instance: Whenever your contactform is used give it a value of one or two dollar and log it as “contactform submitted”.

For more information see: http://www.google.com/support/analytics/bin/answer.py?answer=27203&topic=7282

Campaign tracking
For each separate campaign that uses a specific landing page URL, log it! This way revenue from a mailing you did ends up in a separate report.

For more information see: http://www.google.com/support/analytics/bin/answer.py?answer=27254
Or even better: http://www.google.com/support/analytics/bin/answer.py?answer=27248

Tracking link clicks
If you want to track outbound links or see which one link (when there are several links to the same url on one page) was clicked, you need to send a fake page request to Analytics. Just like the /query/ request we did for AdWords you can do an onclick=”urchinTracker(‘/link/….’);” request.

Here is my link to an unobtrusive javascript example:
http://www.dicabrio.com/javascript/unobtrusive-google-analytics.php

Using Google Analytics
Google Analytics is a fine web analytics tool for companies that make some money online. When you start earning the big bucks every content/navigation optimization can save you thousands of dollars and you will probably be better of with someting like Webtrends or Clicktracks. These give almost instant reports and can give custom visitor segmentation throughout all your reports.

Another reason not to use Google Analytics is: Do you want to give Google this much information on your website and is it unbiased?

Get acquainted with web analytics with Google Analytics, but continue to professional web analytics software!

Read these when you are finished:

• Google Analytics Filters
• Google Analytics without Javascript
]]> http://www.vdgraaf.info/google-analytics-tweaks.html/feed 0