Bing Search Engine and Awstats Log Stats Analyzer

I run two dedicated servers running Vitualmin with Awstats and Webalizer log stats analyzer programs installed. I don’t find Webalizer stats particularly useful, so disable running Webalizer log analyzer.

I like the detail of Awstats log analyzer, but looking at the Awstats website it appears they don’t update particularly frequently, as I type this the last stable update was Awstat 6.9 2008-12-28 (almost 7 months ago) and the latest beta release Awstats 6.95 2009-05-29 (almost 2 months old).

Bing is a new Search engine (new as far as it’s domain name anyway) and so Awstats 6.9 doesn’t include Bing stats reporting. Awstats 6.95 does include Bing stats reporting, but I normally don’t use beta releases on a deployed server, too risky, so as far as Awstats is concerned I’m working with Awstats 6.9 until Awstats 6.95 goes stable release.

Virtualmin reports the version installed as 6.8 but it has the same date as the 6.9 release, so think I’m running the latest stable version of Awstats. Unfortunately Awstats 6.9 doesn’t include reporting for the search engine Bing from Microsoft so I have to add it manually.

Not sure why, but Microsoft keeps renaming it’s search engines:

MSN Search = Windows Live Search = Live Search = Bing Search Engine

I’m sure Microsoft has made major changes to their search engine over the years, but could you imagine Google changing it’s name so often! Quite funny really, taken Microsoft about 10 years to realise it has to copy Google in it’s search engine branding, give it an easy to remember almost made up name. I wonder if Binging will ever be considered a noun like Googling is

Note: My wife is reading this over my shoulder and is arguing Googling is a verb not a noun! I think it’s a noun, and it’s my website, so there!!!

Yesterday was the 40th anniversary of the moon landing and I own a site about conspiracies including a page about the Moon Landing Hoax.

This site normally sees 500 to 1,000 visitors a day, yesterday it received over 20,000 visitors (over 30,000 AdSense impressions, kerching ) so was very interested to see where it was all coming from. Well a lot of it (most of it) was from the new Bing search engine, but because Awstats lacks Bing reporting I couldn’t pull any useful Keyword data via Awstats log analyzer!

When I search for relevant searches in Bing the site is listed top 15, so wasn’t sure where all the traffic was coming from (I couldn’t be bothered looking through the raw log files). So time to update Awstats to report Bing.

Adding Bing Search Engine Log Stats to Awstats

To add a search engine to Awstats you have to edit a file called search_engines.pm

In Virtualmin the file is located under

/usr/share/awstats/lib/search_engines.pm

Downloaded a copy and edited it in a text editor.

Under %SearchEnginesSearchIDOrder_list1=( add

'bing\.com',

I added it above ‘base\.google\.’,

Under %SearchEnginesHashID=( add

'bing\.com','bing',

I added it above ‘base\.google\.’,'google_base’,

Under %SearchEnginesKnownUrl=( add

'bing','q=',

I added it above ‘alexa’,'q=’,

Under %SearchEnginesHashLib=( add

'bing','<a href="http://www.bing.com/" title="Search Engine Home Page [new window]" target="_blank">Bing</a>',

I added it above ‘alexa’,'<a href=”http://www.alexa.com/” title=”Search Engine Home Page [new window]” target=”_blank”>Alexa</a>’,

Save the file and upload over the original (don’t forget to make a backup).

That should do it.

It worked, but only for new search engine data.

Rebuilding Awstats Stats Data Log Files

To get Awstats to rebuild the log file data for July again you have to delete the file that holds the July 2009 log file stats and either wait for the cron job to rebuild the months file (should be within 24 hours) or run the cron job yourself (I went with the latter).

This would be a real pain if I wanted this information for all my domains (over 100), but I’m only interested in this one domain for this one month, so not a big deal.

Log into the domains ftp server and go to the root of the server (BELOW /public_html/, so you have to go back a level, not up). Load the folder /awstats/ here you will find each months data files.

The format of these files is

awstatsDATE.domain.com.txt

Where DATE is the date in this format monthyear, for example:

awstats072009.conspiracy-theories-hoax.com.txt

is July 2009 data file.

On my dedicated server there are two files for each month, one for the www version one for the non www version, not sure why this is since I 301 redirect the non www to the www version.

Delete or better yet rename these two files (by renaming you have a backup on the server, easy to find if something goes wrong).

If you check Awstats log analyzer for the domain now the last month will be empty, but previous months will still be there.

Either wait for the Awstats cron job to run (should be under 24 hours wait) or find the relevant cron job and run it now. I ran it now and this rebuilt July 2009 data.

I read about rebuilding all months, but it sounded a big pain and as I don’t need older data in this format (Bing is a new search engine and Google sends most traffic) I didn’t bother.

Now for the BIG surprise.

Bing Search Engine is my New Friend

The new rebuilt stats data showed over 17,000 visitors to one search engine result, one I’d not checked or thought of as important:

moon landing hoax letter c

Since I don’t use Bing as a search engine (I love Google, I love Google…..) I can only assume a link to that SERP was on the Bing home page or something for that day.

David Law