Alexa Rankings Debunked? I Still Think It’s Useful.

by Teresa Valdez Klein on March 5, 2007

John Battelle linked today to Peter Norvig’s comparison study of Alexa vs. actual site statistics.

We all know that Alexa ranking is a fuzzy measure of a site’s actual traffic. It is extremely vulnerable to selection bias. But I contend that it’s still a useful tool.

When we were working to determine which bloggers should get press passes to CES, we looked at Alexa ranking as one of many factors to determine whether or not the blogger had a significant enough audience to qualify as “press.” When combined with a number of other qualitative and quantitative factors, Alexa rank can be a good indicator.

Norvig’s results should serve as a useful reminder that no one statistic or qualitative assessment — especially one that is susceptible to so much bias — should be used as the definitive indicator of a site’s merit.

{ 3 comments… read them below or add one }

1 Geoff Mack 03.06.07 at 9:54 am

Hi Teresa, and thanks for the plug. I am troubled by the recent blog postings by Peter Norvig, Matt Cuts and John Battelle, among others. While their conclusions about Alexa’s selection bias are not entirely wrong, as we clearly admit on our own site, their methodology is completely and utterly flawed. What is so troubling is that these smart people, industry leaders, have such a naive understanding of traffic.

Let me just pick on Peter for a second. He has taken visitors and pageview numbers from 5 different sites and compared them to Alexa Reach and Alexa Pageview numbers. But where did he get his data? Did they all come from the same program? No. The numbers were self-reported and were all generated from different programs. Why is this a problem? This is a major, major problem, and by itself completely invalidates his test because different programs count different things AND count them in different ways. For example, do they count RSS feeds? What about multiple views of the same page by the same user? What about bots? In Peter’s site stats, he is counting all of those. In Matt’s site, it appears that none of them are counted. Oops.

And, then there is Alexa. Does Alexa count these things the same way? No. All stats counting programs are different. Alexa doesn’t count bots/crawlers. Alexa discards multiple views of the same page by the same user. Alexa doesn’t count whole pageviews, rather, it counts them as a fraction of all pageviews on the Web. These things are different.

So, while the conclusions of these Internet professionals are essentially correct, that Alexa has a selection bias, the naive and sloppy analysis is disturbing, irresponsible and less reliable than the stats that they criticize.

2 Russell Rockefeller 04.13.07 at 9:07 pm

You know. I never did understand how Alexa became such a benchmark in the first place. It was completely obvious to me since day one that the results were inaccurate. I think a lot of the industry went along with the idea of increasing Alexa rank because it was so easy to influence. SEO optimizers could point to a better Alexa ranking and say “See you got your money’s worth!”.

3 Ahmad 04.17.07 at 11:56 am

You guys all make a good point, but you have to realize that since it is the only gauge that helps to assess advertising value, web value, and overall comfort of the consumer it is important…If you are dealing with a person with a 12K or below alexa rank…You are dealing with a heavy hitter…

Thanks,

Ahmad

Sponsored links

advertise here