SPECIAL REPORT: Nielsen Online Chief Explains the Science of Counting Web Traffic

By: Jennifer Saba When the topic of traffic measurement arises, many online managers express exasperation, confusion, and sometimes, downright anger. For several years debate has raged in the Web world -- and not just at newspapers -- about the process of pegging page views, unique visitors, and/or time spent at sites. When they?re unhappy with the latest numbers, many question the effectiveness of the measuring tool. If they like the latest reports, everything may seem just fine -- until another audience-measurement tool casts doubt.

But there?s one thing nearly everyone agrees on: Measuring audience data on the Web can be maddening.

Panel-based measurement services such as Nielsen Online (formerly Nielsen//NetRatings) and comScore search the U.S. for willing members to serve on their ?panels.? It?s a survey in which participants download to their computers software allowing those data-collecting firms to track online activity.

The site-centric method, on the other hand, goes directly to Web servers and ticks off how many people come to the site by counting IP addresses.

There are pluses and minuses to both methods. Many feel that site-centric measurement over counts traffic; for example, people who delete their cookies might be counted several times. Yet the panel-based method has been accused of under-counting because of limited survey samples or because they allegedly monitor too few people surfing at work.

But the hard fact is, when comparing panel and site-centric data for the same Web site, the numbers swing wildly. And since traffic is the currency of the Web, this presents a huge problem.

Many newspapers, especially the major metros, subscribe to Nielsen or comScore or a site-centric services (or some combination thereof). But Nielsen enjoys extra prominence since it is used by the Newspaper Association of America to track online activity for the industry as a whole ? as well as for NAA?s NAdbase.

No judgment was made in the selection of Nielsen, says Randy Bennett, the NAA?s vice president of audience and new business development. Since Nielsen Online (like E&P, owned by the Nielsen Co.) was already providing NAA?s monthly industry numbers, it decided to use Nielsen for NAdbase to keep all of its data consistent.

Several circulation executives for metros told E&P last fall that the Audit Bureau of Circulations was requesting that newspapers begin turning in Nielsen metrics for the new Audience Fax, for consistency?s sake, the first time out. When asked about this, ABC says it accepts any metric for auditing.

With all this in mind, E&P?s Jennifer Saba sat down with Manish Bhatia, president of global services and U.S. sales for Nielsen Online, for an extensive chat about how panel-based measurement works and its advantages, as well as its shortcomings.

Bhatia is responsible for sales and client services at Nielsen, and more informally, spreads the gospel of panel-based metrics in general (and Nielsen in particular) while taking on its critics.

A few days before Bhatia spoke with us in his spacious offices at Nielsen?s Manhattan headquarters, he gave a presentation at the Internet Advertising Bureau (IAB) Measurement Leadership Forum before a packed and vocal house of ad buyers and publishers all wanting to know why the numbers don?t match, and asking about solutions. Bhatia addresses some of those questions below.

Take us through the nuts and bolts of these little-understood user panels.

The panel is the product that we are known for the most. Panels are recruited in two ways. One way is RDD [random digit dialing] or over the phone, and Nielsen media does that for us. They call people at random and say, 'Hey, we would like for you to be in our panel.' If you say yes, we send you a piece of tracking software that we would like for you to install on your computers, at home and at work. That software allows us to track your online behavior and your offline behavior and ship that information to us over the Internet.

Besides RDD, what's the other way in which you recruit?

The other way is online [via] other companies that recruit people on the Internet. So it's a combination of randomly recruiting a panel over the phone, and an online survey panel.

Is RDD more efficient, or is online more efficient? Why do you have many more online?

Recruiting online is more cost-effective but RDD gives us, we believe, a better quality panel.

Why is that?

Every member of the population should have a known probability of selection. Which means heavy users, light users, everyone should have equal representation. If you use a medium to recruit a panel, there's a built-in bias. So the people who surf the Internet more are more likely more engaged and more involved. It's like, Nielsen TV would never advertise on television, because that would give them a different skew of usage.

The reason we have two different panels is you absolutely want quality research, and at the same time you want depth. Limiting it to only an RDD panel of 30,000, let's say, doesn't let us go as deep as some clients would want us to go, so we supplement that with online recruitment. Our thing is that the RDD panel tells us exactly what the behavior should be, using that to adjust the online panel and take away the biases that might exist in the online recruitment panel.

Shouldn't the RDD be higher than 30,000>

No. Some would argue it could be lower than that.

The panel members, they are all over the U.S.?
Yes.

When you are tracking The Denver Post, you are tracking who is clicking on the Denver Post all over the country, not just in Denver?

Yes. Not just in Denver. We can slice by DMA and by state because we do have the addresses of the panelists.

What is the average time for panelists? How long do you encourage them to stay?

It varies. I think [in] the RDD panel, we replace roughly 25% of our panel every year. So the maximum someone would be in there is four years. Every month we are bringing new people so the panel is getting refreshed. The online panel varies. I don't have the number off the top of my head. People stay in that panel shorter than the RDD panel.

If I'm picked to be on a panel, what happens next?

Do you own a computer and do you want to do this? If you say yes, we ask you for a lot of demographic information up front, the usual demographics that advertisers are interested in. We put that in our database, then you get the software, and then we start tracking you. Nobody knows you're in a panel.

What about the persistent criticism that panel-based measurements do not really take into account the workplace as much as they should? How many of your panelists put the software on their work computer?

We want the software installed on every single computer that you own. For the RDD panel the work panel is about 3,000 [out of 30,000], for the online panel anywhere between 10,000 and 15,000 [of about 270,000]. I don't have the exact number but somewhere in that neighborhood.

Significantly less than the homebound.

It's a big myth out there that larger panels equal larger numbers. If you have a work panel that is 3,000 and another work panel that is 10,000, guess what? The estimates we get from both those panels are basically the same. ... What larger panels do is make the numbers more stable. One or two people coming and going can't have that much of an effect.

A smaller panel can cause variability in the estimates month to month. It doesn't mean the numbers are lower. You can have a very good panel, but if you are not representing the population correctly you could end up overestimating or underestimating audiences to a Web site.

But if people tend to read newspapers and other news sites at work, are you really capturing that usage?

Yes. Sample size does not determine the size of the estimate. If I could double my panel, if I do a good job recruiting the right people, you could see the estimates go down.

We have three or four different weighting techniques. We do it for both the home and work panel. The weighting and balancing techniques are well documented for all research. Panel size does not dictate the actual value of the estimates.

Now where you can have issues, let's take a site that is targeted toward females, whatever site, I'm expanding my panel but for whatever reason I'm not getting females in the panel. I have a systematic bias, and I'm excluding a population group. Then you can say the numbers are off. The issue all has to do with panel composition. Don't get me wrong, a large sample is better than a small sample, it gives you more depth and more stable estimates. But more important than that is the actual composition. Are you representing all the different users?

Why aren't more people putting their software on their work computer? Is it because companies are very sensitive to employees doing that?

Yes.

So do you see more data from participants who work at smaller companies?

Yes, but let me walk you through another thing. Let's say I have a work sample that is 100 people. I have 70 people from small companies and 30 people from large companies. I know from doing other research that actually the small company should be 35 and the large company should be 65. I know that right off the bat I'm getting twice as many small company people in my panel than I actually should. Right? Statistically, I just weight them down by a factor of 50%. I shrink it down. I take these 30 people and I say actually these are accounting for two people. So I weight them up and I weight them down so I have the right proportion. As long as I know what the actual composition should be.

Do you want to expand your panels to more than 300,000?

You know, it's the law of diminishing returns at some point. The No. 1 thing is the quality of the sample, and the No. 2 is the size of the panel because quality gives you a more accurate estimate and size gives you more depth. The third is the price point. You want to offer a service at a price point that matches the market.

What do you mean by that?

I could easily recruit a panel that is two million people, but that means the cost of the service goes up.

Do you compensate the panel members?

Yes, we do.

What is the compensation?

The people who are approved on the phone, upon installation will get a savings bond of $50 and every six months of participation they get a $50 savings bond that we send to them. And the work people are double that, I think the workplace is $100 and then another $100 every six months.

So it doesn't matter if you are contacted on the phone or online?

These are for people contacted via the phone. The online people go into an ongoing sweepstakes program, then we give them occasional nominal incentives for doing service.

With the work panel, do you encourage people to run it by the company?

Absolutely run it by the company, and we give them a phone number if the company has any questions they can call us. We set up a 1-800 tech support number.

Obviously the national newspapers benefit from national panels. But what about Denver, Dallas, and Seattle? They are not getting as much national traffic, but their DMA is very important. How do you make sure you are counting them correctly?

We do a panel nationally, but because we ask people to give us their addresses we can map them back to a DMA. Let's say The Seattle Times. We give them a national number and we can peel that back and give them a traffic number of people coming within their DMA.

What is your relationship with the NAA and ABC

We make our data available to them. I also understand [ABC] allows data coming in from other sources -- and that can be very confusing. You are mixing apples and oranges.

Why go with Nielsen? What is your selling point?

No. 1 is quality. We are the only company that recruits panels using RDD, which is used for radio, print, TV. And our products and services are in the process of being approved by the MRC [Media Rating Council], which also audits Nielsen TV, which also audits MRI [Mediamark Research Inc.] No. 2 on the media side, the connection we are making with Internet usage with other media. So we already integrated our Internet data with the MRI -- you can actually look at your print audience that comes from MRI, you can look at your Internet audience that comes from our data set, and then there is overlap between the two. We tell you what the overlap is.

One of the things that is becoming more important to newspapers is that it's all about credibility. So when we talk to The New York Times, a lot of people want to know how many people link in to the NYT. We track all of that. When you look at inbound links, more people link to you than the other guys. It gives you a little more dimension.

Speaking of credibility -- looking across the Nielsen numbers, comScore numbers, Omniture, are all just wildly different.

I can tell you that we have the most accurate estimates out there. Let's peel them off one by one. The reason I say that with some reasonable confidence is we are the only company that has panel-based tracking and server-based tracking with site census. We have implemented our own client Web sites with the server data, which is kind of like Omniture data, and we have two sources of information coming right into us and they can compare the numbers.

Where does the site-census data come in?

We tag the Web site with our own code. Every time a page of content from your Web site gets loaded or requested by anybody, we get a message back. We know they are accessing it once, twice, three times.

So right now what we have done working with a lot of clients, Nickelodeon, CNet and so on, we have sat down and instrumented their Web sites with our tags just like Omniture would do with their tags. We look at the panel data and we try to look at why the numbers are different.

So the server data, say, says you have 10 million people and the panel data says you've got 3 million people. The reason for the difference is that the server data counts different things ? cookies, essentially. The server data may include things like spider traffic. Auto refreshes are a kind of a new thing we picked up on, which was causing a lot of differences in the numbers. Like if I go to the Web site and every 10 seconds you are refreshing the homepage. I'm sitting here right now and maybe the homepage has been refreshed maybe like 30 times. Our panel will count only content that was successfully delivered at the request of the user.

Do you know the percentage of data from the cookies?

We actually do, it varies by Web site. Auto-refreshes and things like that ... it's different depending on how the site is structured. There is not a single number you can apply across the board. We have seen that cookie deletion can be as high as 35% to 50%.

You are doing this for CNet and some clients? Why not go out and integrate the site-census data into everything?


I'm glad you brought that up, because (laughs) we are doing that. We just launched a product called Video Census. It does exactly that. We instrument the Web site with our tags and track the video stream being served up and overlay that with the demographic information from our panel. We already have 11 of the top 20 video distribution companies tagged with us. The reason why we are not doing it on the main service is that we want to do it, but the whole solution depends upon the site cooperating with us and tagging. If they want to do it, we'll do it.

There is a newspaper saying, "We don't like your panel-based metrics," and your answer is, "OK, we can help you out here, but you have to allow us to do it?"

That's right.

What's the kickback? That seems like a no-brainer to me.
It's a no-brainer as long as you realize, guess what, your server numbers are going to go down. Your audience estimates are going to be lower than probably what you have under Omniture or any other site-centric server.

Adding the panel-based method is going to pull the number down?

It's going to go down. You have two options. One is that you can sit there and say, my panel data says "X" and my server data says "2X," I'm going to sell off of this. Or maybe you end up with a number that is between those two numbers, which is where you are probably going to end up -- but suddenly you don't have that "2X" number to sell.

What about the agencies and people out there buying online advertising -- are they subscribing to Nielsen?

We have over 100 agencies. Almost all of them. I think it's all the top ones.

Both Nielsen and comScore?

It varies by agency.

What about educating people on what the numbers and panels mean?

The industry needs to educate people who are actively involved in the industry, the IAB [Interactive Advertising Bureau], the publishers, the people who know need to educate the broader community as to why the numbers are different. We know why the numbers are different, so let's get beyond that. The second piece is, given that we have two sets of numbers that don't match, how do we move forward? I actually call it the problem of plenty, which is we do have different sources of information, but as long as you understand the pros and cons of each approach you can find a way to put them together.

It works across other media. This is not the first medium that had to deal with different types of numbers.

What would you do if you were the client?

If I were a client, I would start with the panel data. What the panel data would tell me is a profile of my audience, which I don't get from the server data. It tells me the demographic profile, it will give me competitive information which I don't have right now. It will tell me what my people do across the Internet, which I can't tell from my own data set.

The panel data may not have the depth, then I want to move to my server data. I can follow every single click and people click here, here, and here, I get a 90% follow from this page.

We see the benefit of each of the approaches. So when we go to clients we are not saying this data is wrong, or that data is right, we say for these kinds of questions here is the tool you need. We are not saying that tools like Omniture and site census are bad or inaccurate. We are saying they are better suited to answer different questions. They are not audience-measurement tools, because they are not counting audience. They are basically tracking click streams from computers.