Can You Count on Web Metrics? Debate Continues

By: Jennifer Saba Trying to keep track of the twists and turns of Web measurement ? how various metric and Web analytic companies count unique users, time spent, and page views ? is often confusing, frustrating, and for one major figure in the Interactive agency world, an "embarrassment." That's how David Cohen, executive vice president and U.S. director of digital communications at Universal McCann, summed up his experience during a panel at the E&P/Mediaweek Interactive Media Conference in Las Vegas in May. Going back to clients like Kohl's, Intel, Sony, and L'Oreal and explaining why it's so hard to get an accurate count of say, a site's unique users, he asserted, only slows down the flow of advertising dollars to the Internet.

That's a problem for the entire online community, including newspapers.

Data from the likes of Nielsen Online, comScore, and even internal server numbers are often used to show a Web site's level of engagement. The problem is, when comparing the figures from those different sources for one site, the numbers swing widely. So what's really accurate?

There are several methods currently employed to measure Web data. "Panel-based" measurement firms such as comScore and Nielsen recruit (and even compensate) your average Web surfer to track online habits and then extrapolate numbers. The selling point is that these companies have insight into psychographic and demographic data. Many advertisers, and more than 100 agencies, subscribe to Nielsen. ComScore declined to discuss its clients, including the number of newspaper customers.

Web analytic software from companies such as Omniture, in contrast, measure a client's traffic directly from that client's server, but Omniture does not release that data. If an advertiser is interested in a site's internal numbers, it has to contact that site directly. Then there are firms like Quantcast and ? as of late June ? Google, which use a combination of panel and server analytics to collect data.

Even comparing data from different panel-based firms yields different results. Numbers compiled for one Web site from comScore and Nielsen, for example, vary because the panel sizes and the methodologies of each company differ.

The issues surrounding panel- and server-based metrics have been covered extensively in these pages, and there's still no consensus on which method, if any, should be considered the standard. Representatives from panel-based firms and analytic companies argue that they are not in competition, and that their respective data show different things.

Regardless, the figures from panels and Web analytic software are used to show traffic and other trending patterns. The Audit Bureau of Circulations said that of the 197 participating newspapers in the March 2008 Audience FAX report, 127 submitted Omniture numbers for their Web sites. Of that number, 71 of the participating papers fall in Tier One markets (the roughly top 80 markets measured by Scarborough). Eight papers in Tier One use comScore data, while 27 put forward figures from Nielsen.

The Interactive Advertising Bureau (IAB), the online industry organization keeping tabs on all forms of Web measurement, has not determined which method should be the benchmark. In May 2007, the IAB strongly suggested that the two dominant panel-based measurement firms, comScore and Nielsen, seek out accreditation with the Media Rating Council (MRC). The MRC is an "independent body charted by the U.S. Congress to steward media measurement," according to the IAB. Both comScore and Nielsen (owned by E&P's parent, The Nielsen Co.) got on board and are currently undergoing audits.

In the meantime, it helps to understand the underpinnings of how each of these methods works.

In the January 2008 issue of E&P, Manish Bhatia, Nielsen Online's president of global services, explained at length how Nielsen collects its data. The company recruits panels in two ways: online and through the phone via a method called random digital dial (RDD). Nielsen's total U.S. panel is approximately 300,000 people, with about 30,000 gathered using RDD.

Nielsen encourages panelists ? who receive incentives twice a year ? to install its software on all their computers, including those at work; some 18,000 panelists are tracked at their workplace. Nielsen then weights the results to determine the number of visitors to a Web site. If small-business Web surfers are under-represented, their numbers might be weighted based on what Nielsen knows about small-business Web surfers at large.

The company says it can do this extrapolation down to the DMA (designated market area), as long as it feels it has enough panelists to get an accurate survey. Nielsen tracks usage via servers as well, but the Web site has to allow Nielsen to place tags on its pages.

In that January interview, Bhatia tackled the issue of big swings in data from one metric to the next: "The industry needs to educate people who are actively involved in the industry, the IAB, the publishers, the people who know need to educate the broader community as to why the numbers are different. We know why the numbers are different. So let's get beyond that.

"The second piece is, given that we have two sets of numbers that don't match, how do we move forward? I actually call it the problem of plenty, which is ? we do have different sources of information, but as long as you understand the pros and cons of each you can find a way to put them together. It works across other media. This is not the first medium that had to deal with different types of numbers."

For the current probe, E&P picked two other Web measurement firms ? Omniture and comScore ? and set out to understand how they go about counting traffic and their convictions on using Web metrics.

ComScore
ComScore measures Web usage in a way similar to Nielsen's. The Reston, Va.-based company tracks online figures through a panel of people at work, at home, and in colleges and universities to determine Web activity and demographic information.

The company recruits its panelists through RDD and online surveys. Approximately 5,000 people are collected by phone, while the rest of the roughly 150,000 U.S.-based participants are gathered online for a total "media" panel of about 155,000, according to President/CEO Magid Abraham.

The panel usually turns over in a year's time and comScore offers various incentives such as free gigabytes of online storage (or if you are recruited via the phone, $60 to $100 a year in cash) in order to entice people and keep them around. "The first thing we always appeal to is a chance to make your voice heard," says Abraham. "It is something that may sound like a clich?. There are a lot of people who like to have their opinion heard."

Once someone agrees to be on a panel, they are asked by comScore to install software on the computers they use at home and at work. Abraham says somewhere in the range of 10,000 to 20,000 people install comScore's software on their work computers, running the gamut from small to very large companies.

ComScore combines the data collected from its panels with broader census information and also conducts a survey that tries to capture the composition and demographics of Internet users overall, for weighting purposes.

The fact that far fewer people install comScore's software on their work computers doesn't matter, Abraham says: "It's a question of basically weighting the sample. Here and there, there might be under-representation and maybe over-representation, but on average there is no reason why we would undercount work usage."

There are several benefits to comScore and, by extension, panel-based data (also used by Nielsen) over rival methods, adds Abraham, starting with the fact that his company's figures are neutral: "First and foremost, it needs to be third-party independent." Second, he says, Web analytic software tends to over-count uniques since it's based on measuring cookies: "If you are counting the number of unique cookies versus actual people, the number is pretty high." Since many people delete cookies, a person can be counted multiple times.

The other thing going for panel-based metrics, Abraham says, is that an advertiser can compare different Web sites. If, for example, a packaged goods advertiser wanted to target men interested in sports, it would look at the figures across several sports sites. Advertisers, he says, don't have the time or the energy to go to each individual site to get their internal data and then make sure it's correct: "It's just impractical."

Fourth, "Cookies very rarely have demographics. People are interested in demographics," Abraham says. "I can go on and on."

Abraham explains that it is not necessary to count every single person coming to a Web site in order to enhance its value. He is passionate on the subject of methodologies ? that somehow one form of metrics is vastly superior to another when measuring page views and uniques. That thinking, he says, "is wrong with 100% certainty."

However, he feels strongly that non-panel-based methods of counting do not provide accurate information on a site's real traffic. "The site data is wrong," he says. "It's fraught with problems if you want to measure reach. It's useless for media planning purposes. People like to use it for bragging rights. [Some] complain because the numbers are higher, but they are inflated because of a variety of reasons including cookie deletion. We will put up our data against anybody's site-server data any day, any hour, any minute, anywhere."

To Abraham the focus should be not on which method is better, but rather on how to improve the current methods. "I don't think that anybody who really understands the issue or takes an hour to think about it would actually continue in this debate," he contends. "Now, you can always debate how you can improve the panel metric. That is a much better debate, because nobody is perfect. We are not.

"But from the standpoint of basically deciding, could a census-based solution be used for the industry? The answer is no. It just doesn't work. It has never worked. It's a false illusion, and people should get over it."

Omniture
Omniture goes about the business of measurement quite differently.

Based in Orem, Utah, Omniture provides hosted software that measures Web site activity. Domestically, more than 200 newspapers use Omniture.

Unlike panel-based metrics, Omniture's software monitors a site's activity directly from that site's server through page tags, or "beacons." The idea is that every person coming to the site gets counted. Each Web page (or individual feature within a page, such as a "play" button on a video player) is tagged with a small segment of java script code that alerts Omniture when someone accesses a part of the site.

"It basically turns on the equivalent of a little lighthouse," explains Christopher Parkin, Omniture's senior director/Genesis Solutions. "That java script responds to the stimulus it receives and it basically sends an impulse back to Omniture and says, 'Hey, this thing you wanted me to measure or let you know happened ? just happened.'"

Omniture provides the beacons, which are individually configured to meet customers' needs. "They take what we send them and they copy it and paste it directly within the context of their Web pages," says Parkin. Depending on the customer and the site, set-up can take anywhere from a couple of hours, to ? if it's a sophisticated implementation with lots of streaming media ? a month.

"Take, for example, the Gannett properties," Parkin offers. "We would define for them the base information that they want to track across all of their different respective newspaper properties and then apply minor configuration changes to each of those addresses to the unique desires of each one of their newspapers."

As the beacons fire off information back to the home base of Omniture servers, the company produces reports that are available in real time.

Parkin explains that Omniture's media customers access the information multiple times a day to get a sense of what areas of the page (or stories) draw the greatest number of views, the length of each visit, and the number of ads being viewed: "Based on that information, often times they are able to use it to completely re-orchestrate featured news stories [and] locations of different news stories, based upon how they see people interacting with the page."

Omniture data can also help track trending over time. As a hypothetical example, a newspaper site begins getting heavy traffic around 6 a.m., which starts off the homepage but then moves quickly to the business section before shifting to sports. Then around 10 a.m., the traffic from the homepage starts moving over to the lifestyle and home and garden sections. "We help the newspapers and the media companies understand content consumption and how that maps directly to their segments, so they can tailor their content much more effectively," Parkin says.

One of the drawbacks of Omniture's data is that beacons don't know anything about the person behind the signal; there's no demographic and psychographic information, say critics, who also maintain that Omniture data overcounts the number of visitors since reliable data is based on whether someone's browser allows cookies. If someone regularly deletes their cookies, as many people do, it can skew the unique user data. One person could get counted several times.

Parkin counters by saying that not all cookies are considered equal. There are two types of cookies: first-party and third-party. First-party cookies are assigned by the Web site itself, in this example, Newspaper.com. A third-party cookie is embedded by a "third party" on a site. So Advertising Inc. might embed a cookie in Newspaper.com code.

Omniture doesn't encourage third-party cookies ? and Parkin claims that most people who delete cookies delete those of the third-party variety (a user can specify the level of cookie deletion through a Web browser's security filter). "Typically it's third-party cookies that have the majority of blocks and the majority of deletes," he says. "First-party cookies only see very low single-digit blockage or deletion. It doesn't impact the [Web site's] ability to provide their advertiser with very accurate readings and forecasts."

As for the debate as to which type of metric is better ? panel or Web analytics ? Parkin thinks the media has driven a wedge between methods. "It has been important to note there is a very, very high degree of difference between how we do our measurement and the purposes of our respective technologies," he says.

"I would say that Nielsen and comScore give good directional insight based upon socioeconomic demographic information," Parkin adds. "They provide a relatively rich information set around a definition of segments. The panel does a decent job of keeping track of people who visit particular sites so it gives Nielsen and comScore the ability to provide advertisers with some decent demographic information.

"Where things fall apart," Parkin continues, "is that they don't provide the deep insight into everything that is happening on the Web site by everyone who visits. It's only a highly extrapolated subset of people who visit the site."

According to Nielsen's Bhatia, the data from the two different types of Web counting should be used for different reasons: "We are not saying that tools like Omniture and site census are bad or inaccurate. We are saying they are better suited to answer different questions. They are not audience-measurement tools, because they are not counting audience. They are basically tracking click streams from companies."