The leaky cauldron of media measurement

It may be populated by bean counters and clipboard carriers, but market research could never be described as boring.

Why? Because barely a week passes without one of the audience measurement bodies experiencing a volley of abuse from one corner or another.

“It all needs tearing up and starting again,” says Sue Unerman, director of strategic solutions at MediaCom.

But is the criticism deserved? Is the murky world of media research in crisis? Whether it is The Wireless Group’s Kelvin MacKenzie ranting against Rajar or ZenithOptimedia research guru Frank Harrison bemoaning the shortfalls of the NRS, it is not hard to find strong views on our mediameasurement bodies.

If Unerman had her way, the current measurement systems for the various media – Barb for TV, Rajar for radio, ABC and NRS for newspaper and magazine sales and readership, Jicreg for the regional press and Postar for outdoor – would be torn up and replaced with an integrated system that monitors a consumer’s media habits on the whole, rather than across individual media.

It is a belief that is gathering support from many quarters. Harrison has been one of the most vocal on the subject.

“My thought is that trading currencies have changed very little in recent years, while the marketing industry has changed a lot,” he says.

“The increased use of non-traditional forms of media and the rise of multimedia campaigns has not been matched by an overhaul of how those campaigns are measured.

Media measurement is resolutely silo-based, while the rest of the industry is joining things together.

“As trading currencies, they have become less adequate for the needs we have.”

Jim Marshall, chairman of Starcom UK Group, is equally damning. “When we inhabit a world in which [David] Beckham is a medium in his own right, it is not enough to rely on headcount and silo measurement of media.”

One of the most heavily criticised measurement methods is Rajar’s diary system, which asks a sample of listeners to write down what they have listened to. “My problem is that I’ve never instinctively believed a lot of the radio research,” says Christine Walker of Walker Media. “I’ve always doubted it.”

Her doubts arise from the fact that panel members are asked to note which radio stations they listened to over a given week. The potential fallibility of such a system is currently a cause of great concern, which ranges from Walker not trusting what her gut tells her about its accuracy, to Kelvin MacKenzie suing Rajar for alleged loss of earnings at his speech-based station TalkSport.

McKenzie is leading a campaign to replace the diary method with an electronic form of radio measurement. Trials by GfK using wristwatches – which record what the wearer is hearing and check it against a database to identify the radio or TV station – have been taking place in recent months and their readings are, to many, providing a more accurate measurement for radio. Moreover, with its ability to measure TV channels too, claims are mounting that it is showing the way for- ward for an integrated measurement system.

GfK director Nick North says: “The standard is the single medium, but now we can look at the number of people who switch from TV to radio. It puts TV in the context of a competitor to radio.” Could technology be the answer to the failings of media research? Rajar has been pondering a shift toward electronic measurement for some time. It is testing electronic meters later this year and is launching an industry consultancy to gauge opinions on the future of radio measurement.

It has appointed former Y&R man Mark Cross to head up the consultancy process.

The argument from the radio establishment is that they have no problem with a move to electronic measurement – they just don’t want to rush it and make a mistake.

Rajar chief executive Sally de la Bedoyere says that careful consideration – not the wholesale ripping up of systems – is the way to tackle any shift in media measurement: “Different electronic meters do different things. Our current definition of listening could change if measured electronically. The industry has to be specific about its needs.”

But many in the industry are tired of what they see as Rajar’s stalling.

Walker remembers a meeting with the radio body 10 years ago, when it discussed its plans to look at moving into electronic measurement.

“I’m 50 years old now and nothing has happened,” she says. “Stop talking and get busy.”

It is a sentiment echoed across all quarters of the industry: media may be fast-moving but its measurement tools are often painfully slow.

Barb introduced its new panel in 2002, but it was almost another two years before it reached the promised quota of more than 5,000 households. During that time, it had seen off one chief executive and a chairman and annoyed large swathes of the industry that temporarily lost faith in the currency on which the TV industry depends.

Four years on, the sector is still waiting for parts of the promised new panel to be implemented.

In particular, Barb was to add an additional panel classification – allowing greater depth to the survey. Questions such as whether panel members had a mobile phone and which newspapers they read were to be added.

Andrew Canter, broadcast director at MPG, is one of the frustrated still waiting for its inclusion. He says: “The current research is great at giving a steer on the numbers of people present... but it doesn’t go deep enough,” he says. “When the new panel was introduced we were told the additional panel classification would be there.”

Bjarne Thelin, the new chief executive of Barb, said: “The additional panel classification is expected to be delivered this year,” and he adds that the intricacies of TV measurement mean that any change is a complicated process.

As he looks forward to the upcoming challenges within the TV arena – including the measurement of interactive television and PVR use. He cites cost as the main stumbling block to changes – or even overhauls – of current measurement systems.

Other people are not so kind. “I think vested interest is a factor,” says Walker, and that is an opinion found across the industry.

Measurement bodies are run by and for the industry: Barb, ABC and NRS are paid for by the main TV, newspaper and magazine companies, as well as the IPA. The smaller industry bodies – Rajar, Jicreg and Postar – are funded solely by the industry they represent, but the IPA has an involvement in decisionmaking.

The National Readership Survey, which measures readership levels of newspapers and magazines through a diary system, has also been the target of censure.

One of the main criticisms is that much of its data are based on “average issue readership”.

Like the ABC figures, current data does not allow the readership of a single edition to be pulled out; instead the average readership from Monday to Saturday, or on a Sunday, is listed. In the fast-paced media world, it is just not good enough for many.

“I want to know what’s happening today”, says Walker.

But, in a declining industry, many fear that newspaper bosses themselves are reluctant to break down the data further, as it would detail how far some readerships levels have fallen.

Harrison says: “NRS has been held back by the mixed interests of the publishers that fund it.”

The Audit Bureau of Circulations, which measures circulation of newspapers and magazines, has been holding discussions on whether to shift its own data to break out Monday to Friday sales from that of Saturday.

It has recently included e-publications in its data, but Martyn Gates, director of newspapers and consumer magazines at ABC, says that changes need consensus across the industry. “We’re always changing in response to the industry but ABC rules are agreed by the industry,” he says.

After many rounds of debate, a change in the ABC rules surrounding bulks are currently being rolled out. The detail of where bulk copies of newspapers and magazines are placed will now be given on the ABC certificate. Therefore, a planner or buyer can make a judgement on how good or bad a bulk copy is, depending on whether it is given out on, say, Virgin Atlantic business-class flights or dumped in a dodgy hotel foyer.

It may have taken a long time for such a seemingly simple process to be enacted but there is a lot at stake. For while price and vested interest may play a part in maintaining our measurement systems, the key factor in keeping the status quo is consistency. These measurement bodies may be flawed but they are the trading models on which entire industries are based.

Even small tweaks to their reporting have a destabilising effect. During the teething process of Barb’s new contract, £100m was wiped off Carlton’s share price during a single day of trading. Rather than moving too slowly, Lyn Robinson, research director at the IPA, says: “The roll-out of Barb’s new panel was an example of a body trying to move too quickly.

If you upset the media currency the market faces the consequences.”

The effects of ripping up the entire system, then, would be devastation. The softly-softly approach may be frustrating, but in the majority of cases it is the only way.

Piers Taylor, board planning director at Vizeum, cannot envisage a day when a single source multimedia currency exists; instead he believes the industry should work at achieving greater consistency across the board.

“Our research systems are probably as good as any in the world but the ‘opportunity to see’ in press is different to the ‘opportunity to see’ in TV,” he says.

Taylor is calling for a shift toward “probability” rather than “opportunity” as the standard.

He cites the poster measurement system, Postar, as a good example of a research system working well. It faced similar problems to Rajar seven years ago, when the previous system, Oscar, was accused of not accurately reflecting audience reality. Its research worked on the basis that, to count as a viewer, you had to be within the radius of a poster. Under Oscar, coverage and frequency levels were 90%, but were widely held to not be an accurate picture of the audience.

The system was overhauled and is now among the most admired in the industry.

“Postar has a more scientific view of whether a person is likely to look at it,” Taylor says.

The introduction of the new system – including the measurement of eye movement in the vicinity of posters – meant that cover and frequency fell across the industry and outdoor media appeared less effective. But there was consensus that it was offering a more realistic reading and it had no effect on revenue within the industry.

Opponents of Rajar could heed the lessons of Helen Tridgell, managing director of Postar, who says: “Gently, over time we have moved things on as the technology became available. We haven’t used technology because it is there, but because it is right for us.”

As outdoor grows and evolves, Postar itself moves to keep up with the change. Later this year, it is incorporating rail and underground usage alongside its roadside data and will then move onto buses, taxis and retail.

It has led the outdoor industry away from opportunity and toward greater probability in its measurement and many believe both Rajar and NRS could learn lessons from it – as they both rely too heavily still on opportunity.

But the outdoor industry then (and now) is a comparatively small market which would be hugely affected by any major shifts in its trading currency.

“To do anything to threaten £4bn-worth of business would be looked at sceptically,” Taylor says. “Agencies are relaxed. We have our own skills. As long as it’s used as a trading benchmark there is no problem.”

The murky depths of media research

The various systems relied upon to measure media audiences have more holes in them than a string vest.

Virtually all of the systems used to cover the various types of media exposure have come under fire for having glaring gaps in their accuracy.

Rajar’s problems have been in the news more than most of late. The diary system has been castigated as antiquated but its problems don’t end there.

The organisation has also found it about as hard to recruit 18 to 25-year-old respondents as it is to recruit police officers in Baghdad – leaving stations such as Virgin claiming its key audience is massively underrepresented.

Sample size is still a huge problem for Barb. Despite getting over its disastrous problems in raising the size of its panel, it is still not big enough to measure any channel attracting less than 10,000 viewers at a time, leaving a string of digital channels – and the number is growing fast– with the ridiculous situation of having “zero” ratings. Barb is also next to useless when it comes to measuring interactive audiences.

Many believe Barb’s problems pale into insignificance compared with those of Jicreg. It measures less than half of the titles in the UK but has, with admirable audacity, the cheek to put out readership figures for them all – deriving its figures on a decidedly dodgy multiplication of ABC circulation figures based upon roughly how big the town they operate in is and the type of paper in question.

Postar is regarded as the best outdoor audience measurement survey in the world, yet in still doesn’t measure 20% of the total number of posters, doesn’t measure outdoor ads on public transport and misses out the growing ambient sector.

The reality is that, while the UK may be well ahead of the rest of the globe when it comes to the burning issue of audience measurement, it is far, far from perfect and more pressure is likely to be put on the various organisations involved to get their act together in the months to come.

Return on investment is taking over in media and guesswork in measuring effectiveness is becoming less acceptable.

For while no one denies that each measurement system has its own problems the complexities of creating robust measurement is acknowledged.

Progress has lagged because, without cutting-edge technology that in some cases infringes on personal liberty, measurement is difficult.

“How can you actually check if someone’s looking at an ad?” asks Taylor.

Barb has a re-contact option which makes sure that people are in the room when they say they are, but checking which pages of a newspaper a reader looks at is nigh-on impossible without placing chips in eyeballs.

But as media shifts, the need for new types of measurement is getting stronger and rival measurement systems are challenging the industry standards.

GfK has challenged Rajar to speed up the shift to electronic and Sky has introduced its own at-home panel to provide data on multichannel satellite TV homes. Harrison says that a merger between Barb and Rajar would create enough money to expand at-home panels to 30,000 homes.

Single-source media measurement may seem like a fantasy to some, but technology is quickly making it a reality.

Nick North, a director at GfK, says: “The way technology changes the media landscape demands changes in research technologies. If everyone starts watching TV on their mobiles, then the core of any TV measurement system would have to be the mobile.”

He believes that set-top box measurement will be core to tracking TV audiences, but as mass audiences continue to disperse from TV, so advertisers will look to multi-media research solutions for their campaigns. “The focus will be increasingly on the individual... what they are watching and where they are travelling in a day,” says North.

GfK launches a new generation of its wristwatch measurement technology at the end of the summer, which Rajar will be testing as a radio-measurement tool. But its scope will go beyond radio.

The receiver inside the device will register a repondent’s proximity to a transmitter – which can be placed in a variety of places, from the door of a cinema to a poster. They can also be placed at various locations related to day-to-day life and will passively pick up surrounding broadcast media.

The IPA has responded to the clamour for changes to audience measurement with the introduction of its Touchpoints survey, which goes as far toward single-source measurement as any industry initiative.

Touchpoints aims to fill the gap between the individual media research bodies by fusing them into one and created a “hub” survey of 5,000 respondents to pull them together.

Robinson says: “The data will reflect the market as a whole and be able to be used as a multimedia planning tool which tracks performance across all media.”

It is still at planning stage but is expected to roll out in the first half of next year, and may take media measurement away from its silo mentality into a more holistic way of thinking about media consumption. Crucially, though, it goes even further, shifting the focus of media measurement onto the consumer rather than the medium.

“The single flaw in all forms of research is that it researches the medium rather than the consumer and how they use the medium,” says the group chairman of IPA Futures, Jim Marshall.

Until eyeball-tracking technology comes of age – and the industry has spent three years on a consultancy process before signing up – Touchpoints looks likely to provide the closest vision of the future of media measuremen.

Additional research: Ian Quinn

In defence of media research, from Christina Hartley

It underpins currencies used to trade £12bn annually in the UK. So buyers and sellers must be in accord and trust the quality of the research, whatever the medium.

This is achieved by involvement from both board and technical sides.

The realities of a multimedia world, where consumers are increasingly at ease moving across media platforms, presents us with challenges.

Measuring that crossmedia usage remains the holy grail. But quizzing a respondent on their full range of media habits is tough. It is time consuming for the respondent and costly for the media industry.

Agencies and media owners have invested heavily in developing their own multi-media planning tools – MediaDNA, VIPer and Roar, for example.

Software companies, such as Telmar and KMR-SPC, have their systems as well.

So is the demand for an industry standard tool a real one, when a bespoke product provides a USP and a revenue stream? Also, the industry needs to be clear on what it’s asking for: a planning tool across media, or a measurement of exposure? One methodology will not provide a universal fix.

Electronic surveying of media is not infallible – as the current Rajar debate highlights – and, crucially, its use does not extend beyond electronic media at this point.

These and other questions are generating the most vibrant and dynamic media research agenda in recent history.

And, interestingly, the call for papers for the forthcoming Media Research Group Conference yielded the highest response ever.

What conclusions do I draw from this? The more complex the questions, the more driven research thinkers become and the more established research becomes at the heart of good communications planning.

Christina Hartley is chairman of the Media Research Group and ad marketing director at IPC Media

Audit bodies

ABC: Audit Bureau of Circulations: circulation data for magazines and newspapers ABCe: Audit Bureau of Circulations Electronic: electronic media measurement

Barb: Broadcasters’ Audience Research Board: viewing data for television channels

Jicreg: Joint Industry Committee for Regional Press Research: readership data across local and regional newspapers

NRS: National Readership Survey: readership data across magazines and newspapers

Postar: Audience measurement for roadside panels

Rajar: Radio Joint Audience Research: audience measurement system for the radio industry

Internet: No standard as yet, but NOP and Nielsen NetRatings used