Moving trackers online

Moving trackers online – money-driven madness or just common sense?

Research may have changed but consumers' desire to contribute remains
The past few years have seen significant changes in the New Zealand market research business, not the least of which has been a dramatic change in the way we collect information from consumers and even from businesses. I am old enough to remember the days of door to door interviewing (perhaps just as my grandmother used to talk of the days when salesmen brought their wares to her door…). How society has changed! Not only are people at home much less often now – what with working mothers, 7 day shopping and greater mobility, but we’ve become much more wary of our fellow man and hence less likely to allow the stranger with the clipboard to settle on our loungeroom couch for a Q&A session over a cup of tea.

Of course, all survey methods face challenges – none greater than the simple lack of time. People are busier than ever before – but that in itself hasn’t stopped them from being eager to voice their opinions, curious about new products and services, and just interested in finding out about stuff… It is this basic human curiosity that we rely on, and people’s need to participate in society, that provides us and our clients with a ready base of respondents willing to answer our questions.

As we’ve lost access to catch them at home, at a convenient time, we moved to calling them up – yes, often at dinner time, since that’s when we know they’re most likely to be there. The telephone made us more efficient at getting to the right people, and more effective at getting them to participate in the survey… but that too has become a challenge, as more people forego their landlines for mobile-only communications (with no ready directory of mobile numbers, and a sense that calling a stranger on their mobile is someone much more intrusive than ringing a home-line).

The internet - big change or just postal on steroids?
Enter the internet… and with it, the solution to our problems, or so it seemed. As more survey worked moved online, the challenge remained – to get a “good sample” – but the ways of meeting that challenge changed from random selection, to panel recruitment and management. In fact, long before the internet survey panel, panels of survey respondents existed in developed markets like the USA. Some 50 years ago, major brands were doing their research on postal panels, with respondents sent questionnaires each week to complete and return, with incentives for participation – no different really from today’s internet panels, with only the ‘send’ mechanism changing from a postage stamp to the click of a mouse.

New Zealand has been at the forefront of these changes, being well served by a number of professional panel managers, each focused on building the best panel they can to represent the New Zealand population. As internet penetration levels have soared – and broadband has overtaken dial-up connections – we are coming close to parity of representation between landline-connected people and internet connected people. ESOMAR figures for 2008 report New Zealand towards the top of the list of countries in terms of the proportion of research done online, though we still lag behind Australia in this regard.

Online research - it's about quality
While some suggest our rush to move surveys online has been financially driven, this is in fact not the full story. Research buyers are, in the main, sufficiently knowledgeable to know that the cheapest solution is not always the best – particularly when major business decisions are hinging on the insights that we gather. Increasingly, the evidence is gathering that the online survey – properly designed – gathers better quality, richer information from the consumer than any other methodology, and is much less prone to interviewer bias (one of the key contributors to socially acceptable response that we often see as people overclaim their use of aspirational products, the frequency of ‘good’ behaviours, their desire for ‘green’ products and their belief in values that constitute our societal norms).

By removing the interviewer, giving people time to reflect, and the opportunity to answer our questions in their own time, at their convenience, we get more honest and considered answers. We also get more “right” answers, particularly in markets where there are many brands, or where brand names are similar, or confusing. The ability to show packs, stocked supermarket shelves, brand logos and colours on screen simply gives us more reliable information.

The question remains: “what about the survey sample?” How can we believe that the internet-connected population is a reliable reflection of the whole population? Or should we try to believe that at all? At face value, the answer is no. But that “no” must be viewed in the context of the similar bias that we introduce when we only interview people with a landline phone number, when we only interview people who are at home when we call, who will allow the interviewer into their homes, who live in areas where door to door interviewing is practical… every sample, indeed every experiment we conduct, has inherent bias, and it is in understanding that bias that we make the information useful, rather than in pretending that it doesn’t exist.

In fact, most of the time, none of this matters. We get a brief, we make the case for the best methodology, and interpret the results in the knowledge of the various methodological biases that we’ve introduced – that’s the way research is done.

Migrating surveys online
However, when faced with an existing study, a study that has been conducted year in, year out, perhaps every week or every year for 10 or more years, the issue is magnified. Now we are faced with what will inevitably be a step-change in the numbers – trends which the client values simply because they are trends; trends that give us a touchstone against which to say things are moving up or down, better or worse than before. No matter how closely we match the characteristics of the survey sample – even if we force the same composition in terms of geography, age, gender, ethnicity – it is the inescapable fact that moving a survey from one method to another, even in our experience from one supplier to another using the same method – changes the numbers.

At Synovate New Zealand, we’ve faced this challenge frequently over the past few years, as more and more of our tracking clients have taken the leap to benefit from the better quality data and lower data collection costs that online methods offer… despite the pain of the step change in the numbers. For some, the change has been relatively minor, for others more significant… but all have been carefully managed through the change without any desire to return to their previous CATI survey methods. We’ll be inviting those clients to come together for a sharing workshop in a few months time – to share their views and experiences, and learn from each other – but meanwhile, some general observations about how things change, based on migrations across public opinion, services, FMCG and retail tracking studies…

Prompted awareness levels tend to rise, as brand names are shown and people are more likely to recognise smaller brands in particular.
Unprompted awareness levels may fall, as interviewers are not present to prompt repeatedly for “any others you know of” – in reality, though, we haven’t witnessed much change at all in this area.
Brand usage levels rise for smaller brands as people tend to recall more of the brands they use, that form a smaller part of their usage repertoire. As a consequence, repertoire sizes grow.
Image attribution levels are generally unchanged, though smaller brands appear to benefit a little.
Any brand which has a strong internet skew in its consumer franchise benefits – in some cases dramatically. This effect is one of the more challenging ones, and can be mitigated by prudent application of weighting factors if necessary – certainly something worth considering in advance (since it should be a known ‘problem’).
Interview duration falls – despite richer answers (more ticks in grids, more words in open-enders), it is simply faster to read the question to yourself than to listen to the interviewer reading it out, including the answer options. Of course, most of our clients have quickly filled up the ‘extra time’ with all those things they always wanted to ask but never had time for!

And what about the survey sample? The trade-off here is between speed and quality – and the prudent use of reminders to ensure that those initially sent the survey invitation are reminded several times (equivalent to callbacks) over the course of a few weeks that they’re still welcome into the survey. In ad hoc surveys online, we often sacrifice this element of quality control to ‘fill the quotas as fast as possible’, in extreme cases sending out thousands of invitations to get a few hundred interviews. This practice leads to the currently quoted, relatively low response rates for online surveys (surveys complete / invitations sent). However, good panel managers have techniques in place, often automated, to ensure that the number of invitations in a given week is managed carefully to ensure not only the right number of completed surveys, but also the right mix of respondents in terms of their demographic characteristics. The end result – a survey sample which delivers consistent respondent characteristics over time, allowing for true comparability of results.

Taking surveys online WORKS
So yes, migration online does work – and the benefits, in our experience, far outweigh the downsides. Most of all, it only ‘works’ if the researcher and the client work closely together to deal with the challenges as they arise, and to position the new data as simply that – a different view of the world (hopefully not too different!), that is more useful, but certainly no more right or wrong than the previous view.

Because after all, market research data is never the absolute truth – it’s simply a body of evidence that we gather, carefully and sensitively, to help our clients solve their problems on a more informed basis than if they had been working on gut-feel alone!