Identifying Superforecasters in Online Market Research via Advertisement Testing Surveys

Abstract This research is inspired by the result of the works of Professor Tetlock on prediction science in the geopolitical and economics domains. He suggests that some non-experts are better than experts in predicting the future. This research attempts to identify if a group of individuals with high prediction skill exists in the general public by testing on ad testing surveys.

Modern businesses spend billions of dollars on branding and advertising of their products. Ad testing is commonly used as a tool to gauge the success and effectiveness of such campaigns. A problem faced by ad testing surveys is that the main campaign has to be kept on hold until the survey data is collected. Usually, larger the sample size the longer the delay. If a smaller group of forecasters are able to predict the opinion of a larger sample, the delay faced in ad testing surveys could be minimized.

Data from a prediction survey collected from 659 subjects living in the UK who predicted the best ads from set of 16 ad-pairs, were analyzed in this research. The analysis found that few individuals were able to predict more successfully and with greater confidence than others. Nonetheless, more research in the same domain with greater vigor is needed to fortify the claim.

Index Terms — Advertising testing, Forecasting, Gamification of online surveys, Market Research

This work was completed in August 2016

I. INTRODUCTION

ONLINE market research is a collective term for several types of research such as social media research, web traffic measurement, market measurement, advertising effectiveness research, media research, new product analysis, opinion research and many others [1]. Market research firms are constantly looking at improving the quality of their services. Some of the major areas focused are improving the quality of the data collected, reducing the time taken to complete a project and reducing the overall costs incurred to complete a project.

In terms of improving the quality of the data collected, an innovative concept labelled gamification by market research experts was explored. The concept, though it has a broad and ambiguous definition, is the process of incorporating game components like role-playing, leaderboards, mini- challenges, instant gratification onto a traditional survey. The end result for the survey taker would be more like playing a game than answering a survey. Researchers have found that gamified surveys increase respondent engagement and as a result improve the quality of the data collected.

This study is inspired by Professor Tetlock’s [2] findings on the prediction in the domains of geo politics and economics. He claims that some non-experts are better than experts at predicting future events. Integrating the prediction model for online market research is an area less explored by the research community. This exploratory study attempts to find if Tetlock’s claim holds true in market research domains. In order to restrict the scope to a manageable level this research will focus primarily on ad-testing.

Traditional advertising testing works by sending out a survey to a subset of the target market of the advertisement before the actual campaign is rolled out. The time taken waiting for the whole sample to respond and to derive findings of the advertising testing survey is largely dependent upon the sample it was sent out. Thus it directly impacts the date the actual campaign could be rolled out.

If a smaller set of individuals are identified to consistently predict the success of an ad better than others, the time taken to complete the advertising testing survey could be brought down substantially.

It was explored if there were superforecasters in the general public who could consistently identify the better ad over another in a sensibly gamified advertising testing survey. If results are found to be affirmative, market researchers could assume that the opinion of a handful of superforecasters to hold true against the opinion of a larger sample to whom a research survey is sent out to. The reduction in the sample size will bring a considerable saving in terms of time and cost to administer a market research project.

A. Study area

The primary read of this study was centered around 2 important reads:

Superforecasting: The art and science of prediction by P. Tetlock [1] and

Gamification - Game On!: A look at how gaming techniques can transform your online research by J. Puleston [2]. The other related research papers were aggregated over time through online resources like Google Scholar, ResearchGate.com and other online research paper repositories.

An interactive online survey was designed and developed using HTML / JS technologies. The collected data exported as SPSS .sav file was transformed with Python / SPSS code to provide ranking to forecasters based on the responses given in the survey.

Though not part of the project, an additional web portal is also under development to automate the creation of prediction questions, data collection and analysis.

B. Research Objective

The objective of the research is to determine and rank the best forecasters by analyzing data collected from an advertising testing survey, whose respondents were asked to predict “Which advertisement do you think others like best?” from a set of real-life ad-pairs.

Hypothesis 01: There are some individuals who are able to predict the better advertisement over another

Hypothesis 02: Predictions made by the top forecasters, are as accurate as the opinions of individuals in the sample

II. LITERATURE REVIEW

A. Advertising Testing

Advertising testing is an important step in an effective advertising campaign. The industry breaks down the testing into several stages, pre-testing : carried out before the advertising campaign initiates, testing : carried out while the advertising campaign is on-going, post-testing : carried out after an advertising campaign has ended [4]. At each stage, the effectiveness of the advertising campaign is measured based on advertiser requirement. Millward Brown's analysis finds that consistent pre-testing improves a brand's ad effectiveness by at least 20 percent compared to brands that do not test [5].

Of all the stages, advertising pre-testing takes the most important role. There are many advantages highlighted in carrying out pre-testing. The cost of developing and deploying advertising is rising, and the demand for strong return on investment grows ever greater. Advertisers need to know that investment in new creative will pay off with increased sales. Pre-testing helps to minimize the risk of damaging a brand [6]. The positive perception of a brand could be irreparably damaged if an advertising campaign fails. Having understanding on how an advertising campaign is or not is going to work also allows to enhance the effectiveness of the campaign. Pre-testing could potentially help save money, if the ROI of adverting campaign is found to be not attractive, the money originally intended to be spent on the campaign could be siphoned in other avenues.

Though advertising testing accounts to only 2% of the global market research spend, the impact of not having it is highly significant. Global market research is a rapidly growing industry which drives business innovations and breakthroughs. ESOMAR global market research industry report 2015 estimated the industry value at US$ 43 billon, which is an increase by US$ 3 billion over last year [7].

B. Innovation in online surveys

Market research, which includes advertising testing, works by collecting data from a subset of a population (sample) and generalizing the findings to apply on the population. The growth in internet and mobile technologies have rekindled the industry by replacing traditional methods with online techniques. Online and automated research accounts to over 55% of the total money spent on the industry [3].

The market research industry has traditionally relied upon Gallup [4] based sampling, opinion based data collection and standard quantitative tools like Gaussian curves for collecting and analyzing data, The burst in internet and mobile technologies like social media and big data have made researchers explore non-standard methods like prediction based data collection and Bayesian analytics for analysis [5].

C. Preference for online surveys

Most traditional surveys are carried out use pen-and-paper style surveys, face-to-face interviews and telephone interviews for data collection. Though each method has unique strengths and weaknesses [6], most research data these days are gathered online through the internet. Online surveys are preferred do to several factors [7]. Online survey projects cost very much less and turn out results faster than any other traditional methods [8].

It is easier to reach a very large sample through the internet, though less than 50% of the world population has access to the internet [9], global penetration is increasing rapidly due to advancement in smartphone technology, adoption of social media and through recent projects by Google [10] and Facebook to connect remote areas to the internet.

Due to the dynamic nature of online surveys, they provide leverage for high variety and flexibility. Surveys could be conducted in several formats [11] via email, website, widgets, apps etc. The surveys could also be tailored based on customer demographics, language and experience. Randomization and rotation of question options help to provide complex displays of the same survey to the respondents. Online surveys can require the respondent to answer questions in the order intended by the study designer, as well as prohibit the respondent from looking ahead to later questions. A graphical progress indicator can be shown to help respondents know how far they are into the survey.

D. Prediction Markets

One of the prediction based data collection non-standard approaches that have taken off recently are prediction markets. Prediction markets are work just like real world betting markets; the difference is that instead betting on company stocks, the betting is done on prediction outcomes. Several studies on prediction markets reveal that the results are as accurate as traditional methods [5] [12] [13] [14].

The major advantage of the predictive marketing approach is the need for a smaller sample size, which in turn reduces the overall time taken for data collection. In addition, they also help to separate good ideas over average ones with far greater differentiation than conventional methods and help to identify polarising ideas with low average scores that might have breakthrough potential [15]. Prediction markets have become highly successful that many products are already available as commercial solutions [16]. However, as a word of caution: all researchers agree that more research is needed and some even go on to question prediction market effectiveness arguing that they shift the respondent’s focus to maximizing market profits than providing an honest feedback [17].

E. Gamification

Gamification referred to as the “use of game mechanics in non- gaming contexts” [18]. Practically it can also be defined “the process of making activities more game-like” [19]. Gamification is being applied to many different processes and tasks to enhance the enjoyment of the people involved in doing them. Effects of gamification in online surveys is a much researched topic. Enjoyment is created by making tasks more engaging, fun and interesting to complete, which in turn increases people’s motivation to complete them [20].

[...]

REFERENCES

[1] P. E. Tetlock and D. Gardner, Superforecasting: The art and science of prediction. Signal, 2015.

[2] J. Puleston, “Online Research – Game On!: A look at how gaming techniques can transform yo ur online research,” Proc. Sixth ASC Int. Conf., no. September, pp. 20–50, 2011.

[3] ESOMAR, “Global Market Research 2014,” 2013.

[4] G. Gallup, “Putting public opinion to work,” Scribner’s, vol. 100, pp. 36–39, 1936.

[5] N. Silver, The Signal and the Noise. 2012.

[6] J. W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 2014.

[7] D. A. Dillman, J. D. Smyth, and L. M. Christian, “Internet, phone, mail, and mixed mode surve ys: The tailored design method (4th ed.).,” Internet, phone, mail, and mixed mode surveys: The tailored design method (4th ed.). 2014.

[8] G. Nicolaas, L. Calderwood, P. Lynn, and C. Roberts, “Web Surveys for the General Population: How, why and when?,” 2014.

[9] internetlivestats.com, “Internet Users.” [Online]. Available: http://www.internetlivestats.com/in

[10] S. Katikala, “GoogleTM Project Loo ternet-users/.n,” Rivier Acad. J., vol. 10, no. 2, pp. 3–8, 2014.

[11] M. Schonlau, R. D. Fricker, and M. N. Elliott, Conducting research surveys via e-mail and the

[12] M. Sigala, “Gamification for Crowdso web, vol. 9, no. 3. 2002. urcing Marketing Practices: Applications and Benefits in Tourism,” Adv. Crowdsourcing, no. Brabham 2012, pp. 1–17, 2015.

[13] J. W. Schlack, “Markets, The power of prediction,” 2013. [Online]. Available: http://www.quirks.com/articles/201

[14] S. Slade, “Why Polls Don’t Work,” 3/20130509.aspx. Reason.com. 2016.

[15] J. Kearon and C. Juicer, “Utilizing the wisdom of crowds to slay the sacred cow of scientific sampling,” pp. 13–14, 2007.

[16] Qmarkets, “Prediction Markets Qmarkets,” 2016. [Online]. Available: http://www.qmarkets.net/products/additional-products/prediction-markets/. [Accessed: 25-Jan-2016].

[17] R. Poynter, “The Signal and the Noise : Lessons for marketers, insight professionals,” no. January, pp. 1–10, 2014.

[18] S. Deterding, M. Sicart, L. Nacke, K. O’Hara, and D. Dixon, “Gamification. using game-design e lements in non-gaming contexts,” Proc. 2011 Annu. Conf. E xt. Abstr. Hum. factors Comput. Syst. - CHI EA ’11, p. 2425, 2011.

[19] K. Werbach, “(Re)defining gamification: A process approach,” in Lecture Notes in Computer Scien ce (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2014, vol. 8462 LNCS, pp. 266–272.

[20] J. Puleston and N. Schillewaert, “‘Online Research: Now & Next 2011’ (Warc), King’s Fund, London, 1 March 2011,” Int. J. Mark. Res., vol. 53, no. 4, p. 557, 2011.