Applied Data Analysis

The cornerstone upon which all of these industries have been built centers around data quality. While “data quality” means different things to different people, the most commonly accepted definition involves a combination of data validity, reliability, utility in business, and alignment with known benchmarks. In 2008, the Advertising Research Foundation embarked on an initiative to understand the factors that affect data quality in online survey research. In the ensuing years, there have been two major initiatives, the first to understand the magnitude of the variation observed across multiple providers of the online research sample.

By “sample” we mean, in large part, consumers who have expressed a willingness to participate in a variety of research studies related to their own behaviors regarding products and services, or their general attitudes and beliefs about a wide range of social and or political issues. The more well-established marketing research firms that have their own consumer panels have carefully developed methodologies for recruiting, incentivizing and maintaining large pools of respondents who are engaged in the research process. The ARF’s charge in 2008 was to determine the extent of the variability across a large group of marketing research firms well known in the industry.

Surveys & Forecasts LLC was chosen after an extensive review process to be the primary analytic partner in this significant effort. This was not a small undertaking, involving 17 marketing research firms and over 75,000 interviews. The study design involved a pre-post design, in which the same study was executed one month apart to determine test-retest reliability. In addition, there were two versions of the study, one 15 minute version, and a 30-minute version. This was conducted in order to determine fatigue affects and positional effects for various modules of the study.

While the detailed findings of this research are too voluminous to report here and are available on the ARF website in more detail, the primary findings of this initial 2008 study were:

• Double opt-in consumer panels from the major marketing research firms do, in fact, vary in the level of response for key evaluative and diagnostic measures.
• Responses to stimuli, such as new product concepts, also produce significant variation across marketing research companies.
• There is a high level of correlation between a respondent’s self-reported survey enjoyment and levels of engagement in the survey itself. In particular, topics that are interesting, engaging, and tap into the respondents’ innate curiosity produce an attention span is significantly longer than studies that do not.
• Monetary incentives, such as cash equivalents, promotional points, lottery tickets, and the like, while useful as a “thank you” and to demonstrate appreciation for the respondents’ time are not the primary motivator for participation in online research. Rather, respondents express a desire to shape the marketing process, to have their opinions heard, and to participate in a larger community of individuals who are providing feedback to companies.
• The more recent 2013 initiative known as Foundations of Quality 2.0 was focused on the effects of sampling on data quality. This research has also produced a significant number of academic articles on the effects of quotas, sampling plans, question construction, digital fingerprinting, methodologies, and alternative ways of estimating self-reported data that have created a significant body of knowledge available for researchers worldwide. This latter data set continues to be mind for research insights and has produced significant new knowledge in the research community.

Surveys & Forecasts, LLC was privileged to have been involved in both foundations of quality 1.0 and foundations of quality 2.0 as the primary analyst. We look forward to continuing our productive relationship with the advertising research foundation as it embarks on additional initiatives in 2016 and beyond.