by Bob Walker | Oct 30, 2017 | Uncategorized
Data that’s really, really big
No one goes around saying, “Gee, look at how big that data is!” Well, maybe some people, but they’re weird. At present, there is no unified definition of ‘big data’. Various stakeholders have diverse or self-serving definitions. A major software company defines big data as “a process” in which we “apply computing power to massive and highly complex datasets”. This definition implies a need for computer software – gosh, how convenient.
A more thoughtful approach was recently taken by two British researchers. For data to be considered “big”, they argue, it must have two of the following three characteristics:
- Size: big data is massively large in volume.
- Complexity: big data has highly complex multi-dimensional data structures.
- Technology: big data requires advanced analysis and reporting tools.
OK, not bad. Note that in this definition there are no relatives or absolutes (i.e., that a dataset must be bigger than ‘x’). But we also know intuitively that the data housed at Amazon, Walmart, or Google probably meets these requirements. I like this approach, because it does not reflexively imply a large number of subjects. Big data could be miles deep, yet just one respondent wide (think, for example, about continuous measures of emotion or cognition). Biometric data, for example, easily fits.
Reg Baker (Market Strategies International) has said that ‘big data’ is a term that describes “datasets so large and complex that they cannot be processed or analyzed with conventional software systems”. Rather than size, complexity, or technology, he focused on sources:
- Transactions (sales, web traffic/search)
- Social media (likes, endorsements, comments, reviews)
- Internet of Things (IoT) (sensors or servo-mechanisms)
Perhaps so. If thinking only of sources, I would add biometric/observational data to this list. These data are inherently different: they are narrow and still complex. Observational data might include experiences, ethnography, weather/celestial data, or sources that involve continuously measured activity. Biometric data includes all manner of physiological (sensor) measurement that is then analyzed using highly sophisticated software. In biometric research, the number of observations (subjects) is often < 30, yet the number of data elements captured per subject is enormous.
So, when is data “big”?
A lay person would say that big data implies “massiveness”, so while not wrong, ‘big data’ is somewhat of a misnomer. We need to think of big data in a three-dimensional way. Big data requires “massiveness” in three areas:
- Data elements (i.e., variables)
- Observation units (i.e., subjects)
- Longitudinal units (i.e., time)
Big data typically has a longitudinal aspect (i.e., data collected continuously or over multiple time periods) with frequent updates (i.e., repeat purchase). Additionally, the tools needed to analyze big data (i.e., neural networks, SEM, time series) are significantly different tools than those used for less complex datasets (i.e., spreadsheets). Much like the word “fast”, the word “big” will evolve, too.
Better, cheaper, faster – or just bigger?
In the last 5-10 years we have seen a shift away from reliance on survey research data and analysis, towards a greater belief that ‘big data’ will tell us everything we need to know about trends, customers, and brands. This is reflected in the following data and analysis trends:
- From data/analytics that are scarce and costly, to ubiquitous and cheap. When data is everywhere, and basically free, we assume that there must be more we can do with it.
- From testing specific hypotheses, to relationships discovered by data mining. This is the million-monkey-typewriter hypothesis. Here are some amusing examples.
- From seeking feedback directly, to presuming needs from data patterns. This implies more weight on correlations and modeling than conversation.
- From a foundation of sampling theory and statistical testing, to the presumption of normality (and that all differences are meaningful given ‘big data’ status).
- From data gathered by design, to data “found” in other processes (so, for example, GPS data in a transaction record).
The above is not “wrong” per se; rather, it represents a shift away from critical thinking. ‘Big data’ is shrouded in hype and over-promise. Marketing management’s dreams of never ending insights from big data are just that. Dashboards, visualizations, and marketing mix models are alluring representations of ‘big data’ – some are beautiful and artistic. Yet, isn’t the goal to use ‘big data’ to drive profitable decision-making?
‘Big data’ and survey research – BFFs, like, forever
Survey research must share the bed with ‘big data’, though they will continue to fight over the sheets. Big data can free the survey researcher from having to spend time collecting merely descriptive data (for which human memory is notoriously foggy) and that might otherwise reside in transactional databases. This lets survey research do what it does best: gather opinions and reactions to stimuli.
Over time we will find out that ‘big data’ does a wonderful job of recording behavior, but does less well at predicting. In the near term, there will be a redeployment of resources away from primary and survey research. Some companies will rely on big data too heavily, and in avoiding direct discussions with customers, will suffer.
Opportunity will be there for companies who actively listen, rather than a purely modeled approach. Bridging the gap between self-reported, biometric, and observed behaviors is likely to become the next really “big” thing. Happy Halloween!!
by Bob Walker | Sep 24, 2017 | Uncategorized
Let’s review some standard sales forecast inputs…
When conducting a sales forecasting exercise, there are a number of fairly standard inputs that we have to feed into a forecasting model. In addition to the identification of a target population, and obviously its size, we also have to estimate initial trial, an initial repeat rate, a secondary repeat rate, anticipated purchase cycle, and an assumption about packs bought per buying occasion. We also have to make assumptions about the distribution and awareness build in year one (either entered directly, or estimated from a detailed media plan and separately modeled). And there are obviously other factors, such as sampling and couponing, which don’t need further explanation here.
These are all fairly standard inputs in all sales forecasting models currently available, including those available for “rent” by consultancies. But perhaps the most powerful, yet overlooked, variable in the entire equation is advertising impact. Great creative generates a multiplier effect that is significant – and enough to overcome many of the mathematical constraints inherent in other marketing mix variables.
A simple example will demonstrate this clearly. Let’s assume that we have a sales forecast for a new snack product. An estimated target population, based on consumers within the household, is presumed to be 35MM, and from a concept-product test we have estimated trial of 20%, and first-year repeat of 65% (i.e., about 2/3 will make one repeat purchase in the launch year). We’ll assume, for simplicity that the average number of packs per purchase occasion (for both trial and repeat) are an average of 1.0. Similarly, if we take a typical distribution and awareness curve based on an average marketing plan, we might assume, say, 50% distribution by the end of year one, and an average 40% brand awareness achieved in the marketing plan.
The base forecast
So, the simple math would be something like: (35MM x 50% x 40% x 20% = 1.75MM trial purchases) + (1.75MM x 65% = 1.14MM repeat purchases) for a total volume of around 2.9MM units. But that also assumes average creative (based on a copy test vs. ad norms). But what if we have an advertising agency that was able to create a great campaign and wonderful creative?
In the context of an average sales forecast, the difference between 40% and 50% awareness, while significant, is 10%. Given the asymptotic nature of advertising awareness, the marginal return of each additional advertising dollar flattens the curve out rapidly. The same is true for distribution, albeit with a more linear relationship between spending and added points of sale. A company’s ability to go from say 50% distribution to 70% distribution is still going to be significantly more costly, because the large distribution outlets have already been captured, and what remains is the difficult blocking and tackling of smaller regional players which are, by default, more expensive to gain.
But let’s get back to the impact of good creative on the same marketing plan. It is not unreasonable, nor is it uncommon, to get great creative from your agency partners for the same amount of money that it might cost you for mediocre, or even lousy, creative. Yet the multiplier effect of great creative is far greater than what might be achieved through nominal increases in awareness or distribution.
Great creative is the most powerful multiplier
Assuming that the product proposition is sound, a 40% improvement in ad impact would race through the entire sales modeling exercise like freight train. Let’s assume that we can get really superior creative from our wonderful agency friends, and are able to generate an ad impact score of 1.4 (where the norm would be 1.0). That takes our 2.9MM unit forecast in year one to 4.1MM – a far greater impact than a marginal 10% increase in awareness or distribution (which would only get us to around 3.2MM – 25% less). Yet we always think that both awareness, distribution, and use-up rates are the holy grail in a sales forecast.
Of course, the fundamental inputs of a sales forecast are truly important, but the ability to connect with the consumer through great creative can often turn a mediocre story into one that truly resonates in the marketplace and captures consumers’ imagination.
So, when assessing the performance of your creative, especially in the context of a new product launch, give your agencies room to run! They may just come back to you with the kind of creative that will produce the multiplier effect we just saw. And that can make the difference between a winning formula vs. playing for a tie.
More appreciation is needed for the pieces of a sales forecast that really matter: great creative, with a great product behind it, is almost an unstoppable winning formula.
by Bob Walker | Aug 30, 2017 | Uncategorized
Yes, Virginia, “marketing research” is different
With school back in session, my thoughts naturally turn to… etymology! No, not the study of bugs, the study of words and their meaning. Every so often, I find myself revisiting a question that seems to shadow those of us in the research industry: the distinction that people make (or do not make) between the words “market research” and “marketing research”. The terms are often used interchangeably and without much thought. I often see industry publications, web sites, and top tier consultancies doing this, which is disconcerting, because they mean different things.
In the most literal sense, “market research” implies research about markets — how big they are, how fast they are growing, who has more market share, and other descriptive information.
That is not to say that ‘market research’ does not deal with diagnostics, or with extremely large data sets, or with the incredible and ever-expanding set of tools available for targeting. This is especially true in the world of media measurement, sales analysis, and transaction-level analysis. These work products are largely based on passively collected or observational data; rather than attempting to understand behavior, attitudes, or drivers of brand choice through proactive investigation, we deduce with the data available.
“Marketing” research refers to the marketing process, and by definition, subsumes all market research activities. Marketing research looks forward, and asks why – why has something happened, why are markets changing, and where are consumers headed next? It is the active nature of research and its focus on the marketing process that distinguishes marketing research from virtually every other business discipline. Market research looks at the rear view mirror: at what has happened, which is presumably predictive of what will happen, because that is what happened last time. That is helpful in a general sense.
Marketing research activities are designed to answer key marketing questions so that a business decision can be made: for example, go or no go, expand or sell off, segment A or B, what message. It is also the relentless nature of this exploration that makes it so valuable to every organization. If a company does not have its eyes looking up and outward, towards the ever evolving sea of consumer change, it is destined to become a captive of its own narrow metrics, unable to look outside its own boundaries for opportunities that can breathe new life into the organization.
Innovation and marketing
Peter Drucker, esteemed management professor, said that organizations have just two functions: innovation and marketing. Marketing research is integral to both: in the identification of new business opportunities, in assessing the market, in configuring the product, and in marketing communications. ‘Market research’ measures and reports on performance but fails to address the most basic question of all, which is: why?
Perhaps the most rewarding aspect of “marketing research” is that it not only works alongside marketing at virtually every step, but in fact guides where marketing needs to go, because research is the closest to the customer and the information he or she provides. The very best organizations know this, welcoming an active, assertive marketing research function to the table as partners in brand management.
The confusion in terminology may also rest on the fact that it is difficult to explain to others what it is we actually do. I sometimes resort to telling people that I conduct customer satisfaction studies or analyze survey data, because these are a bit more tangible. But, in reality, they are simply activities and tasks. The tasks fail to convey the rich nature of uncovering consumer insights, guidance for marketing strategy, or communications refinement that marketing researchers perform. Using “insights” is equally vague, since we don’t “manage” insights, but we do use research to uncover them. So, perhaps we need a better, more descriptive term for what we do, although exactly what that is eludes me at the moment!
Manufacturer vs. supplier roles differ
Unlike consultants, corporate marketing researchers have additional burdens that external researchers do not, because they live with the brand 24/7 (not just the lifespan of an engagement or a project). They must make sure that any and all research studies are appropriate for each stage of the brand’s support plan or life cycle.
Corporate marketing researchers must also always act as the conscience of the brand, and resist persuasive yet inexperienced marketing managers who make can make reactionary decisions (OMG, our “likes” are declining!) that can significantly dilute a brand’s equity.
Every organization needs to have a strong, well-respected marketing research function (or work with thoughtful marketing research consultants) to support innovation and marketing. And, marketing researchers must have the courage of their convictions, and not be afraid to challenge old assumptions.
Check one only
So, which type of researcher are you? One who reports facts, figures, and delivers data, or one who understands brand objectives, collaboratively challenges assumptions, and designs the appropriate research to support key business decisions? If it’s the latter, I’d call you a marketing researcher.
by Bob Walker | Aug 11, 2017 | Uncategorized
Sort of like an Eskimo tracker
“Tracking” studies have been a staple of marketing research departments for decades, yet they still mean different things to different researchers and companies. Generally speaking, conventional thinking holds that tracking studies are surveys conducted on an ongoing or wave basis, and intended to gather marketing intelligence on brands, advertising, messaging, users, or usage. It is the repeated nature (the same questions over time) that distinguishes tracking from other types of research. Tracking can certainly include diagnostic measures, but such measures tend not to be extensive.
The primary emphasis of tracking is, or should be, on being an efficient barometer of brand performance. The focus is on key performance indicators (KPIs), which typically include unaided and aided brand and advertising awareness, brand purchasing, usage occasions, performance measures (attributes), and basic classification questions. If we were to further extend out from these basic measures, we move into the realm of strategic research, which tracking is most certainly not — although tracking can point to areas that need to be explored in greater depth. Knowing where that line is crossed is a judgment call, hence each client must make his or her own decision regarding the “sweet spot” of breadth and depth.
With a steady stream of multiple data sources, from customer service to social media, tracking research has fallen out of favor somewhat. While many marketing research departments continue to conduct large-scale trackers, their utility and actionability are increasingly called into question given the multitude of other seemingly substitutable sources. Some clients appear quite comfortable relying solely on sales data, and the output of search engines that scrape social sites and web commentary.
If tracking hasn’t proven to be useful, you’re doing it wrong
No one will argue that (syndicated) sales data or site transaction data isn’t critical, but these data sources are not very useful forward-looking predictors. Social listening, while improving, produces data that is event-specific and volatile, and relying on the spiky nature of this data forces marketing into making knee-jerk reactions, and also diverts attention and resources away from more serious strategic management decisions. Hence a question is: What types of feedback should be institutionalized in the decision-making process? This seems to leave a fairly wide-open playing field for more thoughtfully designed and executed feedback tools, which tracking research most certainly can be. The recent (June 2017) decision by P&G to cut $125MM+ in digital ad spending was most certainly informed by smart tracking. Tracking research can and does impact significant business decisions.
The most tired, predictable criticisms of tracking research are that it is expensive, slow and, as a result, not particularly actionable. Let’s knock these straw men down one at a time.
If conducting consumer satisfaction research, sample acquisition costs are near-zero (more on that later). Of course, costs are incurred if your tracking program requires external sample. Costs obviously increase if you work in narrowly defined categories, limited geographies, or where incidence is low and screening costs are high. However, in most widely penetrated categories, incidence is not a huge factor, and the added benefits of sample quality are strong counterarguments against erratic sources with little quality control. Reliable data is rarely free, so perhaps the better case is a value story, and not purely a rebuttal using costs alone. Spending $100K on a smart tracking program to assess the impact of a $20MM ad budget seems compelling to me.
Management has to listen
The slowness critique is also a red herring, in that most tracking programs conducted online are done so continuously, and can be aggregated in any number of increments to reflect category dynamism or rate of change. If you are dealing with faster moving categories, or need more extensive subgroup analysis, then more frequent reporting is implied. Aggregations at the weekly, monthly, or quarterly levels are typical; some experimentation is needed to choose the right frequency of reporting for your business.
Last, the actionability of tracking research is always a function of both design and execution. With (a) the correct sample frame, (b) questions clearly linked to business processes, and (c) smart reporting (including exception reports and alerts), tracking becomes your primary radar screen and course correction tool for the business. As you likely know, customer satisfaction research gives us the added advantage of including a key field, such as a customer or company ID, which in turn lets us link survey results to transaction history. Once connected, we can unpack the relationship between perceptions and behavior at the respondent level, have the ability to develop predictive models, and inform and monitor specific marketing actions. We’ve had much success in this area.
To many of you, especially more senior researchers, most of this is not particularly newsworthy. But, it does raise the question of why tracking is not more widely used, and mined more fully, and better embedded within organizations. There are probably many reasons, although the most obvious include:
- Not re-examining your tracking program on a frequent enough basis (e.g., quarterly) to make sure that category, brand, attribute, or diagnostic measures are up to date. Failing to keep tracking programs current lets them lapse into obscurity.
- Seeking input from multiple people or departments, to identify consensus areas that are most important for workflow, and to identify when changes are needed.
- Internally framing tracking research as a continuous improvement tool (Deming), and not just another form of sales data, which it not.
- Not developing control bands on key measures to help identify data points that fall outside normative ranges and require corrective action. This avoids marketing uncertainty over what is, or is not, meaningful.
- Missed opportunities to push reporting deeper into the organization using dashboards, visualizations, or exceptions/alerts to specific individuals or teams.
- Increasing added value by including modules on hot topics that can be swapped in or out, depending on current market conditions (while keeping survey length brief).
- Failing to connect the sample frame or key questions to specific business decisions.
- Failing to leverage the linkage between survey responses and transactional data.
- Taking tracking research too far. Most buying decisions are emotional, hence tracking research can only get us part of the way there. Other forms of research (qual only, hybrid quant-qual, strategic) are still needed to continue to learn about consumer behavior.
Let’s improve the measures
One last (loosely related) thought… given all of the customer satisfaction work being done today, one metric that we need to develop more fully is a prospect-to-conversion score. Tracking is a perfect vehicle for this, yet customer satisfaction studies rarely look outside the world of the brand itself. Much like NPS, creating a BPS (brand potential score) may prove to be a far more useful metric than knowing how satisfied current customers are. Over time, all customers defect or die. They must be replaced with new customers predisposed to the brand. Understanding this potential will be much more useful to the long-term health of the business. Tracking can help get you there.
by Bob Walker | Jun 27, 2017 | Uncategorized
An old trap
From time to time I find myself getting into trouble when designing research and thinking about incidence, sometimes referred to as prevalence, or rate of occurrence. The trouble I find myself in relates to the the net percent of respondents who will, ultimately, qualify for a study based on the screening criteria set by my client. This is true for both off-line as well as online studies (though the cost implications for off-line studies are often greater). For simple projects, in which I have general sample specifications, the process is straightforward and uncomplicated. Sample providers generally understand a request for a “general population sample”, typically resulting in a proportionately representative sample based on four census regions, ages 21 to 64, and proportionately balanced on race, ethnicity, income, and education. The problem I experience, however, is typically the result of special groups or “boosts” that can confound an overall incidence level – and significantly affect costs.
An example
Let’s say I have asked my sample provider for a “gen pop” sample, and they quote me $5 per complete assuming 90% incidence. That means for every 1,000 people I screen, and all things being equal, 900 will qualify. But say that my client has three small brands A, B, and C, each at lower incidence levels, and they would like a readable sample of 200 per brand. And, let’s say that these three brands have past year purchase incidence rates of 15%, 10%, and 5% respectively. Assuming a sample of 1,000, I would end up with 150, 100, and 50 respondents, respectively. It will cost me significantly more for these respondents then for my “gen pop” sample. At 15%, my cost per completed interview (CPI) might be $20, at 10% it might be $30, and at 5% it might be $50. Note that I am using dramatic changes in the CPI for example purposes.
So, what should I do in terms of giving sample specs to my providers? I don’t want to receive costs that are too low, and base my proposal on those costs, only to get the project and then have much higher out-of-pocket expenses than foreseen. By the same token, I don’t want to overestimate my sample costs, and risk losing the project because I have not calculated correctly and am noncompetitive.
We also have the added complexity of brand interaction, meaning that some of those respondents who are using the brand at 15% may also be using the brand at 5%. So here are the probable outcomes based on this scenario:
- General population (“gen pop”) sample of 1,000 respondents
- 150 A’s who will fall out of the gen pop sample
- 100 B’s who will fall out of the gen pop sample
- 50 C’s who will fall out of the gen pop sample
Assuming that they are normally distributed, we also need to consider interactions between the brand quota groups, for example:
- 15 B’s will fall from the A sample (10% x 150)
- 7-8 C’s will fall from the A sample (5% x 150)
- 5 C’s will fall from the B sample (5% x 100)
Cost implications of a miscalculation
So, out of our total 600 quota (3 groups x 200 per group), approximately 328 will fall from the rep sample, comprised of 150 A’s, 115 B’s, and 63 C’s. This means that we will be short approximately 272 respondents at a mix of incidences (50 A’s, 85 B’s, and 137 C’s). In order to fill these remaining quotas, and assuming they were filled independently, we might think that we need (333+580+2,740)=3,653 to find our 272, or 7.4% incidence. But, in reality, we need only the 2,740 as the others will fall, hence 272/2740 or almost exactly 10%. That figure was basically the $30 CPI figure, for a total cost of $8,160. Had we calculated independently, costs would have been (50 x $20) + (85 x $30) + (137 x $50) = $10,400, about 27% higher than the overlap method – pretty meaningful, especially when computed over many studies during the year.
So, when mapping out your sampling plan, pay attention to the subtleties of incidence. The cost implications can be significant, as well as the odds of having your proposal remaining in the pool of competitive bids, and you in the good graces of your clients.
by Bob Walker | May 26, 2017 | Uncategorized
We’ve seen this before
A recent review of tracking data with one of my e-commerce clients involved a discussion around NPS, or the “Net Promoter Score”, a measure that has been widely adopted in survey and customer satisfaction research. In research and marketing circles, familiarity with NPS is high, as is the 0-10 (thus 11) point scale from which it is derived. The scale defines three groups: Promoters, Passives, and Detractors. The percent who are Promoters minus the percent who are Detractors (multiplied by 100) is the NPS. I won’t go into the mechanics of how the groups are defined because that, too, is widely known. The issue for my client was that their NPS score was low when compared to other e-commerce sites and/or businesses they considered to be competitors. The question was: Why?
My client runs a dealer network, in which dealers sell their wares through the client’s e-commerce platform. You will recall that NPS is based on the concept of a recommendation to friends or colleagues. However, in a dealer network, dealers are effectively competing with one another. As a result, a dealer’s willingness to recommend my client’s products actually results in the dealer helping a competitor, which is not exactly what the dealer had in mind! We might find this same dynamic in a number of business-to-business or competitive marketplaces in which a recommendation is, effectively, self-defeating. This makes NPS totally inappropriate for businesses in which a “recommendation” is unnatural. I can recall personal situations when people have asked me for contractor recommendations; I’ll give them names but hope they won’t steal my network!
This seemingly small hiccup in the conceptual approach to NPS raised a larger issue. Many senior marketing executives believe that they need but one consumer measure (gathered in customer research) to determine customer health, and can overlook other measures of satisfaction or happiness that have also proven to be valuable indicators. Specifically, overall satisfaction, various measures of quality, perceived value, appropriateness for use, and so on all have their place and, collectively, provide a multi-dimensional view of customer health. There are also other self-reported behavioral measures (e.g., frequency of purchase, sizes, flavors, etc.) as well as attitudinal measures that can aid in interpretation of the broader consumer mindset.
Never supported by data
In his wonderful book “How Brands Grow”, Byron Sharp thoroughly debunks the notion of NPS, as well as the “thought experiment” that assumed zero cost for customer retention. You may recall that NPS was presumed to measure the level of retention a company could expect to achieve. There was little empirical evidence to support the relationship between high NPS and high customer retention. Sharp goes on to demonstrate that high levels of customer retention are (a) impossible to achieve, even in businesses presumed to have high levels of loyalty, due to natural switching behavior and (b) even high levels of customer retention cannot generate any significant positive impact on revenue because the number of truly loyal customers is so small.
Like a bad penny
Yet NPS persists. Why? Initially, NPS adoption was driven by its Harvard Business School imprimatur, but more importantly, it is simply due to inertia: it has already become institutionalized at many companies. For those who do not have time to dig below the surface, NPS serves a purpose in that it is simple. And, there are available published benchmarks which lend some comfort to marketing management — i.e., there is always some professional pressure to rely on what is au courant: the SVP of marketing will look snappy in the C-suite.
I suppose that in organizations with limited resources, who can only afford to ask one summary question, NPS will do. But an overall happiness, satisfaction, or liking measure is always superior. In my statistical work linking evaluative measures, such as intent to purchase, satisfaction, or willingness to recommend, I have consistently found higher correlations between satisfaction and a hard dependent measures of consumption (i.e., transactions or sales) than with a willingness to recommend (NPS). I suspect that this is because “willingness to recommend” is one step further removed from actual product experience or satisfaction. For example, think of a deodorant or toothpaste. You may be very happy with your deodorant, in which case you will report high satisfaction, but will you go out of your way to recommended it to someone else? That doesn’t pass the smell test (sorry, too tempting a pun).
If you are considering a customer satisfaction or tracking program, and are exploring what measures to use, experiment a bit. Choose from other measures that are more directly linked to customer happiness and brand health, and build a normative database that is best for your business around them. I highly recommend it (oops)!
