Marketing and strategy Archives

Don’t Let A/B Testing Mislead You

by Bob Walker | Oct 24, 2022 | Analytics, Communications, Marketing and strategy, Marketing research

A recent conversation on A/B testing with a client revealed an interesting perspective about messaging and positioning. The client, extolling their company’s rigorous A/B testing approach, failed to recognize a simple but scary fact: it is easy to compare multiple versions of a sub-optimal message. In the end, you end up with a “less-worse” version of an already weak message. This is not equivalent to building brand value over time — and building a moat around your brand’s essence.

What were they thinking?

The client had overlooked the obvious by ignoring underlying reasons to buy. Instead of testing which alternative was more persuasive based on price, the more important question they should have been asking was: what is the underlying motivation behind purchase? What segments, personas, or buyer types fall into our wheelhouse? Why should our brand be considered in this crowded category? This client, and so many others, seem to miss a simple tenant of marketing: why give away your marketing advantage so early in the game?

This client’s products have significant performance advantages over others in the category, yet they were A/B testing multiple executions built around being a lower-cost, value alternative. If they had taken just a little time to understand buyer behavior, they would have realized that price can be a relatively small factor in the buying decision when the brand looms large.

In this case, A/B testing was fueling a race to the bottom. By choosing the “less worse” option, the client had already decided that they would primarily compete on price, pushing them deeper into a commodity mindset for the customer.

When misused, A/B testing behaves like a cost-reduction test. There are many instructive lessons here: a well-known case is Maxwell House coffee. Over the course of a many years, the company increased its use of lower quality beans in the blend to cut its COGS. It conducted taste tests to make sure that consumers did not detect a difference when compared to the previous blend. But market share began to fall. Why? Because they never tested it against the original formula. What if there were thousands of Maxwell Houses across the globe instead of Starbucks? In the same way, test between meaningful options, rather than confine your evaluation to a narrow set of sub-optimal choices.

Be smart. A/B testing works best when the strategy is well-defined and plays to your advantage. First figure out what that is. Focus on highly persuasive messages that support your brand, rather than identify the best way to discount your business into oblivion. Don’t give away the store when you don’t have to.

Reframing Marketing Research Spending as an Option-Creating Investment

by Bob Walker | Jun 24, 2021 | Business strategy, Marketing and strategy, Marketing research

If you have been in business long enough, you know that the hard work of research is sometimes seen as optional or discretionary by some management teams because it’s hard to calculate the true ROI of research. But we’re thinking about this all wrong. Companies should be thinking about research as a way to separate winners from losers, and move the winners to market as quickly as possible at the lowest possible total cost. So let’s flip it around and consider the value of research using an investment framework.

Business spending and investments fall into three broad buckets, which are:

Infrastructure investments that include the costs of standing the business up and keeping it running at a baseline level. This includes the sunk costs of office space, utilities, computers, distribution centers, manufacturing, and support staff. The business cannot run without them, and the ROI cannot be easily calculated because it’s the paid-in capital needed to get the flywheel spinning.

Variable cost investments include all short-term spending to promote the company’s products. ROI calculations work best in these situations because there is a beginning, a middle, and an end to the spending and the program that is being run. The ROI question is: when I spend a dollar, how much will I get back (in the near term)? For example, advertisers and media companies obsessively focus on maximizing ROI by targeting (i.e., MTA), which is amazing but does little to identify promising ideas or address business strategy: it’s simply optimizing ad spend.

Option creating investments are by far the most interesting! These investments are made for marketing research. An “option creating investment” lets me put a little money down on the table to give me the option of owning something later that is worth much more. If I spend money for an “option” but it’s not going to pay off, then I walk away and let the option expire. Alternatively, if I have a winner, I am in the money. If I put $2 million down and I get back $20 million, my ROI is 10x and it’s time to exercise my option. The product moves over to the ROI category, supported by variable cost investments.

The other option is, of course, launching a product that you did not test and watching it fail spectacularly. All you have to be is right. But if you’re wrong and you’re the CEO, you might be looking for a new job!

Here’s a quick example. Let’s say we have 10 ideas, and each one of them costs $5K to test. Half of them move on to an R&D product development phase at $75K each, and these all move on to a product evaluation phase at $15K each. Two of these then move forward to test market at $500K, but only one of them performs well enough for a regional launch costing $2MM. All in (including the losers) I have spent $3.5MM.

The launched product achieves $10MM in Year 1 sales at a gross margin of 60%, or $6MM. My ROI (including the cost of all my losers) is 171%. If I have two winners, my ROI is even better at 343%. And I am not breaking out the cost of research alone, which is much smaller – I am including all of the costs associated with the launch.

Option creating investments can also be made in customer satisfaction research to identify additional ideas to insert into your screening programs. Over the course of time, the amount of money you may spend in research testing will be rounding error compared to the amount of money made by the winners.

Knowing what won’t work is as valuable as knowing what will by researching effectively. Well-designed research will continuously feed successful business performance and yield great ROI!

My thanks to Jay Kingley of Centricity for helping to shape this thinking!

Curation: The Next Wave of Marketing

by Bob Walker | May 26, 2021 | Business strategy, Marketing and strategy, Strategic research

Whenever we go to Amazon, or Netflix, or any other site, we are immediately presented with dozens, if not hundreds, of choices. Many of these choices are randomly selected by the retailer based on past purchase behavior across the buyer’s digital mesh. Across multiple devices, the company knows our age, sex, and geographical location, and perhaps can algorithmically make some deterministic assumptions about what we like or don’t like.

But that has yet to translate into something that is presented to the customer as a reasonable choice set. It is no wonder that consumers feel bombarded by choice. They are simply overwhelmed.

We are presented, every day, in multiple contexts, with too many choices. We are presented with too many choices when we read digital publications. We are presented with too many choices when we look at the social media feeds of LinkedIn or Facebook. We are presented with too many choices when looking for a TV show or a movie. Humans are simply not capable of synthesizing hundreds, if not thousands, of choice alternatives when they are presented as a mass (mess?) of individual decisions. Our cognitive capability collapses under the weight of all of the choice decisions that must be made when presented with too much choice.

Companies generally, and advertisers and media assets in particular, have failed to make the leap from choice to curation. This is a huge opportunity for marketers in simplifying the marketing message, making the overall customer experience that much less burdensome and taxing, and draws the consumer closer to the value proposition that attracted the consumer in the first place.

As a general rule, consumers do not like other people making decisions for them. A good case in point is grocery shopping. Yet, in urban environments, direct delivery makes much more sense due to the many obstacles for grocery shopping in congested cities. A grocery shopper doesn’t have to fight city traffic, load up a car, drive to and from their apartment, or leave and subsequently enter parking garages to get the week’s groceries. One of my former clients, FreshDirect, learned early on that their business model wasn’t solely built around the ability to deliver high-quality produce at reasonable prices. The secret ingredient were their drivers. The drivers knew their customers at a personal level, and were able to create a curated experience by making sure that certain things were done to the customer’s exact specifications.

Why is curation so hard? E-commerce has not figured this out at all. Not long ago I ordered tires for my road bike from Amazon. On a subsequent login, Amazon suggested other road bike tires I might be interested in — for a product category purchased annually (at best). The lack of synchronization between recommendations, purchase frequency, and my likely need was stunningly dumb. Yes, Amazon is enormous, yes they make lots of money, but they still have not moved the needle on the concept of curation in any meaningful sense. What if Amazon had a viable competitor that really understood curation?

On the flipside, one of my favorite examples of curation is Spotify. Once again, one would think that Apple (iTunes) would have figured this out long ago, but Spotify is a wonder. If I want music for concentration, there is a curated playlist. If I want calming classical in the background, there is a curated playlist. Do they get it right all the time? No, but they are pretty close most of the time. And I don’t mind if they miss. AN 80% hit rate is pretty good to me. At least there are humans involved in the decision-making process. OK, yes, perhaps also an algorithm, but at least it is a collaborative effort.

Marketers would be well advised to start thinking about how to anticipate the kinds of products and services that customers will be looking for in a world where choice is overly abundant. Curation is one of the ways that marketers can demonstrate that they are tuned in to what customers are seeking, rather than blindly and programmatically jamming messages at them without any thought to the choice overload that they create. Does the marketer want to convey something meaningful, or add more noise? So far it has been the latter.

I hope that more marketing and advertising initiatives will consider the notion that humans are very, very good at intuiting what other humans might like or enjoy. The concept of curation can form a much-needed bridge between the antiseptic world of algorithmic decision-making and true human connection.

The New World of Reaction-Based Marketing

by Bob Walker | Mar 25, 2021 | Analytics, Marketing and strategy, Marketing research

Those of us who have spent some time in research departments tend to think in linear terms. By that I mean that there is a “classic” sequence to follow to understand customer needs, new product opportunities, line extensions, new advertising, etc. For example, we might start with a strategic study to understand buyer needs and behavior, identify segments or personas, follow that with benefit screening or concept testing to assess interest, then move into advertising concepts, and then marry that with the product development track with R&D, address any deficiencies, move into a test market, and then a national introduction.

Not. Those days are long gone. There is no appetite for “research” or “insights” in the classic flow referred to above.

This is most obvious when you look at the revenue of major research firms, which have grown anemically the last five years. While it is important to understand customer/buyer needs, research can’t add nearly as much value until it understands the digital landscape. Joel Rubinson and Bill Harvey have written about this eloquently. Those of us who consider ourselves insights experts or researchers must come to grips with the fact that most companies have no interest in spending much time conversing with customers, even when it has strategic value.

Most companies feel the need to respond or react to what is happening right now, in real time. In fact, for many companies, response time is the only thing that matters. We are in an age of data lakes, auctions, programmatic and ROI – a world of reaction-based marketing. In this world, a brand demands that for every nickel it spends, a nickel in sales should be generated. Companies are not interested in convincing you that their product is superior, or meets your needs, or fits your lifestyle unless they get paid back. Nor are they interested in the protective benefits of long-term brand building. This is a finance-centric rather than marketing-centric philosophy.

Reaction-based marketing has four primary characteristics which distinguish it from traditional marketing and brand building

ROI is the primary KPI used to measure marketing success.
Decisions are engineering-driven, not consumer needs-driven.
Decisions are event-based, not strategically- or equity-driven.
The cost of failure is less than the cost of testing.

A great example is Amazon, which alone has created these exact marketing conditions. It is a complete ecosystem for testing all elements of the marketing mix (excluding distribution, which it owns). Yet has Amazon not created the perfect ecosystem for driving brands into commodity status? A great example is alkaline batteries: Duracell currently cells 24 AAA batteries for $16. Amazon sells 48 AAA batteries for $15. I wonder who wins that battle?

As researchers and insights experts, where we add value is the missing link between all of the automated ecosystems that are competing for the consumer’s attention, and how the consumer thinks and feels. That market is wide open.

This “T” is for Testing

by Bob Walker | Jul 8, 2020 | Marketing and strategy, Marketing research, Medium-sized business, Research design, Small-sized business

Small-to-medium sized businesses (SMBs) can use a simple reference model for their marketing and customer insights efforts. By focusing on what I call the “Three T’s” (targeting, testing, and tracking), your business operations will be continually guided and improved by staying focused on your core target, supported by continuous testing of new products and services, and by objectively monitoring your progress over time. Today let’s take a closer look at one of the these areas: testing.

Instill a Testing Mindset

Testing can involve a multitude of variables, but as an SMB you need to think about two fundamental dimensions when considering a test of any kind: tests that affect your brand, and tests to determine whether your ad spend is working.

My colleague Joel Rubinson makes this key distinction between what he calls performance vs. brand marketing. Each supports the other; they are not in opposition to one another. In simple terms, performance marketing is focused primarily on media allocation and the optimization of your ad spending (ROAS), while brand marketing is focused primarily on finding the best ways to communicate the fundamental premise of your brand, such as your brand’s features, benefits, and desired end-user target.

It is in this context that I want you to think about two types of testing: what I will call brand concept testing of the brand promise (in various concept formats); and performance testing, the most commonly used form being A/B testing.

Brand Concept Testing

When we want to communicate the essence of a brand or an idea to a prospective customer, we do it with a stimulus known as a concept. A concept is an idea before it is marketed solely for testing purposes, so that we can understand consumer reactions to it. We test concepts to reduce the risk of making a mistake when launching an idea, to find the best way to describe an idea, and how to best communicate to our target audience.

A concept needs to communicate a compelling end-benefit or a solution to a problem using language that a target consumer can not only understand, but internalize and relate to emotionally. Concepts can differ significantly in their language, layout, image content, and other characteristics. The format of your concept will vary depending on the type of information you need for your brand and the type of test you are planning.

There are concepts written specifically for screening purposes that have very little detail or descriptive information (on purpose). There are concepts that go a little further, with more descriptive information, but still short of being fully developed. And then there are concepts that are close to finished advertising, much like you would see on a landing page or in print media. Here are three types of brand concepts that you should know:

Kernels are ideas or benefits presented as statements.

Kernels are used in screening tests to efficiently identify winners vs. losers. Kernels are evaluated on just a few measures (e.g., purchase intent, uniqueness, superiority), and each respondent sees all kernels, and each is assessed on all measures. Kernels can be attributes, benefits, or distinct ideas. This type of test is also called a “benefit screen”.

If kernels are distinct ideas, the analysis focuses on the top performers and profiling them on demographics, geographical areas, or other methods such as attitudinal scoring. If the stimuli are end-benefits or positioning statements, we can use tools to identify underlying themes that might convey an even bigger thematic idea.

White card concepts are simply words on a page without high-quality images or fluffy language.

White card concepts are typically comprised of 4-8 sentences, factually stating the problem, usage situation, or need; they also state the end-benefit, solution, or final result provided. White card concepts can be existing products, stand-alone ideas, line extensions, or new uses and repositionings. They can include price, flavors, sizes, dosing, brand name, packaging information, and even a basic visual (i.e., a B&W drawing). Because the goal is to test the waters in a bit more detail, some diagnostic questions are included – but the number of questions is limited because we are typically evaluating multiple concept ideas.

Full concepts are used to capture more complete reactions, and when fine-tuning your messaging or language is essential prior to a launch or ad spending.

Full concepts often have the benefit of qualitative insights to develop language, positioning, tone, or emotion of an ideas that showed promise in previous screening work. Full concept testing can be done “monadically” (i.e., the respondent sees one idea at a time in its own cell of respondents), or in a sequential design.

Full concepts are longer, written to include all that might be conveyed in the time an ad is exposed (e.g., 15- or 30-seconds). They can also be more elaborate in their situational set-up or premise, use of demonstration cases, or other info.

A/B Testing

You might think that A/B testing always provides a clear choice, but there is usually more to the story than the difference between two variables.

The A vs. B variants you are testing might be affected by a series of previous decisions made long before either A or B were evaluated head-to-head. For example, if A is your current campaign that includes search, PR, and Facebook ads, and B does not leverage the campaign you are now running, your test is already biased against B. Or, perhaps your objective was impressions, but one option delivered much higher conversions. So, interpreting results can quickly become more complicated than perhaps they first appeared.

But, for argument sake, let’s assume that A and B start from the same point, and neither will be biased by previous advertising or spending level decisions. If so, A/B testing can be interpreted unbiasedly, and executed within any number of environments, such as CRM systems (HubSpot, Salesforce, etc.), dedicated A/B testing environments such as Central Control or Unbounce, and even some popular web hosting platforms provide an opportunity to conduct simple A/B tests.

Mechanically, the ad testing company you work with will develop two (or more) landing pages (A, B, C, etc.) and visitors to your site will be randomly redirected to one of those variants. Google Analytics and other web traffic statistics can be utilized to determine which variant is most effective at, for example: lowering bounce rates, achieving conversions, increasing CTRs, or other metrics you choose. A/B test designs can also revolve around content of the landing page, the overall site experience, or changes to ad spend, placement, location, context/environment, and more (see above brand concept formats).

Scratching the Surface

There are a multitude of different testing and design options for you to consider as an SMB. I have given you a taste, so get out there and test! Working with a marketing insights and research expert is your best guarantee that the type of concept and testing environment is designed, executed, and analyzed effectively. At Surveys & Forecasts, LLC we have worked with many different companies to help them develop optimal brand strategies and concepts, identify which execution best communicates their brand’s proposition, and which marketing program is most effective for their limited ad dollars.

Are You an SMB? Use the Three T’s!

by Bob Walker | Jul 8, 2020 | Customer satisfaction, Marketing and strategy, Medium-sized business, Small-sized business

If you work for a small-to-medium-sized company (SMB) and are in a marketing role, you are no doubt strapped for resources in the form of people, money, or time. This is especially true when it comes to research for understanding customers, prospects, and how to best message and market your product or service. The day-to-day challenges of managing your business can easily get in the way of thinking and planning about ways to grow or expand.

A “plan” does not scream out for immediate attention. Plans assume that conditions are stable and predictable — but in the new era of COVID-19, conditions are far from ordinary. As a colleague of mine recently said, “If you see lots of umbrellas, you can conclude that it’s raining, but it doesn’t tell you when it will rain next.” Or if there is a hurricane coming. The obvious challenge we are ALL facing now is how to meet future customer needs given the restrictions imposed by COVID-19, and a future that is far from certain.

What kind of planning can you do right now? Certainly, long-term planning will be challenging, but let’s start with some simple rules — what I call the “three T’s” of SMB marketing.

Targeting: involves identifying your target audience, the best way to reach your target, and the best way to communicate your story (messaging).

Work hard to make sure your user target is right. Begin to dimensionalize it by using readily available data sources. A great place to start is the US Census (business and county data patterns), and companies that sell secondary research (e.g., “pre-packaged” reports on various categories, from companies like Statista or Packaged Facts).

Ask yourself: what problem are you solving? How is your solution better than the competition? Use custom research to better define your target and refine your USP. Common descriptors are “market studies”, “positioning research”, and “market segmentation ” to understand and size your market. In some cases you can link survey data with shopper data (which can be appended to your surveys) to understand who you should target. Some of this can be done subjectively based on your own knowledge, but back up your opinions with some hard facts.

If you are in a fast moving consumer goods category (FMCG), or any category where customers have multiple buying opportunities during the year, consider targeting loyal customers or loyal competitors. They are unlikely to be consuming all of their category volume solely from either you or the other guy. This is sometimes called a “share of requirements” strategy — i.e., the share your brand has of total category consumption.

As another close-in strategy, find non-buyers who fit the profile (demographically, attitudinally) of loyal customers, as this is likely to reap rewards. Initially, maximize your reach (i.e., the total number of people exposed to your message), but do not over-emphasize it. If launching a product, it is initially a numbers game. Not every consumer or prospect is equally valuable/profitable, but you have to start somewhere.

If you are lucky enough to have a marketing or media budget, now is the time to re-examine all of the possible ways to target customers… via social media (e.g., Facebook, Instagram, Pinterest, LinkedIn), advanced and addressable TV (read more from media expert Bill Harvey here), and linear (live) and local TV. But more importantly, is your messaging on-strategy?

Testing: test different messages, selling points, products, or features in each channel you advertise, to educate and communicate. Test, test, test!

Testing is an iterative, tactical process that also feeds your business strategy. The more you learn, the more your strategy will adapt. You must continuously test to find that winning message, product, or formulation for your target (or targets). Give special consideration to testing among loyal customers (if you have them) or best case prospects. As noted, loyal customers are an ideal research audience, and can provide significant insight, as they are already pre-disposed to you behaviorally and attitudinally.

Pay special attention to testing your creative. Experts in media attribution assign about 80% to the impact of creative (i.e., the combined impact of the raw information about your brand’s story, combined with words, images and sound) on conversion. Put another way, if your message resonates, the teacher will give you an automatic ‘B’! The rest depends on the delivery mechanism and overall customer journey. With so much fertile ground in the creative itself, focus your initial efforts here.

Many platforms allow you to A/B test different offers in real time (i.e., pricing, flavors, colors, products), messaging (different creative executions), and delivery schedules (i.e., continuous advertising, vs. flights). Testing resources here include Central Control or independent platforms like One Count. Don’t get distracted with the more sophisticated attribution modeling companies (e.g., C3 Metrics, Sequent Partners) – as an SMB, you’re just not ready for them yet.

Occasionally, when I talk about doing survey research, SMB clients think I have three heads, but do not discount the power of survey research to provide insight into what consumers are looking for, or to help narrow down options for your retail store or e-commerce site. Get a free SurveyMonkey account, or use forms-based tools like Constant Contact, Google Forms, or Office 365 Forms to gather feedback. Some of these tools are basic, and you may need to graduate to a more sophisticated platform as your testing needs grow, but their basic feature set is excellent.

Here’s another idea: leave your office, go out into the world (mask on, of course), and visit some real stores! Woo-hoo! Yes, lots of commerce is done via the web, but the vast majority is still brick and mortar, and the retail environment is closest to the end customer. If you already have distribution, go visit stores where your product is on-shelf. Note what is working, and what is not. While you are there, check out your competitors, or hire a mystery shopping firm to see if there are problems in finding your product. Are competitors better or worse than you? In what ways? How can you improve?

Tracking: once you have identified your target, and you have tested and identified messages for your target, monitor how well you are doing.

If you can’t measure it, you can’t manage it. Develop forms of continuous reporting, i.e., sales x channel x region x segment; or customer reviews; or other forms of objective feedback. Build monitoring systems using dashboards and visualizations to know whether refinements or adjustments are needed. Consider an ongoing advisory panel comprised of customers or clients, or a heavy user panel, to give you regular feedback on your product or service. Gather ideas from distributors/resellers to learn about issues that you might be unaware of. Talk to competitors if you can. Smartly designed research can yield significant insights.

Most of all, just listen. Keep your ear close to the ground and gather feedback like a sponge. Review each customer rating through an objective lens and see how it can enhance or improve your business. Don’t have a thin skin when it comes to feedback of any kind. Customers can be especially sharp: the internet lets people hide and in turn ‘permission’ to be nasty. Take the high road.

Last, this doesn’t have to be expensive. Well-designed research and good judgment can go a long way to help your business thrive and grow. As your business becomes more successful and more sophisticated, the need for feedback on individual aspects of your business will increase (i.e., individual products, or new customer targets).

As your needs grow, consider working with an expert to help identify problem areas and to refine your overall marketing plan. After all, if your marketing problems were that easy to solve, you would’ve figured them out by now. It will be money well spent.

« Older Entries

Next Entries »