At yesterday's ARF Marketing Accelerator in NYC (November 10, 2026), over half of the presentations focused on one or more aspects of synthetic data. As a marketing researcher, I was most struck by Keith Smith of the Marketing Science Institute (MSI) whose headline was "Plausibility is not validation".
Synthetic respondents, digital twins, and AI simulations can produce results that look convincing without being reliable enough for the decision at hand. One of MSI's stress tests deserves special attention, and is especially concerning for those of us who spend time looking for outliers and special cases that could be the source of inspiration or the next great new product introduction: variability loss.
The distribution of synthetic respondents can match measures of central tendency at the expense of trimming outliers. Although synthetic data generators are highly sophisticated, and use virtual stand-ins to simulate what actual humans might choose, the effect can resemble mean substitution. For those less familiar, mean substitution fills in missing data with average values, sometimes calculated within groups such as age or gender. The averages hold, but variance tends to disappear.
MSI's answer is practical: before using a synthetic sample and validating the output, first decide what decisions you need to make. Broad exploration of new areas perhaps needs less rigor; prioritizing or refining ideas needs more. A forecast or major launch clearly demands independent evidence from real people in real markets. As MSI puts it: set the standard before you see the answer.
This also connects to a way of categorizing synthetic data results that I've been thinking about. If we judge findings by how "plausible" they are, then two kinds of mistakes are possible. We can treat them almost like Type 1 or Type 2 errors in statistics:
- We reject an AI answer or recommendation when it's true, perhaps because it surprised us (think of this as a Type I error); or
- We accept an AI answer or recommendation when it's false, perhaps because it sounds right (think of this as a Type II error)
Both become errors in judgment, yet only one leaves a visible trail. Accepting a wrong answer can lead to a failed launch. Rejecting a true answer leads to a missed opportunity that no one sees. From my experience, most organizations only learn from mistakes that they can see (i.e., when we stop to look at the car wreck). The less visible errors are those they failed to see -- or chose not to.
Variability loss makes missed opportunities more likely. New opportunities often sit at the edges, among a minority of people who respond differently. If synthetic data compresses the edges by eliminating variance, a surprising finding or a new opportunity never surfaces. No one rejects it, because no one sees it. MSI goes on to flag segmentation, forecasting, and innovation as among the highest-risk use cases, which is where the edges matter most.
MSI's checklist, "Is Your Synthetic Evidence Decision-Ready?" is worth reading for any research team using AI. Its central point is key: plausibility alone isn't validation. Because AI is so convincing, and because we all fall prey to the bias in our own beliefs and values, the human tendency for agreeableness is a significant distortion field. That is the real danger. Companies would be well-advised to replace "plausibility" with "hypothesis". The hypothesis then opens the door to further investigation and testing, and ultimately requires proof.