On this page
- What did the Colgate and PyMC Labs study actually test?
- What is Semantic Similarity Rating (SSR)?
- How well did synthetic consumers match real humans?
- The finding the viral thread missed: personas carry the signal
- What replicated and what did not
- What the authors admit about the limits
- What the viral thread gets wrong
- How should an Australian product team use this?
- How do you run an SSR test yourself?
- Frequently asked questions
- What is Semantic Similarity Rating (SSR)?
- Does this mean market research panels are obsolete?
- How accurate are LLM synthetic consumers?
- Can I use synthetic consumers for pricing research?
- Is it safe to send customer data to OpenAI for synthetic consumer research?
Last Updated: 21 November 2026
By AJ Awan, Founder of Flowtivity. Former EY management consultant, TOGAF certified enterprise architect.
Key takeaways
- ▪ Colgate-Palmolive and PyMC Labs tested 57 real concept surveys with 9,300 human responses. Their Semantic Similarity Rating method reached 90% of human test-retest reliability with GPT-4o and 92% with Gemini 2.0 Flash.
- ▪ The method never asks the model for a number. It asks for free text, embeds the answer, and compares it to five anchor statements to produce a probability distribution across the 1 to 5 scale.
- ▪ Asking for a direct 1 to 5 rating failed badly. The models answered "3" almost every time, scoring 0.26 distributional similarity against the real panels.
- ▪ Personas carry the signal. Remove demographics and the distribution match rises to 0.91 while concept ranking collapses from 92% to 50%.
- ▪ Age, income, price tier and concept source patterns replicated. Gender, region and ethnicity did not, and the anchor statements were tuned on these 57 surveys.
- ▪ Use it as a screening instrument. Run synthetic first, spend human panel money on the few concepts that survive, and keep humans in the loop for anything that changes a P&L.
The thread says a toothpaste company killed market research. The paper says something narrower and more useful.
Published on arXiv in October 2025 by researchers at PyMC Labs and Colgate-Palmolive, then resurfaced on X on 21 November 2026 and picked up 166,799 views, the study shows that the problem with LLM survey respondents was never the model. It was the question. Ask for a number and you get mush. Ask for a paragraph and map it back onto the scale with embeddings, and synthetic panels start tracking real ones closely.
That is a real result. It is not the death of an industry. The authors say so in their own conclusion, and the most interesting number in the paper is one the viral thread never mentions.
What did the Colgate and PyMC Labs study actually test?
The study tested 57 real consumer concept surveys on personal care products, run by an unnamed leading manufacturer. Each survey had 150 to 400 US participants, 9,300 responses in total, and every participant rated purchase intent on a five point scale. The researchers then rebuilt each panel with synthetic respondents: a language model told to impersonate a person with the same demographic attributes, shown the same product concept. Success was measured two ways, how closely the synthetic response distribution matched the real one, and how well the synthetic panel ranked the 57 concepts against the human ranking.
The authors are Benjamin Maier, Ulf Aslak, Luca Fiaschi, Nina Rismal, Kemble Fletcher, Christian Luhmann and Thomas Wiecki from PyMC Labs, with Robbie Dow and Kli Pappas from Colgate-Palmolive. The paper is arXiv:2510.08338, first posted 9 October 2025, revised three times that month, and licensed CC BY 4.0.
One definition matters before any number makes sense. The headline metric, correlation attainment, is not accuracy against humans. It is the share of the maximum correlation the human data could achieve against itself. The researchers simulated 2,000 test-retest splits of the real respondents, measured how well one half of a real panel predicted the other half, and treated that as the ceiling. So "90% of human test-retest reliability" means the synthetic panel got 90% of the way to the best score the noisy human data could possibly produce. It does not mean 90% of synthetic respondents agreed with a real person.
What is Semantic Similarity Rating (SSR)?
Semantic Similarity Rating is an elicitation method that converts an LLM's free-text answer into a probability distribution across a Likert scale. Instead of asking for a number, you ask the model how likely it is to buy the product and let it answer in its own words. You then embed that answer and measure its cosine similarity to five reference statements, one for each point on the scale. The similarities become probabilities. The output is not one score, it is a spread of plausible scores for each synthetic respondent, which aggregates into a panel distribution.
The paper compares three ways to ask the same question. Direct Likert rating (DLR) treats the model like a human respondent and asks for a single integer from 1 to 5. Follow-up Likert rating (FLR) lets the model write a short answer first, then asks the same model, now prompted as a "Likert rating expert", to map its own text onto an integer. SSR keeps the text and never collapses it to a single integer. It embeds the response with OpenAI's text-embedding-3-small model, compares it to anchor statements, and produces a probability mass function over the five points.

The anchor design is where most of the craft sits. The researchers wrote six sets of five statements, each set running from purchase improbable to purchase probable, and averaged the resulting distributions across all six. The anchors are short, generic and domain independent, so they can apply to any consumer product. They also subtract the minimum similarity within each set before normalising, because in embedding space the distance between "very unlikely to buy" and "very likely to buy" is small enough that the raw distribution comes out almost flat.

The implementation is open source. The PyMC Labs repository has 189 stars and 49 forks as of 21 November 2026, was created in August 2025 and was last pushed in October 2026. It is Apache-2.0 licensed on the GitHub API, though the README footer still says MIT, and the package installs directly from GitHub with pip.
How well did synthetic consumers match real humans?
Across the 57 surveys, SSR reached 90% correlation attainment with GPT-4o and 92% with Gemini 2.0 Flash, while keeping distributional similarity at 0.88 and 0.80. Direct rating scored about 80% attainment but only 0.26 and 0.39 distributional similarity, which means it got the ranking roughly right while producing response distributions that looked nothing like real panels. Follow-up rating sat in between. The gap between direct and semantic elicitation is the paper's core claim, and it is large.
| Method | What the model does | GPT-4o attainment | GPT-4o KS similarity | Gem-2f attainment | Gem-2f KS similarity |
|---|---|---|---|---|---|
| Direct Likert rating | Returns one integer from 1 to 5 | about 80% | 0.26 | about 80% | 0.39 |
| Follow-up Likert rating | Writes text, then rates its own text | 85% | 0.72 | 90% | 0.59 |
| Semantic similarity rating | Writes text, embeddings map it to anchors | 90% | 0.88 | 92% | 0.80 |
The failure mode of direct rating is worth sitting with, because it is the failure mode most businesses will hit if they try this without reading the method. Asked for a number, both models answered "3" almost every time. They almost never said 1 or 5, while real respondents clustered on 4 and 5. When the researchers adjusted the prompt to push the models toward the extremes, distributional similarity improved slightly but ranking correlation dropped. The models over-corrected toward the shape of human data and lost the signal that makes a concept test useful.
There is a second difference in the numbers. Synthetic purchase intent is more spread out than human purchase intent. Across surveys the human mean was 4.0 with a standard deviation of 0.1. Direct-rating synthetic panels averaged 2.96 with a standard deviation of 0.11, and the spread across concepts was much wider. The models are harsher than people on weak concepts. That is useful for discrimination, and it also means you cannot read a synthetic mean as a demand forecast.
The finding the viral thread missed: personas carry the signal
When the researchers removed the demographic block from the persona prompt and left Gemini 2.0 Flash with nothing but the product concept, distributional similarity rose to 0.91, the best score anywhere in the paper, and the synthetic mean purchase intent matched the human mean exactly at 4.0. Correlation attainment fell to 50%, against 92% for the same model with demographics attached. A synthetic panel can look right and still order your concepts wrongly. Distribution match on its own is not evidence that the panel works.
The authors' explanation is blunt. Without a detailed persona the model defaults to agreeable and stops using the information in the concept. It rates everything positively, which happens to match a human panel that is also skewed positive, and it produces no ranking signal. This is the trap for anyone shopping for a synthetic research tool. A vendor demo that shows a beautifully matched response distribution may be showing you the model's enthusiasm, not its judgement.

This also reframes the headline. The 90% figure depends on persona conditioning, and persona conditioning is exactly the part of the method that the authors say is unreliable for some subgroups.
What replicated and what did not
The synthetic panels reproduced age, income, product category, concept source and price tier patterns. Age followed a concave curve, with younger and older respondents rating lower than the middle cohorts, and GPT-4o mirrored it. Income statements that implied budget problems produced lower purchase intent, and both models replicated that. Category IV concepts rated highest and Category I lowest. Concepts from "Source B" drew negative reactions from humans and synthetic respondents alike. Price tiers 3 and 4 rated higher than Tier 1. Gender and dwelling region did not replicate well, and ethnicity was not consistent either.
| Dimension | Result | What it means for a real study |
|---|---|---|
| Age | Concave pattern replicated by GPT-4o. Gemini matched younger cohorts only. | Usable for directional age splits, not for precise cohort sizing. |
| Income | Budget-constrained statements produced lower intent in both models. | Usable, with caution where the persona wording is extreme. |
| Price tier | Tier 3 and Tier 4 higher, Tier 1 lowest, matched by both models. | Usable for ranking concepts by tier. Price elasticity was not tested. |
| Category and concept source | Both patterns replicated closely. | Strongest evidence in the paper, because both are coarse features. |
| Gender and region | Not replicated well, though neither feature moved human intent much. | Do not use synthetic panels for gender or regional splits. |
| Ethnicity | Not consistently replicated. | Treat any subgroup claim as unvalidated until tested against humans. |
The authors are direct about this. Persona conditioning "cannot yet be treated as a reliable proxy for all subpopulations", and researchers "must therefore use caution when interpreting subgroup analyses from synthetic panels". That sentence should be printed on every synthetic research proposal that offers demographic cuts.
What the authors admit about the limits
The paper's limitations section is more honest than the thread built on top of it. The six anchor sets were "manually optimized for the 57 surveys subject to this study", so how well they generalise to other surveys "remains elusive". The method's validity is bounded by what the model already knows about the category, and oral care is one of the most heavily reviewed categories on the internet. The authors write that in domains where that background knowledge is sparse, "SSR will not conjure valid consumer preferences". Hallucination risk is not eliminated. The mapping depends on the embedding model, though swapping to the larger OpenAI embedding model left results virtually unchanged. And the method cannot capture budget constraints, cultural context or marketing exposure.
There is also a counter-datapoint in the appendix that most summaries skip. When the researchers tested a second survey question, "How relevant was the concept?", SSR reached 82% correlation attainment and 0.81 distributional similarity, while the simpler follow-up rating reached 91% attainment at 0.62 similarity. The simpler method ranked concepts better on that question while SSR still won on distribution shape. SSR is not universally superior, and the paper says so.
The conclusion is unambiguous: "While not a wholesale replacement for human research, SSR establishes a credible framework for augmenting and accelerating consumer insight generation."

What the viral thread gets wrong
The thread is a good distribution hook and a poor summary. It compresses a 28-page paper into six confident claims, and when you put those claims beside the paper's own text, four survive with qualification, one is a straight extrapolation, and one is contradicted by the authors' conclusion. The table below is the fact-check layer for anyone who saw the post and is now budgeting for a synthetic panel.
| Thread claim | What the paper says | Verdict |
|---|---|---|
| "Killed the entire market research industry" | "Not a wholesale replacement for human research." Recommends synthetic screening, then human panels for promising candidates. | Contradicted |
| "Predict purchase intent by simply asking LLMs to roleplay" | Roleplay is necessary but not sufficient. Free-text elicitation plus embedding mapping is the method. Direct ratings fail. | Partly true |
| "90% of human test-retest reliability" | 90% with GPT-4o, 92% with Gemini 2.0 Flash, as a share of the achievable ceiling. | True, with a definition caveat |
| "Perfectly mirrored how age brackets and income levels react to price changes" | Age and income patterns replicated, as did price tiers. Price changes were never tested. Gender and region did not replicate. | Overstated |
| "Deeper, more critical feedback than actual humans" | Authors report richer rationales, explicit objections and less positivity bias in the synthetic text. | Supported, authors' own read |
| "1,000 interviews overnight, A/B test pricing across every demographic" | 9,300 synthetic responses were generated. Price elasticity and demographic A/B tests were not part of the study. | Extrapolation |
None of this makes the paper less impressive. It makes it more useful, because the limits tell you where to point it.
How should an Australian product team use this?
Treat SSR as a screening instrument with three stages. Stage one runs every concept through a synthetic panel built from your target personas, and it costs almost nothing. Stage two takes the six or so concepts that survive and puts them through a real human panel, which is where the money goes. Stage three takes two concepts into an in-market test. The synthetic stage does not replace the human stage. It decides which concepts deserve one.

The cost arithmetic is ours, not the paper's, and it uses list prices we checked on 21 November 2026. GPT-4o charges about US$2.50 per million input tokens and US$10 per million output tokens. A persona plus concept prompt is roughly 1,000 input tokens and a short answer is about 150 output tokens, which puts one synthetic response at about US$0.004. Forty concepts against 200 personas is 8,000 responses, so about US$32, plus a negligible embedding bill. A human panel of 1,800 completes at an assumed US$5 per complete is US$9,000. The screen is roughly 280 times cheaper, and it removes most of the concepts before you spend the US$9,000.
Two cautions for an Australian team. First, the paper tested GPT-4o and Gemini 2.0 Flash. Model behaviour changes between releases, so anchor sets and calibration need to be re-run whenever you change model, not inherited from a blog post. Second, if your personas are built from real customer records, sending them to a US model endpoint is an overseas disclosure under Australian Privacy Principle 8, which requires reasonable steps before that happens.
That second point is where local inference earns its place. The prompts in this method are short and the outputs are shorter, so throughput is not the constraint. On our dual DGX Spark setup we measure 35 tokens per second on a single stream and 62 tokens per second at four concurrent streams. A thousand synthetic interviews at 150 output tokens each is 150,000 tokens, which is about 71 minutes on one stream and about 40 minutes at four. The "overnight panel" claim is not a stretch, and it runs without a single persona leaving the building.
How do you run an SSR test yourself?

Start with one concept you already have human data for. Build persona profiles from your own panel or census tables, write six anchor sets, elicit free text, map it with embeddings, and compare the synthetic distribution and concept ranking against the real survey before you scale. If ranking correlation is weak, recalibrate the anchors rather than adding more synthetic respondents. The method is cheap enough that the validation step costs less than the mistake it prevents.
- Build the persona set. Use age, income and household structure. Treat gender, region and ethnicity as unvalidated, because the paper found they do not replicate reliably.
- Write six anchor sets. Five short statements per set, from purchase improbable to purchase probable, generic enough to fit any category. Average across sets.
- Elicit text, not numbers. Ask how likely the persona is to purchase, and require a short free-text answer. Sample twice per prompt.
- Map with embeddings. Cosine similarity to every anchor, subtract the minimum in the set, normalise, then aggregate to a panel distribution. The open-source package does this in a few lines.
- Validate before scaling. Compare against one real survey using KS similarity and ranking correlation. Keep the human benchmark on file for the next model change.
Frequently asked questions
What is Semantic Similarity Rating (SSR)?
SSR is an elicitation method that turns an LLM's free-text answer into a probability distribution over a Likert scale. The model writes how likely it is to buy in its own words, that text is embedded, and cosine similarity to five anchor statements becomes the probability of each score. The output is a spread of plausible scores rather than one forced number.
Does this mean market research panels are obsolete?
No. The authors say SSR is not a wholesale replacement for human research. They recommend synthetic screening followed by human panels for the most promising concepts. The method also failed to replicate gender, region and ethnicity patterns, and its anchors were tuned on the 57 surveys in the study.
How accurate are LLM synthetic consumers?
In this benchmark, SSR reached 90% of human test-retest reliability with GPT-4o and 92% with Gemini 2.0 Flash across 57 surveys and 9,300 human responses. That is a share of the maximum correlation the human data can achieve against itself, not agreement with any individual respondent. Direct 1 to 5 ratings scored about 80% with distributional similarity of only 0.26.
Can I use synthetic consumers for pricing research?
Only partly. The study replicated how purchase intent varied across product price tiers. It did not test price elasticity or reactions to a price change, and the authors note the method cannot capture budget constraints, cultural context or marketing exposure. Use it to rank concepts by tier, not to forecast demand at a specific price point.
Is it safe to send customer data to OpenAI for synthetic consumer research?
Australian Privacy Principle 8 requires reasonable steps before disclosing personal information to an overseas recipient, and a US-hosted model endpoint is an overseas disclosure. If personas come from real customer records, that data leaves your control. Running the same prompts against a locally hosted open-weight model avoids the disclosure and keeps the persona set on your own hardware.
What's actually useful in AI, every Friday
We follow everything happening in AI and send what's worth your time: new models, AI agents, and how to put them to work. One email a week, unsubscribe anytime.



