LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings

★

☆

SHARE

Overview

Paper Summary

Conflicts of Interest

Identified Weaknesses

Rating Explanation

Good to know

Topic Hierarchy

File Information

Paper Summary

Paperzilla title

Your Next Survey Respondent: It's a Chatbot (If You Ask Nicely)

This paper introduces Semantic Similarity Rating (SSR), a new method allowing large language models (LLMs) to accurately simulate human purchase intent by having them generate free-text responses, which are then mapped to Likert scales based on semantic similarity. The method, tested on 57 personal care product surveys, achieved 90% human test-retest reliability and produced realistic response distributions, outperforming direct numerical rating requests. It also generated rich qualitative feedback, though the reference statements were manually optimized for this dataset and not all demographics were replicated consistently.

Possible Conflicts of Interest

Two authors (Robbie Dow and Kli Pappas) are employed by Colgate-Palmolive Company, a 'leading corporation in that market.' The study itself analyzes '57 consumer research surveys on personal care product concepts conducted by a leading corporation in that market,' which strongly implies Colgate-Palmolive's internal data. Additionally, the other authors are from PyMC Labs, a company whose description suggests a business interest in 'scalable consumer research simulations.' This constitutes a conflict of interest as the research evaluates a method potentially beneficial to the authors' employers.

Identified Weaknesses

Manually Optimized Reference Statements

The method relies on 'carefully designed reference statements' that were 'manually optimized for the 57 surveys subject to this study.' This makes it 'elusive how well they would perform for other surveys' without significant re-optimization, limiting generalizability and ease of application across diverse domains.

Inconsistent Demographic Replication

While LLMs captured some demographic patterns (e.g., age and income), others like gender, region, and ethnicity were 'not consistently replicated,' suggesting caution when interpreting subgroup analyses from synthetic panels.

Bounded by LLM Training Data Knowledge

The effectiveness of SSR is limited by the LLM's training data. It performed well for oral care products likely due to abundant discussions in the training corpus, but 'will not conjure valid consumer preferences' for domains with sparse background knowledge, posing a risk of hallucination.

Inability to Capture Real-World Contingencies

Synthetic consumers cannot fully simulate real-world factors influencing purchasing behavior, such as budget constraints, cultural context, or marketing exposure.

Dependence on Embedding Model and Similarity Measure

The performance of SSR is sensitive to the choice of embedding model and similarity measure, which may require further benchmarking or optimization for different applications.

Rating Explanation

The paper presents a novel and effective methodological approach (SSR) to address a known limitation of LLMs in consumer research, demonstrating strong performance metrics on a substantial dataset. It's a significant practical contribution. The stated limitations are well-acknowledged and the conflict of interest, while present, does not invalidate the methodology itself.

Good to know

This is our free standard analysis. Paperzilla Pro fact-checks every citation, researches author backgrounds and funding sources, and uses advanced AI reasoning for more thorough insights.

Explore Pro →