Large Language Models Attempt to Replace Testable Humans in Psychological Measurements

A new study investigates whether a large language model could replace part of the traditional human experimentation when developing psychological surveys and tests. Building psychometric scales, such as personality or well-being measures, typically requires large samples, multiple rounds of testing, and a lot of resources. Now researchers are experimenting to see if part of this work can be done on a computer, so to speak, in silico, using artificially simulated respondents.

The idea is to ask a large language model to "play" a person with a certain demographic profile who answers the test questions. This would create a synthetic response set that could be used in the experimental phase before actual data collection with real people.

In four pre-registered studies conducted with a dataset of approximately 300 people, researchers examined whether such response sets produced by the language model could replicate the latent structures and measurement properties of human responses. In the first two sub-studies, the model-generated data was compared to real response data from two already validated psychometric scales. In the latter two sub-studies, entirely new scales were constructed using exploratory factor analysis based solely on simulated data, and then it was examined whether the discovered structures generalized to new, later collected datasets.

The study does not yet replace traditional psychological testing, but it explores to what extent synthetic population data could speed up and reduce the cost of early-stage scale development. If latent structures and measurement accuracy are reliably replicated, large language models could become a new tool in the design of psychological measures.

Source: In Silico Development of Psychometric Scales: Feasibility of Representative Population Data Simulation with LLMs, ArXiv (AI).