References
Bail, C. A. Can generative AI improve social science? Proc. Natl Acad. Sci. USA 121, e2314021121 (2024).
Grossmann, I. et al. AI and the transformation of social science research. Science 380, 1108–1109 (2023).
Luo, X. et al. Large language models surpass human experts in predicting neuroscience results. Nat. Hum. Behav. 9, 305–315 (2025).
Crockett, M. & Messeri, L. Should large language models replace human participants? Preprint at PsyArXiv https://doi.org/10.31234/osf.io/4zdx9 (2024).
Abdurahman, S. et al. Perils and opportunities in using large language models in psychological research. PNAS Nexus 3, pgae245 (2024).
Argyle, L. P. et al. Out of one, many: using language models to simulate human samples. Polit. Anal. 31, 337–351 (2023).
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B. & Larson, J. M. Synthetic replacements for human survey data? The perils of large language models. Polit. Anal. 32, 401–416 (2024).
Atari, M., Xue, M. J., Park, P. S., Blasi, D. & Henrich, J. Which humans? Preprint at PsyArXiv https://doi.org/10.31234/osf.io/5b26t (2023).
Dillion, D., Tandon, N., Gu, Y. & Gray, K. Can AI language models replace human participants? Trends Cogn. Sci. 27, 597–600 (2023).
Filippas, A., Horton, J. J. & Manning, B. S. Large language models as simulated economic agents: what can we learn from homo silicus? In Proc. 25th ACM Conference on Economics and Computation 614–615 (ACM, 2024).
Binz, M. & Schulz, E. Using cognitive psychology to understand GPT-3. Proc. Natl Acad. Sci. USA 120, e2218523120 (2023).
Bai, H., Voelkel, J. G., Muldowney, S., Eichstaedt, J. C. & Willer, R. LLM-generated messages can persuade humans on policy issues. Nat. Commun. 16, 6037 (2025).
Hackenburg, K. et al. The levers of political persuasion with conversational artificial intelligence. Science 390, eaea3884 (2025).
Meta Fundamental AI Research Diplomacy Team (FAIR) et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science 378, 1067–1074 (2022).
Park, J. S. et al. Generative agents: interactive simulacra of human behavior. In Proc. 36th Annual ACM Symposium on User Interface Software and Technology (eds Follmer, S. et al.) 1–22 (ACM, 2023).
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A. & Kersting, K. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nat. Mach. Intell. 4, 258–268 (2022).
Yin, Y., Jia, N. & Wakslak, C. J. AI can help people feel heard, but an AI label diminishes this impact. Proc. Natl Acad. Sci. USA 121, e2319112121 (2024).
Lehr, S. A., Caliskan, A., Liyanage, S. & Banaji, M. R. ChatGPT as research scientist: probing GPT’s capabilities as a research librarian, research ethicist, data generator, and data predictor. Proc. Natl Acad. Sci. USA 121, e2404328121 (2024).
Time-sharing Experiments for the Social Sciences. TESS www.tessexperiments.org (2026).
Coppock, A., Leeper, T. J. & Mullinix, K. J. Generalizability of heterogeneous treatment effect estimates across samples. Proc. Natl Acad. Sci. USA 115, 12441–12446 (2018).
Chang, S., Kennedy, A., Leonard, A. & List, J. A. 12 Best Practices for Leveraging Generative AI in Experimental Research (National Bureau of Economic Research, 2024).
Broska, D., Howes, M. & van Loon, A. The mixed subjects design: treating large language models as potentially informative observations. Sociol. Methods Res. 54, 1074–1109 (2025).
Chu, J. Y. et al. Academics are more specific, and practitioners more sensitive, in forecasting interventions to strengthen democratic attitudes. Proc. Natl Acad. Sci. USA 121, e2307008121 (2024).
DellaVigna, S. & Linos, E. RCTs to scale: comprehensive evidence from two nudge units. Econometrica 90, 81–116 (2022).
Dreber, A. et al. Using prediction markets to estimate the reproducibility of scientific research. Proc. Natl Acad. Sci. USA 112, 15343–15347 (2015).
Milkman, K. L. et al. Megastudies improve the impact of applied behavioural science. Nature 600, 478–483 (2021).
Harding, J., D’Alessandro, W., Laskowski, N. G. & Long, R. AI language models cannot replace human research participants. AI Soc. 39, 2603–2605 (2024).
Messeri, L. & Crockett, M. J. Artificial intelligence and illusions of understanding in scientific research. Nature 627, 49–58 (2024).
Binz, M. et al. A foundation model to predict and capture human cognition. Nature 644, 1002–1009 (2025).
Petrov, N. B., Serapio-García, G. & Rentfrow, J. Limited ability of LLMs to simulate human psychological behaviours: a psychometric analysis. Preprint at https://arxiv.org/abs/2405.07248 (2024).
Tjuatja, L., Chen, V., Wu, T., Talwalkwar, A. & Neubig, G. Do LLMs exhibit human-like response biases? A case study in survey design. Trans. Assoc. Comput. Linguist. 12, 1011–1026 (2024).
Chen, Y., Hu, Y. & Lu, Y. Predicting field experiments with large language models. Preprint at https://arxiv.org/abs/2504.01167 (2025).
Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. In Proc. 11th International Conference on Learning Representations (poster) (ICLR, 2023).
Arora, S. et al. Ask me anything: a simple strategy for prompting language models. In Proc. 11th International Conference on Learning Representations (ICLR, 2023).
Voronov, A., Wolf, L. & Ryabinin, M. Mind your format: towards consistent evaluation of in-context learning improvements. In Findings of the Association for Computational Linguistics: ACL 2024 (eds Ku, L.-W. et al.) 6287–6310 (ACL, 2024).
Hou, B., O’Connor, J., Andreas, J., Chang, S. & Zhang, Y. Promptboosting: black-box text classification with ten forward passes. In Proc. International Conference on Machine Learning (eds Krause, A. et al.) 13309–13324 (PMLR, 2023).
Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).
Milkman, K. L. et al. Megastudy shows that reminders boost vaccination but adding free rides does not. Nature 631, 179–188 (2024).
Milkman, K. L. et al. A 680,000-person megastudy of nudges to encourage vaccination in pharmacies. Proc. Natl Acad. Sci. USA 119, e2115126119 (2022).
Zickfeld, J. H. et al. Effectiveness of ex ante honesty oaths in reducing dishonesty depends on content. Nat. Hum. Behav. 9, 169–187 (2025).
Broockman, D. E., Kalla, J. L., Caballero, C. & Easton, M. Political practitioners poorly predict which messages persuade the public. Proc. Natl Acad. Sci. USA 121, e2400076121 (2024).
DellaVigna, S. & Vivalt, E. Forecasting social science: evidence from 100 projects. National Bureau of Economic Research https://doi.org/10.3386/w34493 (2025).
Allen, J., Watts, D. J. & Rand, D. G. Quantifying the impact of misinformation and vaccine-skeptical content on Facebook. Science 384, eadk3451 (2024).
DellaVigna, S. & Pope, D. What motivates effort? Evidence and expert forecasts. Rev. Econ. Stud. 85, 1029–1069 (2018).
Vlasceanu, M. et al. Addressing climate change with behavioral science: a global intervention tournament in 63 countries. Sci. Adv. 10, eadj5778 (2024).
Tappin, B. M., Wittenberg, C., Hewitt, L. B., Berinsky, A. J. & Rand, D. G. Quantifying the potential persuasive returns to political microtargeting. Proc. Natl Acad. Sci. USA 120, e2216261120 (2023).
Voelkel, J. G. et al. Megastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity. Science 386, eadh4764 (2024).
Voelkel, J. G. et al. A registered report megastudy on the persuasiveness of the most-cited climate messages. Nat. Clim. Chang. 16, 214–225 (2026).
Saccardo, S. et al. Field testing the transferability of behavioural science knowledge on promoting vaccinations. Nat. Hum. Behav. 8, 878–890 (2024).
Mason, K. et al. A megastudy of behavioral interventions to increase voter registration ahead of the 2024 US presidential election. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/p6b2e_v3 (2025).
Goldwert, D. et al. A megastudy of behavioral interventions to catalyze public, political, and financial climate advocacy. PNAS Nexus 5, pgaf400 (2026).
Druckman, J. N. Experimental Thinking (Cambridge Univ. Press, 2022).
Freese, J. & Peterson, D. Replication in social science. Annu. Rev. Sociol. 43, 147–165 (2017).
Bai, X., Wang, A., Sucholutsky, I. & Griffiths, T. L. Explicitly unbiased large language models still form biased associations. Proc. Natl Acad. Sci. USA 122, e2416228122 (2025).
Wang, A., Morgenstern, J. & Dickerson, J. P. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nat. Mach. Intell. 7, 400–411 (2025).
Park, P. S., Schoenegger, P. & Zhu, C. Diminished diversity-of-thought in a standard large language model. Behav. Res. Methods 56, 5754–5770 (2024).
Frank, M. C. Openly accessible LLMs can help us to understand human cognition. Nat. Hum. Behav. 7, 1825–1827 (2023).
Wack, M., Ehrett, C., Linvill, D. & Warren, P. Generative propaganda: evidence of AI’s impact from a state-backed disinformation campaign. PNAS Nexus 4, pgaf083 (2025).
Response retrieved from ChatGPT-4 on 22 May. ChatGPT https://chatgpt.com/share/2a2b66d0-e9fd-421f-bedd-bb6f9354742a (2024).
Almaatouq, A. et al. Beyond playing 20 questions with nature: integrative experiment design in the social and behavioral sciences. Behav. Brain Sci. 47, e33 (2024).
Zhu, J.-Q., Xie, H., Arumugam, D., Wilson, R. C. & Griffiths, T. L. Using reinforcement learning to train large language models to explain human decisions. Preprint at https://arxiv.org/abs/2505.11614 (2025).
Tyner, A. H. et al. Investigating the replicability of the social and behavioural sciences. Nature 652, 143–150 (2026).
Cremer, D. D. & Kasparov, G. AI should augment human intelligence, not replace it. Harvard Business Review https://hbr.org/2021/03/ai-should-augment-human-intelligence-not-replace-it (2021).
Cummins, J. The threat of analytic flexibility in using large language models to simulate human data: a call to attention. Preprint at https://arxiv.org/abs/2509.13397 (2025).
Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447–453 (2019).
Coppock, A. Generalizing from survey experiments conducted on mechanical turk: a replication approach. Political Sci. Res. Methods 7, 613–628 (2019).
Mullinix, K. J., Leeper, T. J., Druckman, J. N. & Freese, J. The generalizability of survey experiments. J. Exp. Political Sci. 2, 109–138 (2015).
Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023).
Viechtbauer, W. metafor: Meta-Analysis Package for R v.4.8-0 (2015).
Rios, K., Roth, Z. C. & Coleman, T. J. III. The importance of scientists’ intellectual humility for communicating effectively across ideological and identity-based divides. Proc. Natl Acad. Sci. USA 122, e2400930121 (2025).