From Bias to Bots with Human-in-the-Loop (HITL): Assessing Transparency and Enhancing Trust in Large Language Models

This talk explores the potential and challenges of using large language models (LLMs) in social science research, with a focus on ensuring transparency and trust in AI applications. Following HITL principles, our investigation centers around two key concerns: bias in AI-generated responses and the efficacy of AI in detecting inauthentic behavior. In the first study, we delve into the presence of social desirability bias in LLMs. By comparing human and LLM-simulated responses to a set of survey questions, we assess the extent to which this bias, inherent in human respondents, is mirrored in LLM outputs due to their training data. This evaluation is crucial for understanding the reliability and integrity of LLMs as tools for capturing public opinion. The second study examines the efficacy of temporal features in detecting bot accounts on social media. While most existing bot detection methods rely on semantic, meta, and social interaction information, our research highlights the potential of temporal behavior patterns in identifying bots. This study underscores the importance of integrating more-than-textual information in social science research, demonstrating how LLMs can enhance bot detection methodologies. By addressing these overarching concerns, our research aims to contribute to the development of more transparent and trustworthy AI systems, ensuring that LLMs can be effectively and ethically utilized in social science research.


Speaker: Professor Winson PENG
Date: 13 June 2024 (Wednesday)
Time: 4:00pm – 5:30pm
PosterClick here



Winson Peng (Ph.D. in Communication, City University of Hong Kong) is a Professor in the Department of Communication, Michigan State University (MSU). His research interests include computational social science, health communication, political communication, and mobile analytics. He has published more than 60 articles in highly ranked SSCI and SCI journals. He is currently the associate editor of the Journal of Communication and is the incoming Editor-in-Chief of Human Communication Research. He was the inaugural chair of Computational Methods division of International Communication Association (ICA).