This talk explores the potential and challenges of using large language models (LLMs) in social science research, with a focus on ensuring transparency and trust in AI applications. Following HITL principles, our investigation centers around two key concerns: bias in AI-generated responses and the efficacy of AI in detecting inauthentic behavior. In the first study, we delve into the presence of social desirability bias in LLMs. By comparing human and LLM-simulated responses to a set of survey questions, we assess the extent to which this bias, inherent in human respondents, is mirrored in LLM outputs due to their training data. This evaluation is crucial for understanding the reliability and integrity of LLMs as tools for capturing public opinion. The second study examines the efficacy of temporal features in detecting bot accounts on social media. While most existing bot detection methods rely on semantic, meta, and social interaction information, our research highlights the potential of temporal behavior patterns in identifying bots. This study underscores the importance of integrating more-than-textual information in social science research, demonstrating how LLMs can enhance bot detection methodologies. By addressing these overarching concerns, our research aims to contribute to the development of more transparent and trustworthy AI systems, ensuring that LLMs can be effectively and ethically utilized in social science research.

Speaker: Professor Winson PENG
Date: 13 June 2024 (Wednesday)
Time: 4:00pm – 5:30pm
Poster: Click here