15-499/899 Special Topics: Social Agents Guest Lecture October 24, 2024 2:00pm — 3:20pm Location: In Person - Newell-Simon 3002 Speaker: MARTEN SAPP, Assistant Professor, Language Technologies Institute, Carnegie Mellon University http://maartensap.com/ Promise and Pitfalls of Safe and Social LLM Agents LLMs are increasingly playing agentic and social roles by interacting with users and tools. In this talk, I will discuss the shortcomings and pitfalls of LLM-based agents. First, I will introduce Sotopia, a new framework for social simulation with LLM agents, and Sotopia-Eval, a multi-dimensional framework for quantitatively evaluating social skills of LLM in interactions. Showing in Sotopia that LLM agents are still far from humans' interaction skills, I will then discuss the crucial role of information asymmetry, showing that LLMs cannot properly deal with the information asymmetry that is present in real interactions. Finally, I will introduce HAICosystem, a new framework for simulating and evaluating the safety of LLM agents in user-AI-tool interactions, showcasing the many safety issues that could arise from increasingly autonomous and tool-using agents. I will conclude with some future directions towards safe and social LLM agents. Course / Faculty Host: Hirokazu Shirado Add event to Google Add event to iCal
