CMU Foundation and Language Model Seminar
— 12:30pm
Location:
In Person and Virtual - ET
-
Newell-Simon 4305 and Zoom
Speaker:
JACOB SPRINGER
,
Ph.D. Student, Machine Learning Department, Carnegie Mellon University
https://sprin.xyz/
This talk will present over our recent work with the same title on improving text embeddings extracted from large language models. Many prior language model text embeddings are derived from masked language models (such as BERT, RoBERTa, T5, etc). However, autoregressive language models (GPT, LLaMA, etc) have recently been shown to be highly capable on many NLP tasks, and thus it is surprising that their use for embeddings has been limited.
In our work, we show that when used for text embeddings, autoregressive language models exhibit a failure mode: token embeddings derived from autoregressive language models cannot contain information from tokens that appear later in the input. To address this limitation, we propose a simple approach, “echo embeddings,” in which we repeat the input twice in context and extract embeddings from the second occurrence. In this talk, I will discuss the failure mode, method, and evaluations.
—
Jacob Springer is a PhD student in the machine learning department at CMU. His research focuses on the failure modes of deep learning algorithms, with recent interest in large language models and embeddings.
In Person and Zoom Participation. See announcement.
Event Website:
https://cmuflame.org/events.html