Pallavi Koppol

Interactive Machine Learning from Humans: Knowledge Sharing via Mutual Feedback Degree Type: Ph.D. in Computer Science
Advisor(s): Reid Simmons, Henny Admoni
Graduated: December 2023

Abstract:

People regularly interact with human-in-the-loop learning (HiLL) agents that attempt to adapt to their priorities, tastes, and preferences. Examples of such systems include web search engines, movie recommender systems, text prediction, and even large-language model based chat applications. To be adaptive, these HiLL systems must first learn an accurate model of an individual's behavior and preferences.

The ability to learn such a model depends on the quality of the information the HiLL system is able to elicit from the people with whom it interacts, and how well it is then able to leverage that information. Typically, this information is generated via a loop where an agent or model takes actions or makes suggestions that a person responds to with some feedback, and that response is then used to train future behavior. Henceforth, we will refer to this query-feedback pair as an interaction.

We note that the informativeness of a learning interaction is limited by how fully it empowers a person to share their knowledge. We demonstrate that it is possible to improve a person's teaching performance by providing them with (1) more appropriate modalities for sharing feedback (i.e. interaction types) and (2) insight into the context of the learner they are instructing. Our approach therefore moves towards a model that prioritizes a human teacher's ability to provide informative feedback.

To do this, we first formalize the space of interactions that can be used to learn from human feedback and present four interaction archetypes: Showing, Categorizing, Sorting, and Evaluating. Then, we analyze the effects that these different interaction types may have on learning outcomes via both direct and indirect influences on collected training data. We build on this to contribute a learning approach that enables an algorithmic learner to learn from multiple interaction types based on which would be the most immediately informative. Finally, we develop and evaluate an interaction type-based approach towards bridging the gap between an algorithmic learner and a human teacher's mental model of that learner.

Thesis Committee:
Reid Simmons (Co-Chair)
Henny Admoni (Co-Chair)
Rayid Ghani
Gonzalo Ramos (Microsoft Research)

Srinivasan Seshan, Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Keywords:
Human-in-the-Loop Learning, Learning From Human Feedback, Interactive Machine Learning, Learning from Demonstration, Preference Learning, Active Learning