Artificial Intelligence Seminar

— 1:00pm

Location:
In Person and Virtual - ET - Newell-Simon 3305 and Zoom

Speaker:
ELAN ROSENFELD , Ph.D. Student, Machine Learning Department, Carnegie Mellon University
https://www.cs.cmu.edu/~elan/

Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization

There is a growing list of intriguing properties of neural network optimization, including specific patterns in their training dynamics (e.g. simplicity bias, edge of stability, grokking) and the unexplained effectiveness of various tools (e.g. batch normalization, SAM, Adam). Extensive study of these properties has so far yielded only a partial understanding of their origins—and their relation to one another is even less clear. What is it about gradient descent on neural networks that gives rise to these phenomena?

In this talk, I will present our recent experiments which offer a new perspective on many of these findings and suggest that they may have a shared underlying cause. Our investigation identifies and explores the significant influence of paired groups of outliers with what we call Opposing Signals: large magnitude features that dominate the network’s output throughout most of training and cause large gradients pointing in opposite directions.

Though our experiments shed some light on these outliers’ influence, we lack a complete understanding of their precise effect on network training dynamics. Instead, I’ll share our working hypothesis via a high-level explanation, and I’ll describe initial experiments which verify some of its qualitative predictions. We hope a deeper understanding of this phenomenon will enable future principled improvements to neural network optimization.

—

Elan Rosenfeld is a final year PhD student in CMU MLD advised by Profs. Andrej Risteski and Pradeep Ravikumar. His research focuses on principled approaches to understanding and improving robustness, representation learning, and generalization in deep learning.

The AI Seminar is sponsored by SambaNova Systems.

In Person and Zoom Participation. See announcement.

Event Website:
http://www.cs.cmu.edu/~aiseminar/

Add event to Google
Add event to iCal