Title:Challenges and advances in the training of Energy-Based Models: theoretical insights on non-equilibrium sampling and overfitting

Abstract

In the context of unsupervised learning and generative models in artificial intelligence, energy-based models (EBMs) provide a simple way to describe arbitrary complex datasets, by encoding their empirical distribution into a Boltzmann probability law with a given energy function. EBMs can be used to generate new data statistically equivalent to the training set, and to extract microscopic information, i.e. effective interactions among data components. In this talk, I will review the basic foundations of EBMs from a statistical physics point of view, highlighting their connection with maximum-entropy models. I will then focus on typical challenges/issues related to the training, which is usually performed through an iterative maximum-likelihood procedure. The first problem is related to the sampling protocol, which is required to estimate one of the terms appearing in the log-likelihood’s gradient: in this context, I will discuss a recent work that theoretically describes recent proposed training strategies based on non-persistent short Markov Chain MonteCarlo (MCMC) runs as an efficient way to generate good-quality samples. The second part focuses on the impact of finite amount of data in the model’s reconstruction, a problem related to the concept of overfitting in machine learning: I will discuss recent theoretical results that analyze the training dynamics of simple EBMs (focusing on the Gaussian model), and show how i) overfitting emerges from an interplay of different learning timescales associated to different principal components of the data and ii) how different data-correction protocols can be used to mitigate the impact of overfitting for EBMs that employ low-order sufficient statistics of the data.

Thursday, 6 February 2025, 12:30 h
Conference Room. Edificio I+D