Master Class: Generative Stochastic Networks: How to Get Rid of Approximate Inference over Latent Variables

SpeakerYoshua Bengio
AffiliationDepartment of Computer Science and Operations Research, University of Montreal
DateWednesday, 23 Oct 2013
Time13:00 - 14:00
LocationGalton Lecture Theatre, 1-19 Torrington Place
Event seriesMaster Class: Yoshua Bengio (21 Oct - 24 Oct 2013)

Recent work showed that denoising auto-encoders can be interpreted as generative models. We generalize these results to arbitrary parametrizations that learn to reconstruct their input and where noise is injected, not just in input, but also in intermediate computations. We show that under reasonable assumptions (the parametrization is rich enough to provide a consistent estimator, and it prevents the learner from just copying its input in output and producing a dirac output distribution), such models are consistent estimators of the data generating distributions, and that they define the estimated distribution through a Markov chain that consists at each step in re-injecting sampled reconstructions as a sequence of inputs into the unfolded computational graph. As a consequence, one can define deep architectures similar to deep Boltzmann machines in that units are stochastic, that the model can learn to generate a distribution similar to its training distribution, that it can easily handle missing inputs, but without the troubling problem of intractable partition function and intractable inference as stumbling blocks for both training and using these models. In particular, we argue that if the underlying latent variables of a graphical model form a highly multimodal posterior (given the input), none of the currently known training methods can appropriately deal with this multimodality (when the number modes is much greater than the number of MCMC samples one is willing to perform, and when the structure of the posterior cannot be easily approximated by some tractable variational approximation). In contrast, the proposed models can simply be trained by back-propagating the reconstruction error (seen as log-likelihood of reconstruction) into the parameters, benefiting from the power and ease of training recently demonstrated for deep supervised networks with dropout noise.

The video recording of this masterclass is available here:
Oct 23: Generative Stochastic Networks: How to Get Rid of Approximate Inference over Latent Variables

This event is funded by DeepMind Technologies.

iCalendar csml_id_150.ics