Generative Models for Discovering Sparse Distributed Representations

Geoffrey E. Hinton and Zoubin Ghahramani

We describe a hierarchical, generative model that can be viewed as a non-linear generalization of factor analysis and can be implemented in a neural network. The model uses bottom-up, top-down and lateral connections to perform Bayesian perceptual inference correctly. Once perceptual inference has been performed the connection strengths can be updated using a very simple learning rule that only requires locally available information. We demonstrate that the network learns to extract sparse, distributed, hierarchical representations.

To appear in Phil. Trans. Royal Society B postscript.

Back to Zoubin Ghahramani's Home Page,