Restricted Boltzmann Machines (RBMs) are a type of generative stochastic artificial neural network that can learn a probability distribution over its set of inputs. They were initially invented by Geoffrey Hinton and Terry Sejnowski in 1985, under the name Harmonium. RBMs have been used for dimensionality reduction, classification, regression, collaborative filtering, feature learning, and topic modeling. They are particularly known for their role in deep learning, where they have been used as building blocks for more complex models such as Deep Belief Networks (DBNs).
Structure and Functioning:
An RBM consists of two layers: a visible layer that represents the input data and a hidden layer that captures features or patterns in the data. These layers are fully connected to each other, but there are no connections within a layer, a restriction that simplifies the learning algorithm. The "restricted" part of their name comes from this limitation, distinguishing them from general Boltzmann machines, which allow intra-layer connections and are much harder to train.
Learning in RBMs:
RBMs learn through a process called contrastive divergence, a method introduced by Geoffrey Hinton to efficiently approximate the likelihood gradient. The goal is to adjust the weights between the visible and hidden layers so that the model can accurately reconstruct the input data after passing it through a hidden layer of features. Learning involves iteratively updating these weights to reduce the difference between the original input and its reconstruction.
Key Features:
-
Energy-Based Model: RBMs are energy-based models. They associate an energy level to each configuration of the variables (visible and hidden units). The network learns the weights in such a way that low energy is assigned to observed (or desirable) configurations, making them more probable.
-
Binary or Continuous Units: The original RBMs were defined with binary units, meaning each neuron could be in one of two states (e.g., 0 or 1). However, variations of RBMs can handle continuous data, such as Gaussian RBMs, where the visible units are assumed to have a Gaussian distribution.
Applications:
- Feature Learning: RBMs can learn to automatically discover and represent patterns in the input data, making them useful for feature extraction in unsupervised learning tasks.
- Collaborative Filtering: They have been applied to collaborative filtering to make personalized recommendations by learning the preferences of users.
- Pretraining for Deep Neural Networks: Perhaps one of the most impactful uses of RBMs has been in the pretraining of deep neural networks. Before the advent of more effective training techniques, RBMs were used to initialize the weights of deep networks in a layer-wise fashion, improving the training process and the final performance of the network.
Despite the rise of other techniques like autoencoders and generative adversarial networks (GANs) for many of these tasks, RBMs remain an important concept in the history and development of neural networks and deep learning.