Energy-Based Models & Boltzmann Machines

An energy-based model (EBM) describes which configurations are compatible by assigning each configuration $x$ a scalar energy $E_\theta(x)$. Low energy means preferred; high energy means disfavored. This formulation can express dependencies without requiring a normalized output layer, but a generative EBM still has to normalize a distribution or sample from one—and those operations are usually where the computational difficulty lives.

Energy is a score before it is a probability #

For a temperature absorbed into the energy scale, an unconditional probabilistic EBM defines

$$p_\theta(x)=\frac{\exp[-E_\theta(x)]}{Z(\theta)}, \qquad Z(\theta)=\int \exp[-E_\theta(x)]\,dx.$$

For discrete $x$, the integral becomes a sum. The partition function $Z$ ensures probabilities total one. Adding the same constant to every energy changes $Z$ by the compensating factor and leaves $p(x)$ unchanged, so absolute energy has no meaning; only differences matter.

A conditional model similarly uses

$$p_\theta(y\mid x)=\frac{\exp[-E_\theta(x,y)]}{\sum_{y'}\exp[-E_\theta(x,y')]}.$$

If the candidate set for $y$ is small, this normalization can be tractable. If $x$ is a high-dimensional image, waveform, or binary vector, enumerating all configurations is impossible. That freedom–tractability exchange is the central EBM trade-off.

Maximum likelihood has a positive and a negative phase #

Differentiating the negative log-likelihood of one observation gives

$$\nabla_\theta[-\log p_\theta(x)] =\nabla_\theta E_\theta(x) -\mathbb E_{x'\sim p_\theta}\!\left[\nabla_\theta E_\theta(x')\right].$$

The first term is data-dependent: lower the energy around observed configurations. The second is model-dependent: raise energy where the model currently places probability. Without the negative phase, every configuration could be driven downward together and the model would learn no normalized distribution.

The expectation under $p_\theta$ is commonly estimated with Markov-chain Monte Carlo (MCMC), such as Langevin dynamics for continuous variables or Gibbs sampling for conditionally tractable discrete models. Training then contains a nested approximation: parameters change while a finite sampler tries to track the changing distribution.

Slow mixing causes a characteristic failure. Chains remain near their initial modes, the negative phase misses other low-energy regions, and the model can create spurious basins that training never penalizes. More sampler steps reduce one source of bias but increase cost; they do not guarantee mixing in a multimodal landscape.

Restricted Boltzmann machines make Gibbs updates simple #

A binary restricted Boltzmann machine (RBM) has visible variables $v$, hidden variables $h$, biases $a,b$, and no within-layer edges:

$$E(v,h)=-a^\top v-b^\top h-v^\top Wh.$$

The bipartite restriction makes units conditionally independent within a layer:

$$p(h_j=1\mid v)=\sigma\!\left(b_j+W_{:j}^\top v\right), \qquad p(v_i=1\mid h)=\sigma\!\left(a_i+W_{i:}h\right).$$

All hidden units can therefore be sampled in parallel given $v$, followed by all visible units given $h$. Alternating these two block updates is Gibbs sampling. Marginalizing the hidden layer yields a visible free energy

$$F(v)=-a^\top v-\sum_j\log\left(1+\exp[b_j+W_{:j}^\top v]\right),$$

with $p(v)\propto\exp[-F(v)]$. Free energy removes explicit hidden-state enumeration, but it does not make the partition function tractable.

Categorical visible groups produce softmax conditionals #

A categorical variable with $K$ possible values cannot be represented as $K$ independent Bernoulli units: exactly one state must be active. Encode group $i$ by a one-hot vector $v_{i1},\ldots,v_{iK}$ with $\sum_k v_{ik}=1$, and define its state logit given $h$ as

$$\alpha_{ik}(h)=c_{ik}+\sum_j h_jW_{ijk}.$$

Holding $h$ fixed, every energy term outside group $i$ cancels from the conditional ratio. Normalizing only over the $K$ valid states gives

$$p(v_{ik}=1\mid h) =\frac{\exp[\alpha_{ik}(h)]} {\sum_{\ell=1}^{K}\exp[\alpha_{i\ell}(h)]}.$$

If visible groups are conditionally independent given $h$, the full conditional is a product of per-group softmax distributions:

$$p(V\mid h)=\prod_i \frac{\exp\!\left[\sum_k v_{ik}\alpha_{ik}(h)\right]} {\sum_{\ell=1}^{K}\exp[\alpha_{i\ell}(h)]}.$$

The denominator must remain inside the product. Replacing the product of group normalizers with one sum across groups would couple unrelated categorical variables and would not normalize over the Cartesian product of their states.

This construction was used for discrete movie ratings and word observations. Its cost is sampling or scoring all $K$ states for each group. When $K$ is a very large vocabulary, the ordinary Gibbs update is linear in $K$; alternative transition kernels such as carefully designed Metropolis–Hastings updates trade exact block sampling for a cheaper Markov transition whose mixing must then be assessed.

CD and PCD are sampler approximations, not objective names #

Contrastive divergence with $k$ steps (CD-$k$) initializes a Gibbs chain at a training example, runs $k$ alternating updates, and uses the reconstruction endpoint for the negative phase. Short chains are cheap and often learn useful features, but the update is generally a biased approximation to the likelihood gradient. Reconstruction error is not log-likelihood and can improve while the global model distribution remains poor.

Persistent contrastive divergence (PCD) retains chains between parameter updates. If parameters change slowly relative to mixing, persistent particles can stay closer to the current model distribution than chains repeatedly restarted at data. PCD still fails when chains mix slowly, learning rates move the landscape too far, or the particle population misses modes.

The word contrastive here concerns a difference between data and model distributions. It is not the same mechanism as modern representation-learning losses that contrast augmented examples.

Deep belief and deep Boltzmann networks are not the same model #

A deep belief network (DBN) stacks RBMs for greedy layer-wise pretraining, but its final directed/undirected semantics differ from a fully undirected deep Boltzmann machine (DBM). A DBM couples multiple hidden layers in one energy function and requires approximate inference over them. Historically, these models helped establish unsupervised pretraining when deep feed-forward networks were difficult to optimize.

Modern normalization, initialization, residual connections, large labeled datasets, autoregressive models, variational methods, GANs, and diffusion/score models changed the default toolset. RBMs and DBNs remain important for understanding latent-variable EBMs, MCMC-biased learning, and positive/negative phases; they are not the default architecture merely because the data are binary or sequential.

Modern EBMs are broader than Boltzmann machines #

A neural network can directly parameterize $E_\theta(x)$ or $E_\theta(x,y)$ without RBM conditional independence. Maximum-likelihood training may use MCMC; score matching learns derivatives of log density and eliminates the normalizing constant from the objective; noise-contrastive estimation distinguishes data from a known noise distribution. These methods impose different evidence requirements and failure modes.

Score-based diffusion models are closely related to learning gradients of log density across noise levels, but “EBM” and “diffusion model” are not interchangeable labels. An EBM specifies an energy or unnormalized density; a diffusion model specifies a time-indexed corruption and reverse-generation process. Their mathematics intersects through score estimation.

A diagnostic checklist #

Before trusting an EBM experiment, record:

1. the exact energy and whether it represents a joint, conditional, or marginal model;
2. whether $Z$ is tractable, estimated, or avoided by another objective;
3. the negative-sample initialization, transition kernel, step count, temperature, and persistent state;
4. mixing diagnostics across independent chains and modes, not only one attractive trajectory;
5. held-out likelihood estimation with its uncertainty when likelihood is claimed;
6. sample fidelity, coverage, and memorization evidence separately from training loss;
7. whether generation uses the same sampler and number of steps evaluated during validation.

An EBM is useful precisely because it separates “what configurations should be compatible?” from a fixed normalized architecture. That separation is also the trap: the learned energy, the sampler, and the evaluation protocol are three coupled parts of one model, not interchangeable implementation details.