Training Restricted Boltzmann Machines on Word Observations
The paper identifies the linear-in-vocabulary cost of Gibbs updates for K-way RBM visible units and replaces exact softmax block sampling with Metropolis–Hastings transitions whose proposal can be sampled in constant time.
Categorical observations fit an RBM naturally, but a full Gibbs draw from a -way softmax requires scoring and normalizing all categories. For word vocabularies in the tens or hundreds of thousands, that transition dominates training.
Durable contribution #
The paper keeps the one-hot categorical energy but substitutes Metropolis–Hastings transitions for exact visible-block Gibbs draws. With an efficiently sampled proposal distribution and cached or sparse computations, the per-transition cost can avoid dependence on vocabulary size. This does not make the draw independent; it constructs a Markov kernel targeting the required conditional.
The distinction is important: a cheaper transition is useful only if acceptance and mixing are adequate. The experiments compare proposals and numbers of transitions, measure movement through word states, and then evaluate learned representations on downstream tasks.
Limits #
The method targets a historical RBM/NLP bottleneck. Its proposal quality depends on the data distribution, and a small fixed number of M-H moves can remain strongly correlated with the previous state. Downstream gains do not by themselves show that the chain closely approximates equilibrium. GTK uses the paper to explain the computational consequence of large categorical groups, not as a current NLP recommendation.
Visual audit #
All eight physical pages were rendered and read. The graphical-model diagram, acceptance/mixing plots, nearest-neighbour table, and downstream result tables were inspected. No visual is retained: the algorithmic dependency on and the exact-vs-M-H sampling distinction are clearer in prose and equations, while the result plots are benchmark-specific.