Symbolic Music Representation & Generation

Symbolic music generation predicts an abstract performance or score—notes, onsets, durations, velocity, meter, controls, and instrument roles—rather than generating audio samples directly. This removes timbre and acoustics from the first modeling stage, making long musical structure more tractable and output more editable. It also makes the representation contract load-bearing: a model cannot learn timing, articulation, polyphony, or pedal behavior that preprocessing has discarded.

MIDI is an event protocol, not a complete musical meaning #

MIDI messages describe performance events and controls. A note number identifies pitch, while note-on/note-off, velocity, channels, controllers, program selection, pitch bend, and timing context describe how an instrument should respond. Standard MIDI files store timestamped events; MIDI 2.0 extends message resolution and packages MIDI 1.0 and 2.0 messages in Universal MIDI Packets.

None of these facts makes a MIDI file a canonical score. Enharmonic spelling, voices, fingering, phrase boundaries, intended harmony, and even note duration may be absent or inferred. Sustain pedal can make a note sound after its key-release event. Tempo changes map musical beats to wall-clock time. A learning pipeline must state which semantics it reconstructs and which it discards.

Three representation families expose different structure #

Piano-roll grids #

A piano roll quantizes time into steps and assigns one coordinate per pitch, often producing a binary or velocity-valued matrix $X_{t,p}$. It represents simultaneous notes naturally and supports convolutional or frame-wise models. Its costs are proportional to time resolution times pitch range, including long stretches with few events.

Quantization moves onsets and offsets. A grid fine enough for expressive timing becomes long and sparse; a coarse grid erases swing, tuplets, ornaments, and microtiming. Multiple channels are needed to distinguish onset, held state, velocity, instrument, and pedal semantics.

Event-token streams #

An event representation serializes note, timing, duration, velocity, bar, position, tempo, and control events into tokens. Sparse passages become compact, and autoregressive sequence models fit naturally. But simultaneous events need an ordering convention even when musical time does not order them.

MIDI-like tokenization often uses note-on, note-off, time-shift, and velocity events. REMI instead exposes metrical context with bar and within-bar position tokens plus note duration, making beat structure easier to model. Event vocabularies trade sequence length against vocabulary size and factorization: one compound token is short but combinatorial; separate attributes are reusable but lengthen the sequence.

Factored and multi-track representations #

Separate streams can preserve instrument roles, melody versus accompaniment, pitch versus duration, or inter-onset versus note-length timing. Multi-track models must capture both within-track dynamics and cross-track coordination. Independent generators can yield locally plausible tracks that clash harmonically or rhythmically; one shared generator can coordinate tracks but may blur role-specific behavior.

Build the representation as a reversible contract #

For every corpus, define:

• pitch range and treatment of drums or unpitched events;
• time unit, quantization grid, tolerance, and tempo-map behavior;
• note-on, note-off, overlap, repeated-note, and dangling-note rules;
• sustain and other controller handling;
• velocity binning and normalization;
• meter, pickup measures, tuplets, and bar segmentation;
• track/instrument mapping, merged channels, and transposition;
• maximum sequence length, crop boundaries, and whether notes may cross them;
• encoder–decoder round-trip tests on representative files.

A valid round trip need not reproduce byte-identical MIDI, but it must preserve the musical information the task claims to model. Test chords with simultaneous attacks, pedal-held notes, tempo changes, overlapping same-pitch notes, and a crop through an active note.

The task determines the causal information path #

“Generate music” hides distinct contracts:

• unconditional composition from a start token;
• continuation given a prefix;
• accompaniment conditioned on melody or chords;
• harmonization with fixed rhythm or voices;
• infilling between known surrounding passages;
• arrangement across instruments;
• performance rendering from a score.

The attention mask, state reset, conditioning channels, and evaluation set must match the contract. A bidirectional infilling model is not a causal live accompanist. A model trained on complete bars may fail when asked to continue from a partial beat. Teacher-forced next-token likelihood does not exercise errors that accumulate during free-running generation.

Architecture follows representation, not fashion #

RNNs and LSTMs maintain a streaming state with bounded per-step memory, but long-range gradients and state retention remain difficult. Transformers make distant token interaction direct and train positions in parallel, while dense attention grows quadratically with sequence length. Relative timing and position matter because musical similarity often depends on interval and metric relationship rather than absolute token index.

Grid models can generate a chord in one step; token models may need several ordered events for the same chord. Conversely, event models skip empty time efficiently. Compare architectures only after controlling representation, context length, parameter count, sampling policy, and corpus split; otherwise the experiment confounds the model with the encoding.

Dataset provenance is part of the model #

Randomly splitting short windows leaks pieces, performers, arrangements, and near-duplicate MIDI files across train and test sets. Split by composition or source before chopping into windows. Deduplicate transpositions and alternate encodings when they would make a held-out item a trivial variant of training material.

Record licenses, attribution requirements, performers, instruments, capture process, alignment precision, and preprocessing version. MAESTRO is valuable because aligned MIDI and audio connect symbolic performance to waveform rendering, but its concert-piano repertoire is a specific distribution, not “music” in general.

Generation is a controlled sampling process #

Autoregressive generation samples a token distribution repeatedly. Temperature, top-$k$/top-$p$ truncation, repetition penalties, hard grammar masks, and rejection rules change the output distribution. They are part of the evaluated system and must not be tuned by listening to test outputs without disclosure.

Hard constraints are appropriate for protocol validity—balanced note-on/off, legal time movement, instrument ranges—when they implement the representation contract. Musical rules such as “always stay in key” are creative constraints, not validity checks; imposing them changes the task and may suppress intended chromaticism.

Evaluate music at multiple levels #

Token negative log-likelihood measures predictive probability under one representation. Syntax-validity checks catch malformed events. Pitch-class, duration, polyphony, rhythm, repetition, and inter-track statistics diagnose distributional differences. None alone establishes musical quality or usefulness.

Use blinded listening or editing studies matched to the application, and report fidelity, diversity/coverage, memorization, and long-range structure separately. Render all compared systems with the same sound engine and loudness policy so timbre does not masquerade as compositional quality. The broader protocol belongs in Evaluating Generative Models.