Digital Modulation: ASK, FSK, PSK & QAM

Digital modulation maps groups of bits onto a finite set of waveforms. Because every practical scheme builds those waveforms from two quadrature carriers, $\cos(2\pi f_c t)$ and $-\sin(2\pi f_c t)$, the whole set collapses into a picture: a constellation of points in a two-dimensional I/Q plane. ASK moves points along a radius, PSK moves them around a circle, QAM uses the plane properly, and FSK is the one family that refuses to fit in two dimensions at all.

Once you have that picture, one number does almost all the work. The receiver adds Gaussian noise to the transmitted point and picks the nearest legal one; the probability it picks wrong is set by the minimum Euclidean distance $d_{\min}$ between constellation points measured against the noise standard deviation. Everything else in this entry — how many bits a symbol carries, how the bits are labelled, how the pulses are shaped, how much bandwidth results, and how hard the power amplifier has to work — is downstream of that single geometric fact.

The constellation, and the one bound that matters #

A symbol is a complex number $x_i = a_i + jb_i$ transmitted as

$$s_i(t) = a_i\,\varphi_1(t) + b_i\,\varphi_2(t), \qquad \varphi_1(t)=\sqrt{\tfrac{2}{T}}\,p(t)\cos 2\pi f_c t,\;\; \varphi_2(t)=-\sqrt{\tfrac{2}{T}}\,p(t)\sin 2\pi f_c t$$

with $p(t)$ the pulse shape. A matched filter followed by a sampler recovers $x_i$ plus a complex Gaussian noise sample whose real and imaginary parts each have variance $\sigma^2 = N_0/2$. Detection is then a nearest-neighbour search in the plane — nothing about carriers or radio survives into the decision.

Cioffi's nearest-neighbour union bound is the general form of the error expression, and it is worth internalising because every scheme-specific formula below is a special case of it:

$$P_e \le N_e \cdot Q\!\left(\frac{d_{\min}}{2\sigma}\right), \qquad Q(x) = \frac{1}{\sqrt{2\pi}}\int_x^{\infty} e^{-t^2/2}\,dt$$

$N_e$ is the average number of decision-region boundaries surrounding a symbol. For square $M$-QAM, $N_e = 4\left(1 - 1/\sqrt{M}\right)$ — interior points have four neighbours, edge and corner points fewer.

flowchart LR
  B["Bits"] --> G["Gray mapper<br/>k bits to 1 symbol"]
  G --> D["Differential<br/>encoder (optional)"]
  D --> P["RRC pulse<br/>shaping filter"]
  P --> U["I/Q upconvert<br/>to fc"]
  U --> PA["Power<br/>amplifier"]
  PA --> CH(("AWGN +<br/>fading"))
  CH --> LNA["LNA, downconvert"]
  LNA --> MF["RRC matched<br/>filter"]
  MF --> S["Symbol-rate<br/>sampler"]
  S --> DET["Nearest-neighbour<br/>detector"]
  DET --> DM["De-map<br/>to bits"]

The two RRC blocks are the same filter split in half; the pair multiplies out to a raised cosine, which is the zero-ISI shape. That is the subject of a later section, and it is the reason the sampler in the middle sees clean symbols rather than a smear.

The four families #

The dimension row is the load-bearing one. PSK spends two dimensions but only uses the perimeter of the available energy budget, so beyond 8PSK its points get crowded on the circle while the interior sits empty — which is exactly why nothing above 8PSK is deployed and why 16-QAM beats 16-PSK by about 4 dB. FSK goes the other way: it buys distance by spending bandwidth, one orthogonal dimension per tone, which is a good trade only when spectrum is free and power is not.

Error probability, exactly #

Write $\gamma_b = E_b/N_0$ (energy per information bit) and $k = \log_2 M$ bits per symbol, so $E_s = k E_b$. Comparing at fixed $E_b/N_0$ rather than fixed $E_s/N_0$ is what makes the comparison honest — otherwise high-order schemes get their extra bits for free.

BPSK and QPSK. Two antipodal points at $\pm\sqrt{E_b}$; QPSK is two independent BPSK channels on I and Q. Both are exact:

$$P_b^{\text{BPSK}} = P_b^{\text{QPSK}} = Q\!\left(\sqrt{2\gamma_b}\right), \qquad P_s^{\text{QPSK}} = 2Q\!\left(\sqrt{2\gamma_b}\right) - Q^2\!\left(\sqrt{2\gamma_b}\right)$$

M-PSK, $M \ge 8$. Points at radius $\sqrt{E_s}$, angular spacing $2\pi/M$, so $d_{\min} = 2\sqrt{E_s}\sin(\pi/M)$. Two dominant neighbours:

$$P_s \approx 2\,Q\!\left(\sqrt{2k\gamma_b}\;\sin\frac{\pi}{M}\right), \qquad P_b \approx \frac{P_s}{k} \;\text{(Gray)}$$

Square M-QAM ($M = 4, 16, 64, 256, \ldots$). Two independent $\sqrt{M}$-PAM constellations. With $\gamma_s = E_s/N_0$ the symbol error rate is exact:

$$P_s = 1 - \left[1 - 2\left(1 - \tfrac{1}{\sqrt{M}}\right)Q\!\left(\sqrt{\frac{3\gamma_s}{M-1}}\right)\right]^{2}$$

and the standard Gray-coded bit-error approximation, which is the one you actually use, is

$$\boxed{\;P_b \approx \frac{4}{k}\left(1 - \frac{1}{\sqrt{M}}\right) Q\!\left(\sqrt{\frac{3k}{M-1}\,\gamma_b}\right)\;}$$

The $3k/(M-1)$ factor is the whole story of higher-order QAM: the argument of $Q$ shrinks roughly as $\sqrt{k/M}$, so each doubling of $M$ (one more bit per symbol) costs on the order of 3 dB once you are past 16-QAM.

Binary ASK and FSK. With $E_b$ the average bit energy, coherent on-off keying and coherent orthogonal binary FSK have the same expression, 3 dB worse than BPSK, because their signal points are orthogonal rather than antipodal:

$$P_b^{\text{coh OOK}} = P_b^{\text{coh FSK}} = Q\!\left(\sqrt{\gamma_b}\right), \qquad P_b^{\text{noncoh FSK}} = \tfrac{1}{2}e^{-\gamma_b/2}$$

What it costs, in dB #

Required $E_b/N_0$ for an uncoded BER of $10^{-6}$ on AWGN, solved from the expressions above:

Three things to read off it. 16-QAM beats 16-PSK by 4.0 dB at the same spectral efficiency, which is why nobody deploys 16-PSK. 8PSK and 16-QAM sit within half a dB of each other, so 8PSK survives only where a near-constant envelope matters (satellite, EDGE) and 16-QAM wins everywhere else because it carries a fourth bit for 0.45 dB. And past 16-QAM each extra bit per symbol costs roughly 2.4 dB (16→64→256→1024→4096: 4.38, 4.74, 4.99, 5.17 dB per two bits), so the last constellation you can close a link on is a very sharp function of margin.

Gray coding: why BER ≈ SER / k #

The formulas above quietly assumed $P_b \approx P_s / k$. That is only true with the right bit labelling.

At the SNRs that matter, essentially every symbol error is a slip to a nearest neighbour — errors to farther points are suppressed by an additional exponential. Gray coding labels the constellation so that any two nearest neighbours differ in exactly one bit position. A nearest-neighbour symbol error then produces exactly one bit error out of $k$, so

$$P_b \approx \frac{1}{k}P_s$$

Without it, a single symbol slip can flip several bits. Square QAM makes Gray coding trivial: Gray-code the $\sqrt{M}$-PAM levels on each axis independently and concatenate the two labels — which is exactly what the 16-QAM panel in the figure shows, and why every horizontal or vertical neighbour there differs in one bit. 3GPP does the same for EDGE 8PSK: TS 45.004 Table 1 is a Gray map onto the eight phases.

ASK and FSK, and the continuous-phase branch #

ASK in its pure form barely exists in radio — it is one-dimensional, so it wastes half the plane, and its envelope swings to zero, which is the worst case for a power amplifier. It survives where intensity is the only thing you can modulate: on-off keying in optical links, and multi-level PAM in wireline (PAM-4 on 100G-per-lane Ethernet and on DDR5's command bus). The moment you have a phase reference, ASK becomes the I-axis of a QAM constellation and the family name stops being useful.

FSK transmits one of $M$ tones. It is the odd one out because its signal points are orthogonal, living in $M$ dimensions rather than 2, so $d_{\min}$ does not shrink as $M$ grows — but bandwidth does. Coherently detected tones are orthogonal when spaced by $\Delta f = 1/(2T)$; noncoherent detection needs $\Delta f = 1/T$. The $1/(2T)$ figure is the "minimum shift" that names the next scheme.

MSK and GMSK #

Plain FSK switches tones abruptly, and a phase discontinuity has an infinite-bandwidth spectrum. Continuous-phase FSK instead integrates the data into the phase, so the waveform never jumps. The modulation index $h = 2\,\Delta f\,T$ sets how much phase accumulates per symbol; $h = 1/2$ gives exactly $\pm\pi/2$ per bit and tone spacing $1/(2T)$ — minimum shift keying, the smallest spacing that keeps the two tones orthogonal.

GMSK pre-filters the data with a Gaussian filter before the phase integrator. That trades a deliberate, bounded amount of ISI for a much faster spectral roll-off. GSM specifies it exactly (3GPP TS 45.004 §2.4–2.5): the modulating impulses excite $g(t) = h(t) * \mathrm{rect}(t/T)$ where $h(t)$ is a Gaussian with $BT = 0.3$, the modulation index is $h = 1/2$ ("maximum phase change in radians is $\pi/2$ per data interval"), and the symbol rate is $1/T = 1625/6$ ksymb/s $\approx$ 270.833 ksymb/s in a 200 kHz channel.

Bluetooth Basic Rate is the same family at a different operating point: GFSK, $BT = 0.5$, modulation index between 0.28 and 0.35, 1 Msym/s — deliberately below MSK's $h = 0.5$, which narrows the spectrum further at the cost of distance.

Differential encoding and the phase ambiguity #

A coherent PSK receiver recovers the carrier from the received signal itself, typically by a nonlinearity that removes the modulation. Squaring recovers the carrier of a BPSK signal — but also destroys the sign, so the recovered reference is right or inverted with equal probability. For M-PSK the fourth-power or M-th-power loop leaves an $M$-fold ambiguity. Nothing in the received waveform distinguishes the branches: the constellation has rotational symmetry, and that symmetry is exactly the problem.

Two standard fixes, and they are not the same thing:

• Differential encoding with coherent detection. Encode data in the change of phase, then detect coherently and difference the decisions. A constant unknown rotation cancels. Cost: an isolated symbol error corrupts two consecutive differences, so the BER roughly doubles — $P_b = 2p(1-p)$ with $p = Q(\sqrt{2\gamma_b})$, which at BER $10^{-6}$ is a penalty of only 0.25 dB. GSM does this on GMSK: TS 45.004 §2.3 differentially encodes each data value, $\hat{d}_i = d_i \oplus d_{i-1}$, before the modulator.
• Differentially coherent detection (DPSK). Skip carrier recovery entirely and use the previous symbol as the phase reference. No carrier loop, no ambiguity, and immunity to slow phase drift. Cost is larger because the reference itself is noisy: $P_b^{\text{DBPSK}} = \tfrac{1}{2}e^{-\gamma_b}$, a 0.65 dB penalty at BER $10^{-6}$, growing to roughly 2–3 dB for differentially detected QPSK.

Bluetooth EDR takes the second route on both of its higher modulations: "π/4 rotated differential encoded quaternary phase shift keying (π/4-DQPSK)" at 2 Mb/s and "differential encoded 8-ary phase shift keying (8DPSK)" at 3 Mb/s.

Pulse shaping: Nyquist, raised cosine, and the bandwidth you actually get #

A rectangular symbol pulse has a $\mathrm{sinc}$ spectrum whose sidelobes decay at only 6 dB/octave — unusable next to a neighbouring channel. Band-limiting it, though, spreads each symbol in time so that it leaks into its neighbours' sampling instants. That leakage is intersymbol interference, and Nyquist's 1928 result says exactly how to band-limit without it.

His criterion in modern form: sampling the received pulse at $t = nT$ picks out one symbol cleanly if and only if the folded spectrum is flat,

$$\sum_{n=-\infty}^{\infty} H\!\left(f + \frac{n}{T}\right) = T \quad \text{for all } f \qquad \Longleftrightarrow \qquad h(nT) = \begin{cases}1 & n = 0\\ 0 & n \neq 0\end{cases}$$

Nyquist stated both halves. His minimum-bandwidth result — "the minimum band width required for unambiguous interpretation is substantially equal, numerically, to the speed of signaling and is substantially independent of the number of current values employed" — gives the Nyquist bandwidth: $R_s/2$ at baseband, $R_s$ as an RF (double-sideband) bandwidth. And he showed that adding any component with odd symmetry about the band edge changes nothing at the sampling instants: "shape factors symmetrical about [$f_N$] ... do not contribute anything to the current value at the middle of the signal elements". That vestigial-symmetry observation is precisely what the raised cosine implements.

Raised cosine and its root #

$$H_{RC}(f) = \begin{cases} T, & \lvert f\rvert \le \dfrac{1-\beta}{2T} \\[6pt] \dfrac{T}{2}\left[1 + \cos\!\left(\dfrac{\pi T}{\beta}\left(\lvert f\rvert - \dfrac{1-\beta}{2T}\right)\right)\right], & \dfrac{1-\beta}{2T} < \lvert f\rvert \le \dfrac{1+\beta}{2T} \\[6pt] 0, & \text{otherwise} \end{cases}$$

$$h_{RC}(t) = \mathrm{sinc}\!\left(\frac{t}{T}\right)\cdot\frac{\cos(\pi\beta t/T)}{1 - 4\beta^2 t^2/T^2}$$

The extra $\cos/(1-\cdot)$ factor makes the tails decay as $1/t^3$ instead of $1/t$, which is what makes the pulse usable with real timing recovery.

The filter is then split between transmitter and receiver as two root-raised-cosine (RRC) filters, $\sqrt{H_{RC}}$ each. This is not a convenience: the receive RRC is the matched filter for the transmit pulse, so it maximises SNR at the sampling instant, and the cascade of the two is the raised cosine, so ISI is zero at the sampler. You get both optimality conditions from one factorisation. DVB-S2 writes it out as the normative transmit filter, in the form

$$\lvert H(f)\rvert = \sqrt{\tfrac{1}{2}\left[1 + \sin\!\left(\frac{\pi}{2\alpha f_N}\left(f_N - \lvert f\rvert\right)\right)\right]}, \qquad f_N = \frac{R_s}{2}$$

which is algebraically identical to $\sqrt{H_{RC}}$ over the roll-off band.

The bandwidth that results is the practical payoff:

$$B = (1 + \beta)\,R_s \quad \text{(RF, double-sideband)}$$

DVB-S2 says this in operational terms: "$R_S(1+\alpha)$ corresponds to the theoretical total signal bandwidth after the modulator ... typical $BW/R_S$ ratios are $1+\alpha = 1{,}35$ and $1{,}20$."

Spectral efficiency and the Shannon ceiling #

Spectral efficiency is bits per second per hertz:

$$\eta = \frac{R_b}{B} = \frac{k \cdot R_s}{(1+\beta)R_s} = \frac{k}{1+\beta} \quad \text{bit/s/Hz (uncoded)}$$

With a rate-$R$ channel code it becomes $\eta = kR/(1+\beta)$ — the coding entry's rate penalty and this entry's excess bandwidth are two multiplicative discounts on the same number.

The ceiling is Shannon's. Substituting $P/N = \eta\,E_b/N_0$ into $\eta = \log_2(1 + P/N)$ gives the minimum $E_b/N_0$ at which any code could work at that efficiency:

$$\frac{E_b}{N_0} \ge \frac{2^{\eta}-1}{\eta}$$

The unconstrained bound's own floor, $\lim_{\eta \to 0}(2^\eta - 1)/\eta = \ln 2 \approx -1.59$ dB, is where power-limited links live; the top of the table is where bandwidth-limited ones do.

Peak-to-average power ratio #

Noise sets how many points you can pack. The power amplifier sets whether you can transmit them. PAPR is the ratio of the instantaneous peak envelope power to the average:

$$\text{PAPR} = \frac{\max_t \lvert v(t)\rvert^2}{\overline{\lvert v(t)\rvert^2}}$$

An amplifier is efficient near saturation and linear only well below it. A signal with high PAPR must be backed off so its peaks stay in the linear region, which means the average transmit power — the thing your link budget cares about — drops by roughly the back-off, and DC-to-RF efficiency drops with it. Clip instead and you get spectral regrowth into the adjacent channel and a raised error floor.

Two distinct numbers get called PAPR, and conflating them is the usual mistake:

Column 2 is $\max\lVert x_i\rVert^2 / \overline{\lVert x_i\rVert^2}$ over the constellation points — for square $M$-QAM, $\left[2(\sqrt{M}-1)^2\right] / \left[\tfrac{2}{3}(M-1)\right]$, which is 1.8 for 16-QAM, matching Cioffi's stated figure. Columns 3–4 are the 99.99th-percentile envelope power of the filtered continuous-time waveform, computed for this entry by simulation (200 000 random symbols, 8× oversampling, 25-symbol RRC span).

Two deployed consequences worth carrying:

• DVB-S2's constellation choice is a PAPR decision, not only an SNR one. The spec gives the unfiltered peak-to-average as 0 dB for QPSK and 8PSK, 1.05–1.11 dB for 16APSK and 1.97–2.12 dB for 32APSK — and says 16APSK/32APSK "are inherently more sensitive to non-linear distortions and would require quasi-linear transponders (i.e. with larger Output-Back-Off, OBO)". The APSK rings exist instead of square QAM precisely to keep that number low on a nonlinear transponder.
• 5G NR carries π/2-BPSK for coverage-limited uplink. TS 38.211 lists it first among the modulation mappers. Its ~2 dB waveform PAPR against QPSK's ~4–5 dB is directly recoverable as PA back-off, i.e. as transmit power, i.e. as cell-edge range.

OFDM is far worse than any of this: summing $N$ independently modulated subcarriers gives a near-Gaussian envelope whose PAPR grows with $N$ — the theoretical peak is $10\log_{10} N$ dB, and deployed systems sit far below it. The commonly quoted 8–12 dB operating range is folklore here: it is not sourced in this entry. That, not modulation order, is why OFDM systems obsess over back-off and clipping.

Adaptive modulation and coding #

Nothing above chooses one constellation, because a real channel does not hold still. Fading, rain, and distance move the received SNR over tens of dB, and a fixed modulation must be designed for the worst case — wasting the entire margin whenever conditions are better than that. Adaptive modulation and coding measures the channel and picks the highest $(k, R)$ pair that still closes.

5G NR is the cleanest example because the standard publishes the ladder as a table. TS 38.214 MCS index table 2 for the downlink shared channel runs 28 entries from QPSK at target code rate 120/1024 up to 256-QAM at 948/1024:

Read the boundaries. At each modulation change the ladder steps down in code rate — from QPSK at 602/1024 to 16-QAM at 378/1024, from 16-QAM at 658 to 64-QAM at 466 — so that spectral efficiency increases monotonically and continuously across the switch. That is the design: the two knobs are tuned jointly, never independently, and the transition points are where a higher constellation with more coding beats a lower one with less.

DVB-S2 does the same over a satellite link, adapting "on a frame-by-frame basis" using channel-quality reports fed back from each receiving terminal. Its 28 MODCODs span spectral efficiencies from 0.49 to 4.45 bits per symbol at required $E_s/N_0$ from −2.35 dB to +16.05 dB — an 18.4 dB operating range on one waveform. The claimed payoff is large: "ACM systems promise satellite capacity gains of up to 100 % − 200 %" relative to designing for the worst case.

Spectral efficiency and $E_s/N_0$ are from TS 13 of EN 302 307 (64 800-bit FECFRAME, PER $10^{-7}$, AWGN, 50 LDPC iterations); the last column applies the standard's own note, $E_b/N_0 = E_s/N_0 - 10\log_{10}\eta$. Compare 8PSK 3/4 at 4.43 dB against uncoded 8PSK's 13.95 dB in the earlier table: 9.5 dB of coding gain, net of the rate penalty, is what the LDPC/BCH concatenation buys.

At the top end the constellation eventually stops being limited by thermal noise at all. 802.11be's 4096-QAM carries 12 bits per symbol and "increases the nominal rate by 20 % using the same coding rate", but the SNR demand "is high, which is often achieved through strict error vector magnitude (EVM) requirement and beamformed transmission" — that is, the binding constraint has moved from the channel to the transmitter's own phase noise, PA linearity and I/Q imbalance. A required $E_b/N_0$ of 33.7 dB is not something you close by adding power.

Choosing one #

Two habits are worth carrying out of this. Compare at fixed $E_b/N_0$, never fixed $E_s/N_0$ — the latter silently gifts high-order schemes their extra bits. And state the roll-off with every spectral-efficiency figure: $k$ bits per symbol is not $k$ bit/s/Hz, and the 20–35 % you lose to excess bandwidth is the same order as the difference between two adjacent constellations.