Design of Experiments & Randomization

Design of experiments (DOE) is the engineering of evidence. Analysis can quantify uncertainty after data exist, but it cannot generally untangle a treatment from an unrecorded change that always occurred at the same time. A good design creates comparisons that differ in the factor of interest while balancing, blocking, or measuring alternative causes.

Start with the experimental unit #

The experimental unit is the smallest unit independently assigned to a treatment. If one oven receives a temperature setting and ten specimens are baked together, the oven run—not each specimen—is the independently randomized unit. Treating the ten specimens as ten independent replications is pseudoreplication: it makes uncertainty look artificially small because all ten share the same run conditions.

Before choosing a design, specify:

• the response and when/how it is measured;
• controllable factors and their feasible levels;
• nuisance factors such as batch, operator, day, machine, and position;
• the experimental unit and sampling unit;
• the smallest effect worth detecting;
• constraints, safety boundaries, and run cost.

Four mechanisms that create credible comparisons #

Randomization assigns treatments by a chance mechanism. It prevents the experimenter from systematically pairing one treatment with favorable units and turns unmeasured nuisance effects into random variation in expectation. Randomization is not a guarantee that every small experiment is balanced; it supplies the probability model for uncertainty and guards against systematic allocation bias.

Replication applies a treatment to independently randomized experimental units. It estimates run-to-run variability and increases precision. Repeated readings on one unit estimate measurement repeatability, not treatment replication.

Blocking groups units that are similar with respect to a known nuisance variable, then randomizes treatments within each block. Comparing treatments within a day, wafer, patient stratum, or material batch removes between-block variation from the treatment contrast.

Factorial variation changes several factors in the same design. Unlike one-factor-at-a-time experiments, it can reveal interactions and uses every run to estimate more than one effect.

flowchart TD
  U["Eligible experimental units"] --> B["Form blocks using pre-treatment nuisance information"]
  B --> R["Randomize treatments within each block"]
  R --> M["Measure responses using a blinded, fixed protocol"]
  M --> A["Analyze according to assignment and design"]

A $2^2$ factorial example #

Consider two factors, temperature $A$ and catalyst $B$, each at low and high levels. Mean yield is:

The average effect of raising $A$ is

$$\frac{60+72}{2}-\frac{50+54}{2}=14.$$

The average effect of raising $B$ is $8$. But the effect of $B$ is 4 at low $A$ and 12 at high $A$, so the difference of differences is

$$(72-60)-(54-50)=8.$$

That interaction is mechanistic information: the catalyst works much better at high temperature. A one-factor-at-a-time sequence might estimate either 4 or 12 depending on where the other factor was held and falsely treat it as universal.

Replicates are still needed to separate genuine interaction from noise. A linear model for coded levels $x_A,x_B\in\{-1,+1\}$ is

$$y=\beta_0+\beta_Ax_A+\beta_Bx_B+\beta_{AB}x_Ax_B+\varepsilon.$$

The product column $x_Ax_B$ changes sign in the interaction pattern, making the four effects orthogonal in a balanced complete design.

Blocking, pairing, and run order #

Suppose only four runs fit in a day and day-to-day ambient conditions shift the response. Making each day a block and placing all treatment combinations within every day allows treatment comparisons at a common daily baseline. Running all low settings first and high settings later instead confounds treatment with warm-up, drift, or learning.

Blocking should use information known before treatment. Grouping units by a post-treatment variable can open selection bias. Run order should be randomized within operational constraints; if a factor is hard to change, a split-plot design must represent that restricted randomization and its larger whole-plot error.

Designs answer different questions #

• A completely randomized design is appropriate when units are reasonably homogeneous and every treatment can be independently assigned.
• A randomized block design increases precision when a known nuisance variable creates groups of comparable units.
• A full factorial estimates all included interactions but grows exponentially with the number of factors.
• A fractional factorial screens many factors with fewer runs by deliberately aliasing effects. Its resolution states which effects cannot be separated.
• A response-surface design adds center and axial information to model curvature near a promising region; it is not a substitute for initial broad screening.

Common design failures #

Changing several conditions together creates confounding: no analysis can decide which caused the outcome. Optimizing after every few observations without a sequential design changes error rates. Dropping failed runs selectively can condition on treatment consequences. Measuring hundreds of responses and reporting the smallest p-value converts exploration into false confirmation. A nonrandom convenience sample also limits the population to which results generalize, even when treatment assignment within the sample is randomized.

Randomization supports internal causal validity; representative sampling supports external validity. They solve different problems.

A design review checklist #

1. Can every treatment contrast be expressed as a comparison of independently assigned units?
2. What nuisance trend could mimic each proposed effect?
3. Are replication and repeated measurement being kept distinct?
4. Which interactions are scientifically plausible and estimable?
5. Is the analysis model faithful to blocks and restricted randomization?
6. Are exclusions, stopping rules, and primary outcomes fixed before outcomes are inspected?

The cheapest time to remove confounding is before the first run. More sophisticated analysis cannot manufacture the missing comparison afterward.