Features, Encoding & Scaling

A model never sees a customer, waveform, document, or machine. It sees a representation: numbers produced by a measurement and preprocessing pipeline. That representation defines which observations count as nearby, which differences are preserved, and which information is unavailable. Feature work is therefore model design, not clerical preparation.

Begin with meaning and availability #

For every feature record its unit, measurement process, timestamp semantics, valid range, missingness mechanism, update cadence, source owner, and availability at prediction time. “Account balance” could mean current ledger balance, end-of-day posted balance, or a retrospectively corrected value; these are different inputs.

Measurement scales constrain valid operations:

• Nominal categories support equality, not ordering: country code, device family.
• Ordinal values have order but not necessarily equal spacing: severity levels.
• Interval scales have meaningful differences but arbitrary zero: Celsius temperature.
• Ratio scales support ratios because zero is meaningful: elapsed time or mass.

Encoding an ordinal category as 1, 2, 3 asserts both order and equal step size unless the model treats it specially. Encoding an unordered category that way invents a geometry.

Scaling changes the learner's objective #

Standardization transforms feature $j$ using training-set mean $\mu_j$ and standard deviation $s_j$:

$$z_{ij}=\frac{x_{ij}-\mu_j}{s_j}.$$

This is essential when distances, dot products, gradients, or coefficient penalties combine coordinates. Suppose two customers differ by 10 years in age and 1,000 currency units in income. Raw Euclidean distance is dominated by income because $1000^2\gg10^2$, regardless of predictive relevance. Standardizing makes a one-standard-deviation change comparable across coordinates.

Scaling also changes regularization: an $L_2$ penalty on coefficients penalizes a feature measured in meters differently from the same feature measured in millimeters unless features are scaled. Trees usually do not need monotone scaling because splits depend on order, but transformed missingness, precision, and feature interactions can still matter.

Robust scaling by medians and quantiles can resist outliers; log transforms can make multiplicative variation more linear. Transformations must follow domain semantics. Taking a log of signed residuals or adding an arbitrary constant solely to make it possible can destroy meaning.

Categorical variables #

One-hot encoding gives each observed level its own indicator and avoids false ordering. It can become high-dimensional, and unseen levels require an explicit policy. Rare-level grouping reduces variance but may erase a rare high-risk category.

Target encoding replaces a category with an estimate such as its mean label. It is powerful because it imports outcome information—and dangerous for the same reason. Encoding a row using a statistic that includes its own label leaks the answer. Use cross-fitting inside the training data, smooth small categories toward the global mean, and fit the entire encoder separately within every validation fold.

Identifiers deserve suspicion. A user ID may let the model memorize repeated users and look excellent under a random row split while failing for new users. Keep it only when entity-specific prediction is intentional and evaluation represents that use.

Missing values are observations about a process #

Missingness can mean “not measured,” “not applicable,” “below detection limit,” “sensor failed,” or “value withheld.” A single null token collapses these causes. Imputation supplies a computational value; it does not recover the unknown truth.

Median imputation can be reasonable for a baseline, often with a missingness indicator. But fitting the median on all data leaks validation distribution information. If missingness itself depends on the future outcome or a previous decision policy, a model can learn a brittle operational shortcut.

Time windows and aggregation #

Features such as “transactions in the last seven days” need a cutoff anchored at prediction time. Include only events whose availability timestamp precedes the cutoff, not events backfilled later with earlier event times. For a daily model, distinguish a complete previous day from a partially observed current day.

Aggregation changes information. A mean hides volatility; a total confounds rate with exposure duration. Useful summaries often include count, exposure, recency, trend, quantiles, and missing coverage. But hundreds of correlated windows increase selection opportunities and maintenance burden.

Learned representations #

Embeddings map sparse or structured inputs into dense vectors learned from a task. Convolutional features encode local patterns; sequence encoders encode context. These representations can transfer useful invariances, but they also inherit the source corpus, objective, version, and preprocessing. Treat an embedding model as a versioned upstream dependency with its own drift and failure modes.

For text, tokenization defines the units the model can express. For images, resizing and color handling can remove diagnostic detail. For signals, sampling rate and windowing determine which frequencies and transients remain. A sophisticated learner cannot recover information discarded before it sees the input.

A representation audit #

Ask these questions with raw examples:

1. Could this value exist at the prediction cutoff?
2. Does its numeric encoding preserve only valid relationships?
3. Does a unit change or category reorder change predictions unexpectedly?
4. Is missingness stable across sites and policies?
5. Are train and serving computations byte-for-byte equivalent?
6. Can a simple feature identify the label source, split, site, or period?
7. What information has aggregation, clipping, or compression destroyed?

A feature is good when its semantics are stable, its causal availability is honest, and it exposes useful distinctions without importing the target. More columns are not automatically more information.