Framing a Learning Problem: Supervised, Unsupervised & What Not to Learn

Machine learning is useful when a system must make repeated decisions from patterns that are too variable to encode as stable rules, and when representative examples or feedback exist. “Use a classifier” is not a problem statement. The model is one replaceable component inside a chain from observation to action and consequence.

Frame the decision before the prediction #

A complete framing names:

• Decision: what action changes when the output changes?
• Unit: what entity receives one prediction—a transaction, patient visit, machine-hour, or account?
• Prediction time: exactly when must the output be available?
• Inputs: what is observable at that time and legally/operationally usable?
• Target: what future or latent quantity should be predicted, over what horizon?
• Population: to which entities, locations, periods, and policies must it generalize?
• Loss: what are the costs of false positives, false negatives, delay, abstention, and data collection?
• Feedback: how and when will outcomes become observable after deployment?

flowchart TD
  O["Observation at decision time"] --> X["Feature construction"]
  X --> P["Prediction or score"]
  P --> A["Policy chooses action"]
  A --> Y["Outcome under that action"]
  Y --> L["Delayed labels and monitoring"]
  L --> X

The arrow from action to outcome matters. Once a model changes who is inspected, treated, ranked, or approved, observed labels describe outcomes under the deployed policy—not the outcome that would have occurred without it.

Supervision describes the signal available #

Supervised learning observes pairs $(x_i,y_i)$ and estimates a rule that predicts $y$ from $x$. Classification predicts discrete outcomes; regression predicts quantities. The word “supervised” says labels exist, not that they are correct. Labels may be delayed, selectively observed, noisy, or produced by a prior policy.

Unsupervised learning has observations $x_i$ but no task label. It can compress, group, estimate density, or flag unusual points. The output is a structure induced by representation, metric, objective, and hyperparameters. A cluster is not automatically a real-world type.

Semi-supervised methods combine many unlabeled observations with fewer labels. Self-supervised training constructs a prediction target from the data itself, such as a masked token, to learn a reusable representation. Reinforcement learning learns actions from sequential rewards where actions alter later states and data. These are different evidence structures, not a ladder of sophistication.

Worked framing: predictive maintenance #

“Predict failures” leaves every important choice open. A usable specification could be:

At 00:00 each day, score every operating pump using telemetry available through 23:55. Predict whether it will enter a mechanically confirmed failure state within seven days. If the score exceeds a cost-derived threshold, schedule inspection during the next maintenance window.

The unit is a pump-day, but rows from the same pump are dependent. The horizon is seven days. A work-order code entered after diagnosis is unavailable at scoring time and would be leakage. If preventive inspection averts failure, “no failure observed” is not a natural negative label; it is an outcome under intervention. Evaluation should include inspection burden, failures prevented, lead time, and pump-level/temporal splits—not only row accuracy.

A simple hours-since-service rule is the baseline. If it produces equal operational value with clearer failure behavior, ML adds maintenance cost without adding value.

Prediction, explanation, and causation are different #

A predictor estimates an association such as $P(Y\mid X=x)$ under a data-generating policy. It does not generally answer what would happen if $X$ were intervened on. A variable can predict hospital mortality because it records treatment intensity; reducing treatment would not improve survival.

Similarly, feature attribution describes how a fitted predictor uses inputs, not why the world produced an outcome. If the intended decision is causal—choose the treatment that improves an individual's outcome—the target is a contrast between potential outcomes, and the data/design must support it.

Define the target without smuggling in the answer #

Targets often fail in four ways:

1. Proxy mismatch: clicks are used for satisfaction, or arrest for offending. Optimizing the proxy changes behavior and can make it less faithful.
2. Selective labels: loan repayment is observed only for approved applicants; disease confirmation only for tested patients.
3. Temporal leakage: the label or its consequence appears in an input recorded after the prediction cutoff.
4. Ambiguous horizon: “churn” could mean no purchase for 30, 90, or 365 days, creating different tasks.

Write the label-generation query as part of the specification. Audit a sample from raw event history in both positive and negative classes. Measure label delay and revision.

What not to learn #

Do not learn a rule when a deterministic invariant, safety interlock, accounting identity, or legal constraint is known. Keep hard constraints in a policy layer. Do not learn from identifiers that memorize entities without justified generalization. Do not automate an action when errors are unobservable, unappealable, or irreversible and no safe review path exists.

Also stop when the decision is rare and stable enough for a checklist, when representative data cannot be obtained, when the target changes faster than retraining and validation can follow, or when the supposed gain is smaller than measurement uncertainty and operational cost.

Acceptance criteria for the framing #

Before model selection, the team should be able to state a baseline, evaluation split, primary metric, cost/constraint metrics, threshold owner, abstention or human-review path, monitoring plan, and retraining trigger. If two engineers could build different labels from the same prose, the task is not framed yet.

The durable artifact is the decision contract. Algorithms can then compete fairly inside it.