Target leakage, and why we flag it

Target leakage is the failure mode that makes a model look brilliant in training and useless on live pipeline.

What it represents

A leaking field is one that only exists, or only gets filled in, because the outcome already happened. A contract date. A won-reason picklist. A stage field that moves after the deal is done.

The model learns that field, ranks on it, and reports a near-perfect AUC. Then you point it at open records, where the field is empty for everything, and the ranking falls apart. The training number was never a forecast, it was a description of the past written in the past tense.

You meet the screen in two places, and they are not saying the same thing.

Before training, an orange Leakage risk badge sits on the field’s row in Available fields, with the reason in its tooltip, and a warning appears in the diagnostics Recommendations block. Both are read off your whole assembled dataset, not off the fields you selected.

After training, an orange Possible target leakage detected banner appears above your results, naming fields and offering Draft without flagged fields. That one is scoped: it only names fields the model actually trained on.

How it’s calculated

Three independent screens run at refresh, and any one of them can flag a field.

Presence separation asks whether the field being filled in at all lines up with the outcome. It is type-agnostic, so it catches date fields that a value-based test skips. Information Value asks how sharply a field’s values separate conversions from non-conversions, and a value of 1.0 or higher is flagged · a threshold deliberately far above what genuine predictors reach on CRM data. Provenance asks where the column came from, and flags fields belonging to the object that records the conversion itself, which no threshold can catch.

At training time the findings are sorted by whether the model used the field. Findings on trained fields become the banner. Findings on other columns in the dataset demote to a dataset note in the ordinary warnings block, and are never fed to the redraft button.

What to do when a field is flagged

1. Read the reason, not just the badge

   Hover the Leakage risk badge in Available fields. The tooltip carries the screen's own reason: that the field is only populated on conversions, that it separates the two classes implausibly sharply, or that it comes from the object recording the conversion.

   The reason tells you which question to ask next. A provenance flag is usually decisive. A separation flag deserves your own judgement about the field.

2. Check whether the flagged field is in your selection

   This is the step people skip. The pre-training screen runs over the whole assembled dataset, so the recommendation can name fields you never picked, and it will still say training on them inflates accuracy.

   Open Selected Fields & Health on the Data step and look for the name. If it is not there, there is nothing to fix: leave it unselected and carry on. The warning is telling you the column exists in your data, not that your model is using it.

3. If it is selected, ask the one useful question

   At the moment I want a score, is this field already filled in?

   If the answer is no, it cannot help you and it has to come out. If the answer is genuinely yes for open records too, a legitimately powerful field can trip this screen, and you are the one who knows your data. Treat the flag as a screen, not a verdict.

4. Remove it and re-run diagnostics

   Click the field in Available fields to untick it, or use the remove control on its row in Selected Fields & Health. Then go back to Review and Train and press Re-run Diagnostics.

   Do this before training rather than after. Removing the field costs one refresh here · finding out from the banner costs a full run.

5. If the banner appears after a run, take the button

   Draft without flagged fields drops the named fields from the model's current selection and saves that as a draft. It does not start a run, so you can edit further before training. If the flagged fields turn out not to be in the selection at all, it says so and changes nothing.

   The banner is dismissible per run, and it clears on its own once a completed run comes back with no leakage findings. Dismiss it after you have made the call, not instead of making it.

6. Do not expect stability to catch the rest

   Feature stability does not detect leakage. A leaking field leaks consistently across every fold and therefore scores as beautifully stable.

   The screens plus your own knowledge of when each field gets written are the whole defence. Nothing downstream will catch what they miss.

What it means for you

A lower honest number is the better one. The screen exists to stop you shipping a model whose ranking will not survive contact with open records, and the cost of listening to it is always a smaller AUC on the model card.

The one thing to keep straight is which warning you are looking at. Before training, the flag describes your dataset. After training, it describes your model. The first is a suggestion about what not to pick. The second is a defect in what you built.

Check your understanding

A model reports AUC 0.98 and the banner names close_reason. That field is populated on conversions and empty everywhere else, so the model has learned to check whether the outcome was already recorded. Draft without it and retrain: AUC lands at 0.76. The 0.76 is the real number, and it is the one that will hold up when you score open records.

Read that back against the two warnings. Because it appeared as the banner rather than as a picker badge, close_reason was in the training fields, which is exactly why the redraft button had something to remove. Had the same field shown up as a recommendation on the Review and Train step while sitting unselected, the correct action would have been to do nothing at all.