Rules and the per-rule readout

What it represents

One score can be powered by more than one model. Rules decide which record goes to which. Each rule carries criteria on your own fields and a bound model, and the list evaluates top down: the first rule a record matches wins, and the default rule at the bottom catches everything else. A record is never scored twice.

The per-rule readout is the check on that arrangement. It takes the model that is currently bound and asks how well it ranks inside each slice of your population, so you can see whether one model is genuinely serving all of them.

How it’s calculated

The readout reports an overall figure first: AUC on the held-out test records, with the record count and the number of conversions behind it. AUC is a ranking statistic, so it answers “does this model put converters above non-converters”, not “how many did it get right”.

Then it scores slices. Each slice shows its AUC, the share of the population it holds, and a weak flag when the model ranks that slice noticeably worse than it ranks overall. Slices with too few records or too few conversions are not scored at all · they show a short reason instead of a number, because a figure from a handful of conversions is noise. Some slices are suggested: they come from patterns in your data where no rule exists yet.

If your training data has changed since the active run, the readout reports itself unavailable rather than showing figures it cannot stand behind.

What it means for you

Worked example: the overall readout is AUC 0.84 across 20,000 test records holding 1,400 conversions. One suggested slice holds 18% of the population and reads 0.61, flagged weak. That is the case for a dedicated model: train one on that slice and it becomes a rule above the default, while everything else keeps the original model.

Aim for 500 or more conversions inside a rule before splitting it out. Below that, a dedicated model is fitting noise, and the honest move is to leave the slice with the shared model and revisit when you have more history.