How to evaluate a predictive lead scoring model
Which measures answer which question, how each one misleads when read alone, and the traps that make a model test well and fail in production. Demonstrated end to end on a public dataset you can download and rerun yourself.
Two things to know before you read on. The demonstration uses UCI Bank Marketing, which is a consumer bank telemarketing dataset, not B2B. There are no companies in it, no opportunities and no sales cycles. And this page publishes no performance results, on purpose: ax1om does not put model outcome figures in public copy while every account is still on sample data. What is published is the method, which is the part that transfers to your data anyway.
What to measure, and what each measure hides
A scoring model produces an ordering. These are the standard ways to ask how good that ordering is, and each has a characteristic way of flattering a model.
| Measure | What it answers | How to read it | How it misleads |
|---|---|---|---|
| Base rate | What share of records convert overall. | Not a score metric at all, but nothing else means anything without it. Every other number on this list is only interpretable as a ratio against the base rate, which is why it has to be quoted alongside them. | A precision figure quoted with no base rate beside it is unreadable. The same value can be excellent or worthless depending on what it is compared against. |
| AUC-ROC | Given one converted and one non-converted record at random, how often does the model rank the converted one higher. | A threshold-free measure of ranking quality across the whole list. Useful because it does not depend on where you decide to cut. | It weights the whole ranking equally, including the bottom, which no team ever works. A model can improve its AUC by sorting the hopeless half more tidily and deliver nothing to the people using it. |
| Precision at top k | Of the records the model ranks in the top k percent, what share convert. | The closest thing to what a rep experiences, because the top of the list is the part that gets worked. Choose k to match how much of the list your team actually gets through. | It rises automatically as k shrinks, so it is only comparable at a fixed k, and it says nothing about the conversions sitting outside the cut. |
| Lift at top k | Precision at top k divided by the base rate. | The same finding as precision, restated as a multiple so it can be compared across datasets with different base rates. Cumulative by convention: it covers everything down to the cut, not a single band. | A large multiple on a very low base rate can still be operationally small. Lift is a ratio, and a ratio hides the absolute numbers underneath it. |
| Capture at top k | What share of all conversions fall inside the top k percent. | The coverage question, and the one that decides whether a threshold is safe to route on. It tells you what you give up by not working the rest of the list. | Easily confused with precision. Precision asks how clean the top slice is, capture asks how much of the total it contains. They move differently, and answering one does not answer the other. |
Precision, lift and capture at the same k are one finding, not three. Precision at top k divided by the base rate is lift at top k. Precision times k divided by the base rate is capture at top k. Quote one and you have implied the others, so a report that presents all three as independent evidence is padding a single result. Quote none of them with the base rate and none of them can be read at all.
Choose k before you look at the results, and choose it to match operational reality: the share of the list your team genuinely works in a cycle. Choosing k afterwards, once you can see where the model looks best, is a way of reporting the model's luckiest cut.
Seven ways an evaluation lies to you
In rough order of how often they are the reason a model looked good in a notebook and disappointed in a pipeline.
Target leakage
A field that is only knowable after the outcome will predict the outcome almost perfectly, and the model will look extraordinary right up until it meets a live record where that field is empty. This is the most common reason a scoring model tests well and fails in production.
Check: For every field, ask whether its value existed at the moment you would want to score the record. If it did not, it cannot be an input.
Evaluating on the data you tuned on
Early stopping, threshold selection and hyperparameter search all consume information from whatever partition they observe. Report a metric on that same partition and it is optimistic, because the choice was made to suit it.
Check: Split three ways rather than two: train, validate, test. Tune on validation, report on a test set nothing has looked at.
A single split at a single seed
One holdout gives one number with no error bar. Rerun the same configuration at a different seed and it moves. Without repeats there is no way to tell a real improvement from resampling noise.
Check: Repeat across seeds or folds and report the spread, not just the midpoint. Treat a difference smaller than that spread as no difference.
Ignoring the base rate
Conversion data is imbalanced, often heavily. Plain accuracy is meaningless on an imbalanced problem, because predicting that nobody converts scores well on it.
Check: Never use accuracy. Use ranking measures, and quote the base rate next to every one of them.
Reading the wrong slice of the chart
Cumulative and per-band views of the same lift data look similar and say different things. A cumulative reading covers everything down to the cut; a per-band reading covers one slice only. Mixing them produces confident nonsense.
Check: Confirm which convention a chart uses before quoting it. If a band and a running total are plotted the same way, that is a defect in the chart.
Benchmarking on data unlike yours
A public dataset validates a method. It does not forecast a result on your pipeline. Different base rate, different fields, different decision, different answer.
Check: Treat any published benchmark as evidence that a technique works, then measure the technique on your own data before you route anything on it.
Measuring ranking and claiming revenue
Every measure here scores a list ordering. None of them scores an outcome. The ranking only becomes revenue if the team works the list in ranked order.
Check: Treat model quality and operational adoption as two separate questions, and do not let a good number on the first imply the second.
The dataset used for the demonstration
Everything in this table is a property of a public file that anyone can download and verify. None of it describes anything a model did.
| Name | Bank Marketing |
|---|---|
| Source | UCI Machine Learning Repository, dataset 222 |
| Attributed to | S. Moro, P. Cortez and P. Rita |
| Domain | Retail bank direct marketing, Portugal. Consumer telemarketing, not B2B |
| Collected | 2008 to 2013 |
| Unit of observation | One contact attempt with one consumer |
| Label | Whether the contact subscribed to a term deposit |
| Variant used | bank-full.csv · 45,211 rows × 17 columns, 16 inputs and the label |
| Published positive-class rate |
About 11.7% of rows in bank-full.csv carry a
positive label. A documented property of the public file, not a
result
|
The dataset ships in more than one variant, and the counts differ. The identification here rests on two published properties of the file used: its 45,211 × 17 shape, and its positive-class rate of about 11.7%. The other commonly cited variant holds about 41,188 rows at a rate of about 11.27%, which is how the two are told apart.
Why it suits a method demonstration
It is public, free, widely used, and structurally analogous to a lead list in the way that matters here: a large set of records, a low positive rate, and a mix of static attributes and interaction history. That is enough to exercise every measure and every trap on this page, and anyone can rerun it.
Why it is not your pipeline
It is consumer telemarketing from a single Portuguese bank, collected over a decade ago. No companies, no buying committee, no opportunity, no sales cycle, no CRM. Nothing measured on it transfers to B2B pipeline performance, which is why this page publishes the method rather than a number.
How the data was partitioned
The full file goes in unfiltered and is split 80/20, stratified on the label, at a fixed seed of 42. That yields 36,168 rows to train on and 9,043 held out, and the two sum back to the 45,211 rows of the source file. The test side rounds up, which is the standard convention: ceil(0.2 × 45,211) = 9,043.
Stratification is doing real work on data this imbalanced. An unstratified split can hand one side materially fewer conversions purely by chance, and on a low base rate that shifts every measure computed downstream. Fixing the seed is what makes the partition reproducible rather than merely described.
A correction to how ax1om previously described this. Earlier ax1om copy called the benchmark "a standard public ML set of 36,168 records", which conflated the training partition with the dataset. The dataset is 45,211 records. 36,168 is what remained after holding a fifth of it back.
Model configuration
Inputs to the run, not results from it. Where something was not recorded, it is marked as such rather than filled in.
| Element | What is known | Status |
|---|---|---|
| Model family | LightGBM, gradient-boosted decision trees. This is ax1om’s standard classifier, not a variant chosen for the demonstration. | documented |
| Task | Binary classification. The model outputs a conversion likelihood per record, which ax1om surfaces as a 0-100 ranking. The score orders records against each other; it is not a percentage chance that a given record converts, and per-record factors do not sum to it. | documented |
| Split | Stratified 80/20 at a fixed seed of 42. Stratification keeps the positive-class share the same on both sides, which matters on imbalanced data because an unstratified split can hand one side materially fewer conversions by chance. | documented |
| Hyperparameters | 300 trees · learning rate 0.05 · max depth 6 · 31 leaves · min 20 samples per leaf · row and column subsampling at 0.8 · balanced class weights · seed 42 · early stopping after 50 rounds without improvement. These are the documented training defaults, and this path takes them with no overrides. | documented |
| Class balancing | Balanced class weights, which is the default. The minority class is reweighted during fitting rather than resampled, so no rows are duplicated or discarded. | documented |
| Cross-validation | A 5-fold stratified cross-validation runs in the pipeline, but it computes Feature Stability Scores rather than headline performance. Performance is measured on the single holdout, with no variance estimate attached. | documented |
| Encoding | Categorical fields are one-hot encoded and numeric fields pass through unchanged. | documented |
| Feature count | Unreconciled. Applying the documented encoding rules to all 16 input fields of this file yields 51 columns. The run recorded 53. The field selection behind that difference is not published, so it is left as a discrepancy rather than explained away. | not documented |
| Leakage removal | Not documented for this run. See the leakage section, which is the most consequential open item on this page. | not documented |
The early-stopping problem in this configuration
Early stopping is evaluated on the same 9,043 holdout records that performance is then reported on, rather than on a third partition. So the stopping point is chosen to suit the set the model is subsequently scored against, and any figure produced that way is mildly optimistic.
This is trap two from the list above, present in the configuration being documented. It is named here rather than omitted, because a methodology page that hid its own methodology defect would not be worth reading. The fix is a three-way split: train, validate, test, with early stopping watching validation and reporting done on a test set nothing has touched.
The leakage trap in this dataset
This dataset carries a well-known target leak. The duration
field records how long the call lasted, which is only knowable once the
call has ended, and by then the outcome is known too. A model given
duration is reading the answer rather than predicting it.
The UCI documentation says so directly and advises discarding the field
for any realistic predictive exercise.
It is worth sitting with why this one is so instructive. Nothing about
duration looks suspicious in a schema. It is a plain
numeric field, populated on every row, and it is genuinely correlated
with the outcome. The only thing wrong with it is when it
becomes known, and no amount of inspecting the data distribution will
reveal that. Leakage is a question about the timeline, not about the
values.
How ax1om handles it, and what that means for you
On the CSV upload path, leakage handling is advisory rather than enforcing. The pipeline screens for leakage and raises warnings, and the setup assistant proposes exclusions, but the operator decides and nothing removes a field automatically. That is deliberate, since only the operator knows which of their fields are populated before the decision and which after. It also means exclusion is a step you confirm, not one you can assume happened.
Whether duration was excluded in the run described
on this page is not documented. The per-run field selection
lives in a configuration snapshot that has not been published, so the
honest answer is that it is unconfirmed, and this page states it as an
open question rather than resolving it in either direction. It is also
why the unreconciled feature count in the configuration table matters:
publishing the field list would close both questions at once.
SHAP, and what it is for
SHAP decomposes a single prediction into per-field contributions, so a score arrives with the fields that moved it, up and down. ax1om computes it both globally across a model and per record, on the same path this demonstration used.
Two properties are worth stating precisely, because both are routinely misread. Global importance ranks how much a field moves predictions across the dataset, which is not a causal claim and not a statement that changing the field would change an outcome. And per-record contributions explain the movement of a prediction relative to a baseline; they do not sum to the 0-100 score a reader sees, because that score is a ranking derived from the underlying likelihood rather than a total of parts.
In an evaluation, SHAP earns its place as a leakage detector. A single field dominating global importance by a wide margin is the signature worth investigating, since that is what a leaked field looks like from the inside. Sorting importance and asking of the top few "was this knowable before the outcome" catches the failure that no accuracy metric will.
The global importance ranking for the specific run described here was not recorded alongside its configuration. Publishing a plausible ordering would be a fabrication about a run, so none is shown.
Run it yourself
The whole point of publishing a method rather than a number: you can execute this and produce your own figures.
-
Download Bank Marketing from the
UCI Machine Learning Repository
and take
bank-full.csv, all 45,211 rows, unfiltered. - Drop the
durationfield. This is the step that decides whether the exercise measures prediction or measures hindsight. - One-hot encode the categorical fields and leave the numeric fields as they are.
- Split 80/20, stratified on the label, at seed 42. You should get 36,168 training rows and 9,043 holdout rows. If you do not, your split is not stratified or your seed differs, and nothing downstream will match.
- Fit LightGBM with the parameters in the configuration table: 300 trees, learning rate 0.05, depth 6, 31 leaves, minimum 20 samples per leaf, subsample and column sample 0.8, balanced class weights, seed 42, early stopping at 50 rounds. To avoid the defect described above, hold a separate validation slice out of the training partition for early stopping instead of reusing the holdout.
- Score the holdout. Compute AUC. Then sort by score, take the top 20%, and compute its conversion rate for precision, divide that by the file's base rate for lift, and count what share of all conversions fall inside that slice for capture.
- Repeat at several seeds and look at the spread before you believe any single value.
Then do it again on your own data, because that is the only run whose numbers apply to you. A public benchmark shows a method works. It does not forecast your result, and any vendor implying otherwise is describing their dataset, not your pipeline.
Corrections to earlier ax1om copy
Assembling this page turned up three things ax1om had previously described inaccurately. Recording them is cheaper than being caught by a reader who checks.
- The record count. Earlier copy described the benchmark as "a standard public ML set of 36,168 records". That conflated the training partition with the dataset. The dataset holds 45,211 records; 36,168 is the 80% training partition.
- The training time. Earlier copy said the model "trained in" a stated number of seconds. That figure actually measured the whole pipeline end to end, including data assembly, feature build, splitting, SHAP computation, the 5-fold stability cross-validation, scoring and artifact upload. It was not model fit time. The figure itself is withdrawn along with the other outcome numbers.
- The early-stopping design. Never previously disclosed. Early stopping was evaluated on the same holdout partition the metrics were reported on, which makes any figure produced that way mildly optimistic. It is documented in the configuration section above.
The performance figures ax1om previously published have been withdrawn from public copy and are not restated anywhere on this page. ax1om does not publish model outcome numbers while every account is still on sample data. Results will be published when they come from real customer data, with permission, and they will be labelled as case studies.
Common questions
Why are there no performance numbers on this page?
Because ax1om does not publish model outcome figures while every account is still on sample data. That is a standing internal claims rule, not an oversight. A benchmark figure with no customer behind it invites exactly the reading we do not want, which is that it forecasts what you would get. The method is the part that transfers, so the method is what is published. When results come from real customer data, with permission, they will be published as case studies and labelled as such.
What should I actually measure on a lead scoring model?
Precision and capture at the top k percent, where k matches how much of the list your team gets through, with lift as the same finding restated as a multiple of the base rate. Use AUC as a threshold-free summary of ranking quality. Never use plain accuracy: conversion data is imbalanced, and predicting that nothing converts scores well on accuracy while being useless.
Why does this demonstration use a bank telemarketing dataset?
Because it is public, widely used and free to download, so anyone can rerun the method and get their own numbers. UCI Bank Marketing is consumer bank telemarketing, not B2B. It contains no companies, no opportunities and no sales cycles. It is structurally analogous to a lead list in the one way that matters for a method demonstration: a large list, a low base rate, and a mix of static attributes and interaction history.
What is target leakage and how do I check for it?
A field that is only knowable after the outcome has occurred. It predicts the outcome almost perfectly during evaluation and then fails in production, where the field is empty at scoring time. The check is mechanical: for every field, ask whether its value existed at the moment you would want to score the record. In this dataset the trap is call duration, because a call length is only known once the call has ended, by which point so is the result.
Does ax1om remove leaky fields automatically?
Not on the CSV upload path. It screens for leakage and raises warnings, and the setup assistant proposes exclusions, but the operator decides and nothing removes a field on its own. That is a deliberate design choice, since only the operator knows which of their fields are populated before the decision and which after, and it means leakage removal is a step you confirm rather than assume.
Why 36,168 records when the published file has 45,211 rows?
Because 36,168 is not a dataset, it is a training partition. The run used the full 45,211-row file and split it 80/20 stratified at seed 42, leaving 36,168 rows to train on and 9,043 held out. Those two numbers sum to 45,211. Earlier ax1om copy described 36,168 as the size of the dataset, which was wrong, and this page corrects it.
Can I reproduce this?
Yes, and that is the point of publishing it. The dataset, the variant, the split ratio, the stratification, the seed and the full parameter set are all on this page, and the split reproduces the partition counts exactly. You will produce your own performance figures, which is the correct outcome: a method you can rerun is more useful than a number you have to take on trust.
Citing this page
ax1om (2026-08-25). How to evaluate a predictive lead scoring model. Retrieved from https://ax1om.ai/research/how-to-evaluate-a-lead-scoring-model
This page describes a method and carries no performance results. If you are looking for what a model scored, there is nothing here to quote, and that is deliberate.