AUC, honestly

AUC is the headline number on your model card. It is also the number most often misread, so start here.

Four KPI cards across the top of a model. AUC-ROC reads 0.85, labelled Excellent. Lift at top 20% reads 1.98x, against a 36.7% baseline. Precision at top 20% reads 72.5%, at 2.0x baseline. Records trained reads 8,000, with 86 features and a 19 second training time. — The KPI cards on a model · AUC leads, and the three cards beside it exist because AUC alone does not answer what people think it answers.

What it represents

AUC answers one question: if you pick a converter and a non-converter at random, how often does the converter get the higher score?

That is the whole definition. AUC 0.82 means that in 82 out of 100 random pairs, the converter outranked the non-converter. It is a statement about ordering.

It is not accuracy. AUC 0.82 does not mean the model is right 82% of the time, and it does not mean it finds 82% of your converters. A model can hit AUC 0.82 and still put only a handful of converters in your top 100, if converters are rare. Ordering and hit rate are different questions, which is why the card also shows lift and precision.

AUC 0.5 is a coin flip. Anything below 0.5 is worse than random ordering, which usually points at a data problem rather than a modelling one.

How it’s calculated

ax1om splits your labeled records before training, fits the model on the training partition, then scores the held-out partition the model never saw. AUC is computed on those held-out records only, comparing each record’s score against its real outcome.

The band under the number follows the same cut points shown in the app: 0.8 and above reads excellent · 0.7 to 0.8 good · 0.6 to 0.7 fair · below 0.6 reads as needs more data.

What it means for you

Use AUC as a go or no-go signal on the ranking, then stop.

Worked example. Your model reports AUC 0.78 on a dataset with a 4% baseline conversion rate. The ranking is sound, so a sorted worklist is worth handing to a rep. But 0.78 tells you nothing about how many conversions sit in the top 20%. For that, read precision at the top 20% and the lift curve.

Two cautions. An unusually high AUC, say 0.97 on messy CRM data, is more often target leakage than a great model. And AUC says nothing about how closely the score levels track real conversion rates. Ranking quality and calibration are separate properties.