Most customer success teams have a churn model by now, and many of them are genuinely good — they identify accounts at risk weeks before anyone leaves, with accuracy that stands up. And yet retention often does not move.
This is not a modelling failure. It is a category error. Prediction and intervention are different problems, and a model built for one does not answer the other.
The two questions
A churn model answers: given what I can see about this account, how likely are they to leave? That is a question about association. It does not care why the pattern holds, only that it holds reliably.
Retention asks something different: if I do X for this account, how much less likely are they to leave? That is a question about consequence, and it requires you to know which of the things you can see are causes and which are merely symptoms.
A model can be excellent at the first question and tell you nothing about the second.
Why the top features are usually the wrong things to act on
Open your model’s feature importance list and the strongest predictors are typically things like declining login frequency, fewer seats active, support tickets going quiet, or a drop in a core usage metric.
Every one of those is a symptom. Logins fall because the customer already decided the product is not worth the time. The decline is the churn becoming visible, not the reason for it. Acting on a symptom produces exactly the initiatives that feel productive and change nothing.
- —Emailing accounts whose usage dropped, to encourage them to log in more
- —Pushing adoption of the feature that retained customers happen to use most
- —Running a check-in call triggered by a risk score, with no view on whether the call does anything
The second one is worth dwelling on, because it is the most common. Customers who use your advanced feature churn far less. So the team runs a campaign to drive adoption of that feature — and retention does not budge. The feature was never the cause. Customers who were already committed explored more of the product. Adoption was the evidence of commitment, not the source of it.
What actually distinguishes a cause
A useful causal question about churn has a specific shape. It concerns something you can actually do, at a moment you can actually choose, and it compares against a world where you did not do it.
- —Does an onboarding call in the first two weeks reduce 90-day churn — for accounts that would not otherwise have had one?
- —Does assigning a named CSM change renewal rates, or do the better accounts simply get named CSMs?
- —Does a quarterly business review reduce churn, or is it that healthy accounts agree to attend them?
- —Does responding to a support ticket within an hour rather than a day change anything downstream?
Each of these has the same trap built in: the intervention is not assigned at random. Onboarding calls go to accounts that book them. Named CSMs go to bigger contracts. QBRs happen with customers who are already engaged. Compare recipients against non-recipients directly and you will overstate every one of these, often by a lot.
If your team decides who gets an intervention, that decision is entangled with the outcome. Any honest analysis has to account for how the decision was made.
Use both models, for different jobs
This is not an argument against churn prediction. The two approaches do complementary work, and the strongest retention programmes run them together.
- —The churn model chooses who to look at — it is a targeting tool, and a good one
- —The causal analysis chooses what to do about it — which intervention actually shifts the outcome, and by how much
- —Segment-level effects tell you who to do it for, since interventions rarely work equally across your base
That last point tends to be where the money is. An onboarding call might cut churn substantially for self-serve accounts that arrived without a sales conversation, and do almost nothing for enterprise accounts that had six weeks of implementation support. Averaged together, the effect looks mild and nobody prioritises it. Split apart, one clear action emerges for one clear segment.
What you need to run this
One row per account, with whether they churned in a defined window, which interventions they received and when, and the account characteristics that influenced both — size, plan, acquisition channel, tenure, and health at the point the intervention decision was made.
That last qualifier is where most attempts go wrong. Measuring account health today, after the intervention, bakes the effect you are trying to measure into the thing you are controlling for. You need the state of the account at the moment the decision was made, not its state now.