The 5 Ways a Causal Diagram Goes Wrong

A causal diagram is only as good as the thinking behind it. Get the structure wrong and the answer will be wrong too, however sophisticated the maths underneath. The encouraging part is that a handful of patterns account for nearly all the errors, and all five are recognisable without any statistical training.

CausoAI catches many of these automatically during validation. Knowing why they matter helps you build a better diagram in the first place, and makes the warnings easier to act on.

Mistake 1: Holding constant something that both sides caused

Some factors are downstream of everything. If a variable is a result of both the thing you changed and the outcome you care about, holding it constant does not clean up the comparison — it actively distorts it, manufacturing a relationship that was not there.

An example. You are measuring the effect of ad spend on sales, and you decide to hold “lead quality” constant, on the reasonable-sounding grounds that it affects sales. But lead quality is itself produced by two things: the ads that brought the leads in, and the sales team that qualified them. Holding it constant inflates the apparent effect of the ads.

Mistake 2: An arrow pointing the wrong way

Direction changes everything — which factors need holding constant, and whether the question can be answered at all. It is also the single easiest thing to get wrong, because data on its own often cannot settle it.

Satisfaction and retention are correlated. Does satisfaction keep customers, or do customers who stayed reinterpret their experience more warmly? Both stories fit the same numbers. Only knowledge of your business, or clear timing, tells you which is happening.

CausoAI proposes a direction where the data supports one, and timing resolves a great deal — something recorded later cannot have caused something earlier. But for two things measured at the same moment, the call is yours.

Mistake 3: Leaving out the factor behind both sides

This is the most common error in the whole field, and the most expensive. Some factor drives both the thing you changed and the outcome. Leave it out of the diagram and its influence gets credited to your treatment.

Take email frequency and purchases. Customer engagement drives both: engaged customers opt into more email, and they buy more anyway. Leave engagement out and you will conclude that email frequency is enormously effective, because you have handed it the credit for everything engagement was doing.

• —Ask what determines who receives the thing you are studying. Those are your candidates
• —Then ask whether each of those also affects the outcome on its own. If yes, it belongs in the diagram
• —Treat the platform’s automatic suggestions as a starting point, not a finished list — it can only see what is in your file

Mistake 4: Holding constant the thing the effect travels through

The mirror image of mistake 1. Some factors sit in the middle: the treatment causes them, and they cause the outcome. Hold one of those constant and you block the very route the effect was travelling along, making a real effect look like nothing.

Suppose a new onboarding email drives feature adoption, and feature adoption drives retention. Hold feature adoption constant and you are asking what the email does for retention among people whose adoption did not change — which is close to nothing, and not remotely the question you meant to ask.

For the total effect, leave middle factors alone. If you specifically want to know how much of the effect travelled through adoption and how much went around it, that is a separate analysis, and CausoAI can split it out for you.

Mistake 5: Drawing a loop

A diagram cannot contain a circle. If A causes B and B causes A at the same instant, the question “what does A do to B” has no answer — every effect feeds back into its own cause forever.

Loops usually appear when someone tries to draw a genuine feedback dynamic: price affects demand, and demand affects the next price. That dynamic is real. The fix is to add time, so the loop unrolls: price this month affects demand next month, which affects price the month after.

CausoAI checks for loops whenever you save a diagram and points at the arrow causing the problem.

A practical checklist

• —For everything you are holding constant: check it did not happen as a result of both the treatment and the outcome
• —For every arrow: confirm the direction using timing where you have it, and business knowledge where you do not
• —For the treatment: list what determines who receives it, and include anything on that list that also moves the outcome
• —For anything sitting between treatment and outcome: leave it alone unless you deliberately want the effect split apart
• —Read the validation warnings on check 2 — that is where these problems surface