On flexibly switching between forward and reverse causal questions

For the last 7 years, I taught the first year grad methods course in my department, which is interdisciplinary. Students’ advisors might have degrees in psychology, economics, sociology, education, or some other discipline. I’ve noticed that in class, talks, and on Twitter, folks with different training seem to talk past each other in predictable ways. I have tried to describe one of them here, with the hope of improving productive discussion among folks with different perspectives.

1) Some disagreements among social scientists or between social scientists and the public arise when one side is in “forward causal inference” mode and the other side is in “reverse causal question” mode.

One can see this regularly when a researcher presents the results of a randomized experiment designed to test whether a treatment improves some outcome (a tutoring intervention designed to improve student achievement, let’s say), and an audience member asks a pointed question specific not to the findings but rather to their own causal variable of interest—economic inequality, for example. The audience member may ask why the researcher didn’t manipulate their own construct of interest, or perhaps whether the researcher “accounted” for their construct of interest—presumably, in some way other than designing the study to distribute it equally, on expectation, between groups.

After watching these dynamics in action for several years, I think that they have something to do with the distinction between "forward causal inference and reverse causal questions” (see Gelman and Imbens here). The hypothetical researcher presenting the tutoring evaluation is in forward causal inference mode (“What would happen to y if one did x?”) But the hypothetical audience member is in reverse causal question mode. (“Why do some kids have higher achievement test scores than others?”) When people are in these two different modes, they’re likely to talk past one another, except in the unlikely case that both sides agree the claim under debate is that “x, and only x, causes y.” 

2) Individuals or entire subfields can develop blindspots when they spend too much of their time in one of these modes and not enough in the other.

I now think the ability to flexibly flip back and forth between reverse causal question mode and forward causal inference mode is fundamentally important for social science research to be useful. This is not only because it can facilitate more productive discussion in the moment, but also because over time, individuals or entire subfields can develop blindspots when they spend too much of their time in one of these modes and not enough in the other.

A field that never comes out of reverse causal question mode is doomed to long arguments over different stories that are really hard to test and are probably not mutually exclusive accounts (e.g., "people vote this way because of x1"; "no, people vote this way because of x2"; people probably vote the way they do for all sorts of different reasons, and some are more important at certain times and for certain people than others). If some root cause is not what the scientist cares about, they can just focus on their proximal cause of interest. If the proximal cause is not what the scientist cares about, they can focus on their hypothesized root cause, or on some other partially overlapping causal chain, or on a completely distinct one. I think many frustrating discussions within the social sciences play out this way.

An advantage of spending some time in forward causal inference mode is that we can narrow the scope of the argument substantially. If you and I have different stories about why people come to be good at algebra (understanding of the equal sign, motivation, school funding, having an effective teacher), but our stories make identical predictions about what will happen to the posttest distribution of algebra test scores conditional on a long list of hypothetical interventions, maybe our stories aren’t as irreconcilable as they seem to us. Or, if our stories are substantively different, maybe it doesn’t matter as much as we thought it did. For example, if one person argues that “common environmental factors” have a large influence on adults’ test scores and another argues, “no, genes have a large influence,” but both agree that being randomly assigned to an upbringing in a somewhat more advantaged household might raise adult test scores by, say, around 1/3 of a population standard deviation on average, maybe there’s not too much to argue about after all. Or, to the extent there’s a disagreement, it’s more about rhetoric than about human development.

In contrast, a field that stays only in forward causal inference land is prone to miss the forest for the trees. One might spend a career trying to discern whether an effect is exactly 0 or always tiny but variable. When asked to shift to reverse causal question mode, they might construct a picture of the world in which only the kinds of causes that folks study with econometric methods matter (e.g., funding, laws, choices) and not fuzzy concepts that we struggle to define, much less instrument (e.g., culture, personality). I think some subfields within economics are probably sometimes guilty of this, as discussed in Akerlof’s recent paper on the "hardness vs. importance” tradeoff.

3) Students should get practice flexibly switching back and forth between these two modes.

I think it’s fine that people or subfields specialize in the reverse or forward causal mode to some extent, but to facilitate productive discussion among students, I want them to practice switching back and forth between reverse and forward causal questions. One way of doing this is to draw a path diagram with some treatment (let’s say a gifted and talented education program) pointing at adult earnings, and to ask students to identify a long list (maybe 10) confounders that plausibly influence both selection into the treatment and earnings, making estimating the effect of gifted and talented education on earnings difficult in the absence of some source of exogenous variation. This is useful, both because it helps the policy-oriented students think about the big picture and because it helps the students who spend most of their time in “reverse causal question” mode understand the importance of exogenous variation for estimating the causal effect of any of these hypothesized causes on earnings. Comparing causal estimates of different xs on y or to the correlation between x and y is also a useful way to show students that both kinds of questions can help inform each other.

At the end of the class, students don’t all think exactly the same way, which is great. I think the hypothetical audience member who asks “what about inequality?” is probably not fully explained by their different “mode” than the presenter; the audience member might think that the money and attention spent on the tutoring intervention might be better spent on redistribution. And I think it’s fair to question whether a particular line of research using forward causal inference is distracting researchers, the public, etc. from a more important set of causes. So reasonable people might disagree on whether some particular researcher or subfield should be spending more time in reverse or forward mode. But at the end of the class, I think that practice with the forward-reverse distinction helps students communicate, and hopefully they leave with a better understanding of why some people might find both kinds of work to be important.