There’s a certain kind of study we used to see all the time. The researchers ask all participants to make a judgment regarding the exact same question, but then they vary something in the external situation. They change the temperature in the room. Or the song that is playing in the background. Or they do something that’s supposed to make people have a particular emotion, or engage in more reasoning, or show more or less of some other psychological process.
A key lesson of post-replication crisis psychology is these sorts of manipulations don’t usually do much. For example, if you try to change the situation so that people feel more of certain emotions, their philosophical judgements remain pretty much unchanged, and if you try to change the situation so that people engage in more reasoning, their philosophical judgments also remain pretty much unchanged.
Within existing research, one sees a lamentable tendency to think about each of these results separately and give a completely separate explanation of each. Proceeding in this way, one might say that the former result indicates that emotions don’t impact people’s philosophical judgments… and then separately, one might say that the latter result indicates that reasoning doesn’t impact people’s philosophical judgments.
But this misses the larger picture. It sure looks like the reason why we don’t get a big effect when we try to manipulate people’s emotions isn’t due to something super specific about emotion in particular. Instead, we are getting growing evidence that this type of experimental manipulation just generally doesn’t do much.
Suppose you are thinking about what makes certain people more conservative, and you want to know whether it is a matter of some psychological state X (which might be a certain emotion, or a way of reasoning, or anything else). How do you test this hypothesis? The traditional idea was that you would run a study that lasted, say, five minutes in total, in which you temporarily increase the amount of state X and then show that this manipulation leads to a temporary increase in conservatism.
But it now seems like this whole approach just fundamentally does not work. The problem is not that we have the wrong X, or that we aren’t doing exactly the right thing to manipulate it, or anything like that. The problem seems to be that the human mind works in such a way that people’s judgments are stable across these sorts of temporary changes.
A few years ago, I wrote a paper about this topic, but that paper was mostly just about all the little details of the empirical data. I’m thinking that it might be helpful to zoom out a bit and think in a larger way about what we are learning from all of these studies. It seems like we face two different questions: one substantive, one methodological.
The substantive question is: What are we learning about the human mind from the fact that people’s judgments cannot be pushed around by these brief manipulations? I don’t know the answer to this question, but just to bring the key issue out a little more clearly, it might be helpful to consider a simple example.
It seems plausible that my dispositions to have certain emotions led to my interest in philosophy. But suppose we took a random person and, just for a single day, gave that person all the emotions that I typically have. Presumably, having these emotions for a single day would not lead the person to start philosophizing on that day (nor is it the case that if I stopped having these emotions for a single day, I would stop philosophizing for that day). If the emotions have any effect it has to be a much more long-term effect – with the philosophy I do today being shaped by the emotions I’ve had over the past twenty years.
How exactly is this to be understood? It does seem like we’re getting growing evidence that this happens, but I wouldn’t say that we already have a good understanding of how or why it happens.
The methodological question is: If this specific method does not work, how we can test claims about the causal impacts of psychological processes on judgments? Suppose we are wondering whether factor X has a causal impact on people’s judgments. One thing we can do is to check to see whether there is a correlation such that people who are dispositionally higher in factor X are more likely to make certain judgments. There are already lots of great studies of that form, and they have taught a lot about the relevant correlations. But one might legitimately wonder whether this approach provides a real test of the relevant causal claims.
The traditional solution was to try to temporarily manipulate factor X and check for a temporary effect on judgments. But if that doesn’t work, what should we be doing instead?