About a decade ago, I wrote an article on interaction terms. In it, I tried to clear up some common misperceptions about how interaction terms can and should be used in regression (logit, probit, etc.) equations. A fair number of people seem to have heeded most of the advice in the article, but in retrospect I realize that I should have put a much greater emphasis on one very important topic: incomplete or overlapping sets of variables in interaction terms.
Political scientists (and, as far as I can tell, uniquely political scientists) have created and sustained the illusion that we can pick and choose which variables to interact in our statistical models without concern for omitted variable bias. Do you want to understand the impact of being a left-handed Latino Republican on how much the respondent likes the President? Multiply left-handed by Latino by Republican and include that in your regression. Don’t worry about including a variable for left-handed Republican, or left-handed Latino, or even left-handed, because they’re not part of your theory. By the same logic (the thinking goes), if you have a theory about left-handed Latinos and left-handed Republicans, go ahead and put those two interactions in there without including left-handed Latino Republicans, because they’re not a part of your theory.
Boo, political scientists. Booooooooo.
The short version is this: When creating interaction terms, always include every possible lower- and higher-order term in your model. If you’re interacting x1, x2, and x3, include x1x2, x1x3, x2x3, x1, x2, and x3 in the equation. If you’re interacting x1 and x2 and, in the same equation, interacting x1 and x3, do the same thing—include everything all the way up to x1x2x3.
Why?
Think of a coefficient as reflecting some difference between the category you’re interested in and an excluded category. In the example above, the coefficient on left-handed x Latino x Republican reflects the difference between left-handed Latino Republicans and other people.
The key point is… who are these other people?
Obviously, the excluded category includes right-handed Black Democrats. Not so obviously, it includes left-handed people who either aren’t Latinos or aren’t Republicans or both. It includes Republicans who either aren’t left-handed or aren’t Latinos or both. It includes Latinos who either aren’t left-handed or aren’t Republicans or both.
In short, your excluded category is a mess.
“Fine, fine,” you say, “But obviously, the coefficient on left-handed Latino Republican still estimates the effect of being a left-handed Latino Republican, right?”
No, it doesn’t. Because you haven’t controlled for just being a Republican. Or being a left-handed Republican. And so on. And the absence of those controls can produce misleading inferences.
Let’s imagine that Republicans really dislike the President but Latino identity, handedness, and all combinations of the three are irrelevant. You could very well get a significant coefficient—a false positive—on your variable just because left-handed Latino Republicans are more Republican.
Alternately, let’s imagine that left-handed Latino Republicanness really does change people’s attitudes about the President, but that on their own, Republican-ness pulls in one direction, left-handedness pulls in another, and Latino-ness doesn’t matter. You could very well end up with a null result—a false negative—just because your excluded category is so heterogeneous.
So, no. Your excluded category is such a friggin’ nightmare that your coefficient and significance level tell you precisely nothing about the quantity that you’re trying to estimate.
Let that sink in for a moment. Precisely nothing.
So next time you’re reading a journal or watching a presentation at a conference, keep an eye peeled for omitted terms in interactions. You’ll be surprised at how many of the results out there really don’t tell you anything at all.