Here’s How Gender Bias In Social Science Works

A new study finds gender differences in learning, using a methodology guaranteed to maximize gender differences. Bad science in action.

There was a study recently about how people perceive science, particularly erroneous scientific information, when it’s presented in the form of narrative. It came up with one of those gender dichotomies that bad science journalists love, so it’s getting a little attention from that. Unfortunately, a look at the actual experiment shows that they weren’t running the experiment they thought they were, and consequently their results are functionally meaningless. And I can prove it.

First some background. The full study is here, or if you’re in a rush, Colin Schultz has a nice article about it here.

Here, in the experimenter’s own words, is what they thought they were testing and what they believe they found:

This experiment examines the influence of narrative transportation, role of science within the movie, and gender of the viewer on evaluation of incorrect scientific information in fiction. Results show that incorrect science facts accepted as true after seeing identical segments from movies depend on the gender of the participant and a manipulation of the perceived centrality of science to the plot. Men tended to detect more inaccurate science facts when they thought science was central to the plot. Women detected more inaccurate science facts when they thought science was peripheral to the plot, which was presented as a relational story.

Here’s a concise breakdown of how that actually functioned, from Mr. Schultz:

To test their idea, the researchers presented the same film clip to the test subjects in one of two ways, with either the science, or the interpersonal relationships between the characters, being played up.

Barriga, Shapiro and Fernandez found that when the science was presented as the main focus of the clip, men caught 3.23 +/- 0.28 (out of 8) of the errors, while women found 2.46 +/- .30 of the flaws. When the participants were told the story was a tale of love and lust, with lasers filling in as the sidekick, men detected 2.68 +/- 0.28 scientific inaccuracies, and women dismissed 3.14 +/- 0.31 of the errors. The men and women tested caught about the same number of errors, it just depended on how they were presented – even after controlling for things like general science knowledge and level of education.

Now, first off, that gender difference is not huge; it may well just be statistical error in a hat. Gender is the only variable anywhere in the study which shows anything approaching statistically significant variance, which suggests that the researchers seized on it because “our study didn’t find anything at all” doesn’t make for a very compelling paper.

But even assuming statistical significance, even granting, rather generously, that their findings are valid, the result is still bullshit because their methodology is so bad it’d produce a gender difference no matter what the actual content was.

The human mind suffers from what are called priming effects, where unconscious associations cause us to make decisions differently, and in particular, if we’re reminded of a role we have, we’ll behave more in accordance with the stereotype of that role. So the only thing this study was actually testing was whether men would pay more attention to a clip they were told was an action movie, and whether women would pay more attention to a clip they were told was a romance movie. All that stuff about science and learning styles and narrative centrality became irrelevant the minute they primed their subjects with a gender stereotype. The only thing they were testing was priming effect, and surprise surprise, they found out it works. And then reported that they found something else, which is just plain embarrassing.

Some might be skeptical, but this kind of priming effect is depressingly well-documented. One study (PDF link) showed that reminding women that they’re women right before giving them a math test produced worse results, because the subjects started unconsciously running the stereotype that women are bad at math. Another one found they could reverse that effect with Asian-American women, by reminding them of their Asian heritage instead of their gender, thus activating the stereotype that Asians are great at math.

Remember this the next time you see a study about how men and women are soooooo different, amirite? If it’s getting reported, it’s because it supports popular stereotypes, and there’s a good chance that those same stereotypes are all that that study’s finding. A neat little cycle, but not one that tells us anything about any reality behind the assumptions.

 

Photo—jonfeinstein/Flickr

NOW TRENDING ON GMP TV

Super Villain or Not, Parenting Paranoia Ensues
The Garbage Man Explains Happiness
How To Not Suck At Dating

Premium Membership, The Good Men Project

About Noah Brand

Noah Brand is an Editor-at-Large at Good Men Project, and possibly also a cartoon character from the 1930s. His life, when it is written, will read better than it lived. He is usually found in Portland, Oregon, directly underneath a very nice hat.

Comments

  1. For another example of this, see my critique of how household chores are calculated in national time use surveys- right here at GMP: http://goodmenproject.com/featured-content/the-good-life-these-chores-dont-count-on-mens-hidden-second-shift/

    (As a social science researcher myself, it infuriates me when I see poor studies, and especially when “jornalists” uncritically tout spurious results based on a press release and not on understanding what is and isn’t good research methodology)

  2. Liz McKinnell says:

    I’m not sure that this study really merits this sort of attack. The abstract for the study states that the findings were that:

    Results show that incorrect science facts accepted as true after seeing identical segments from movies depend on the gender of the participant and a manipulation of the perceived centrality of science to the plot. Men tended to detect more inaccurate science facts when they thought science was central to the plot. Women detected more inaccurate science facts when they thought science was peripheral to the plot, which was presented as a relational story. The results may have implications for research on media effects, public understanding of science, and gender differences in learning.

    (I can’t get the actual article, because the link from the abstract goes straight to s different but very interesting one on the ethics of science communication)

    There is NOTHING in the abstract at least that says that these responses are ‘hard-wired’ (even if that phrase makes any sense). The study does not propose any form of gender essentialism, or suggest that men and women ‘just are’ a particular way, taken apart from culture.

    There are various explanations that we might give for the results that were found. Some of them (probably pretty rubbish, biased, unsubstantiated ones, because they usually are) might appeal to some sort of essential sexual dimorphism in the brain. Others, like you, may say that this is due to a set of self-imposed stereotypes that we have been trained to absorb.

    But please don’t put these things at the door of the experimenters without good cause. That is as bad as the idiots who leap on this sort of study as justification to send us all back to the kitchen.

  3. So the men overall detect more science inaccuracies? Interesting. Wonder if that’s due to men being more S.T.E.M based?

Trackbacks

  1. [...] A new study finds gender differences in learning, using a methodology guaranteed to maximize gender differences. Bad science in action.  [...]

Speak Your Mind

*