Psychology’s Replication Crisis

And the decline of trust in science

Photo: Five Thirty Eight

The condensed alternative of this article was first published at Areo Magazine.

The science of the mind has never been exclusively objective. To prove that this is true, I want to lay out a theory posed by one of the most well known and celebrated psychologists in the world. The theory goes something like this: during psychosexual development, children begin to experience extremely erotic feelings toward their parent of the opposite sex and hatred for their same-sex parent. Boys compete with their fathers to achieve their mothers’ affection and attention while girls become jealous of their mothers and desire their fathers. This is a defining moment in one’s gender identity. The 20th century psychologist who developed this theory titled the Oedipus complex is Sigmund Freud.

Crazy theories of this kind don’t represent psychology today, one might insist. It’s true that psychological theories of today are not nearly this blatant in their errors. However, it has still been a bumpy ride since Freud. John B. Watson founded behaviorism which dominated the field throughout the 1920’s until about 1960. Behaviorism has been known to treat the human mind as if it were a blank slate at birth. Linguist and cognitive scientist Noam Chomsky explains precisely what’s lacking in the psychological approach by taking a look at language development in children. Language is acquired without being explicitly taught and behaviorism doesn’t offer an account of how this could be so. The obvious conclusion one will come to upon discovery of this fact is that there are some mechanisms in our brains that we are born with; not everything comes from learning or culture. Again though, this was back when the field of psychology was still developing. Surely, the field isn’t as bad today, in the year 2019.

The replication crisis is a term used to describe studies from fields ranging from economics to chemistry, with psychology in the middle of the controversy, that have failed to replicate. Being an essential aspect to determining whether a study is an accurate measurement of reality, this has made people more skeptical of the social sciences in particular. The process of replication is important because if the study of a phenomenon consistently produces similar results then we know it is far more likely to be an objective representation of the real world. This is especially true if the research is done by different scientists and if the methodology is continuously improved.

Studies have now been done in order to test the replicability of past research and the results did not look good. Over the last couple of years, about 200 psychologists have done this experiment and failed in 14 out of 28 cases. The findings that failed to replicate aren’t just popular myths that one might hear from their relative or friend either. Rather, they are studies you will learn about in any social psychology class and even in top tier scholarly journals. Other researchers have attempted to replicate 52 studies published in the Journal of Personality and Social Psychology and Psychological Science. Using a variety of measures to ensure reliability, they found that between 25% and 43% of the papers successfully replicated. Neither group of researchers were able to replicate more than half of the psychology studies in question.

The list of inaccurate yet popular psychological concepts is long:

• Power pose
• Multiple intelligences
• Stanford prison experiment
• Implicit bias training
• Birth order
• Ego depletion
• Sex addiction
• 10,000 hours of practice
• Stereotype inaccuracy
• Growth mindset
• Left/right brained
• Emotional Intelligence
• Attachment styles
• Learning styles
• That there are no psychological sex differences

Let’s take the concept of the power pose as an example. This is the idea that before an important event in which confidence is necessary, one can gain the confidence they need by posing in a way that they associate with being powerful. This is said to help people be more assertive and dominant in social settings. What does the research say though?

This research from 2010 states the conclusion right in the abstract: “Posing in displays of power caused advantaged and adaptive psychological, physiological, and behavioral changes, and these findings suggest that embodiment extends beyond thinking and feeling, to physiology and subsequent behavioral choices.” The central claims in this paper are that power posing causes an increase in testosterone, a decrease in cortisol and produced a larger focus on rewards as well as feelings of empowerment. The authors also note that one must only stand in a power pose for a mere two minutes to experience these results. This data was so intriguing that TED had a social psychologist on to do a speech on it. The concept was also adored by several media outlets. If you didn’t begin to read this paper here, you’ll know that regardless of how the theory is often treated, this research doesn’t hold up to scrutiny.

To start, the sample size is only forty-two. This is an incredibly small number that is very unlikely to generalize to the population at large. A sample under 100 should rarely be taken seriously (and even 100 is pushing it). Another thing to consider when reading a research paper is whether there is a detailed methodology section. If one can’t quite follow the way the experiment was run, perhaps the researchers didn’t put too much effort into ensuring that they followed scientific methods of inquiry. Finally, it is important to seek out other studies done on the same question. If there are other studies with consistent results, there is a higher likelihood that the data will translate into the real world.

Now, let’s take a look at a study that attempted to reproduce the 2010 research above on the concept of a power pose:

This study, as mentioned in the abstract, has a sample size of 200 which is much higher than 42. These researchers, using a similar methodology, “found no significant effect of power posing on hormonal levels or in any of the three behavioral tasks.” Power posing does not increase testosterone, encourage people to focus on rewards, or match the conclusion of the original paper in almost any way. They did find a significant effect on self-reported feelings of power, though self-reports have methodological problems of their own that are less substantial.

This is the general story of the other psychological findings listed above. Though the replication crisis is very important to know about, I haven’t learned about it in any of the five college psych courses I’ve taken nor have I learned about it in popular culture.

Social psychology isn’t even the only field that has a problem with bunk methodologies though. The problem of reproducibility is also present in economics, chemistry and engineering. This is a problem on an even larger scale in fields like gender studies, fat studies, and even sociology to some degree. Consider the case of James Lindsay, Peter Boghossian and Helen Pluckrose, three academics who tested the strength of many of these fields’ peer review process. These academics wrote twenty research papers with completely made-up data — fashionable nonsense — that appeals to the biases of these fields and made an attempt to get them placed in prestigious scientific journals for the fields of gender studies, queer studies and fat studies.

These fabricated data included discussion from the authors about a variety of silly ideas, as explained in their piece in Areo Magazine:

Many papers advocated highly dubious ethics including training men like dogs (“Dog Park”), punishing white male college students for historical slavery by asking them to sit in silence in the floor in chains during class and to be expected to learn from the discomfort (“Progressive Stack”), celebrating morbid obesity as a healthy life-choice (“Fat Bodybuilding”), treating privately conducted masturbation as a form of sexual violence against women (“Masturbation”), and programming superintelligent AI with irrational and ideological nonsense before letting it rule the world (“Feminist AI”). There was also considerable silliness including claiming to have tactfully inspected the genitals of slightly fewer than 10,000 dogs whilst interrogating owners as to their sexuality (“Dog Park”), becoming seemingly mystified about why heterosexual men are attracted to women (“Hooters”), insisting there is something to be learned about feminism by having four guys watch thousands of hours of hardcore pornography over the course of a year while repeatedly taking the Gender and Science Implicit Associations Test (“Porn”), expressing confusion over why people are more concerned about the genitalia others have when considering having sex with them (“CisNorm”), and recommending men anally self-penetrate in order to become less transphobic, more feminist, and more concerned about the horrors of rape culture (“Dildos”). None of this, except that Helen Wilson recorded one “dog rape per hour” at urban dog parks in Portland, Oregon, raised so much as a single reviewer eyebrow, so far as their reports show.

Seven of these papers made it through the peer-review process entirely and seven more were still being reviewed when these academics had to go public with their project. Only six papers had been rejected as insufficiently scientific by these scholarly journals.

Some have argued that the replication crisis has made the field of psychology better. We now know that a lot of people think otherwise. According to Pew Research Center, 44% of republicans believe the scientific method can be used to produce any conclusion the researcher wants. On the other side of the isle, 29% of democrats believe the same to be true.

Perhaps this doubt isn’t for all the wrong reasons either. In my persuasion course which involves a sizable amount of social psychology, my professor stated that we are born blank slates when talking about gender differences in toy preference. It is ridiculous to make this claim because in doing so one is discounting biological factors that play a role in shaping who we are as well as socialization. Firstly, there is evidence to support the claim that children exhibit sex-specific preferences. There have even been studies done on monkeys which have found the exact same sex-specific preferences in infants. Dr. Gad Saad, a psychologist at Concordia University, explains this in greater detail in this Psychology Today article.

When a professor argues something so obviously false such as the hypothesis that everything is learned from socialization, they are giving republicans good reasons to doubt them as well as the universities they work at. To some, right-wingers doubting the credibility of climate science looks less extreme than it really is when they also doubt the things that obviously are false: that men and women do not differ psychologically. When I heard my professor state this, I had to push back. I told her that I was trying to understand the notion that we are born blank slates and asked if evolution arbitrarily ceased above the neck fifty thousand years ago. She then responded by saying this topic is debated by scientists today and that it’s a combination of both nature and nurture. While it is true that nature and nurture collectively shape who we are, it’s disingenuous to argue for blank slate ideology. The idea that we are born blank slates implies that only nurture plays a role.

Left:moderate:right ratio among professors

This ideology is relatively common among social science professors due to the lack of viewpoint diversity among them. In psychology, the left-right ratio is 17 to 1 and most other disciplines in the humanities/social sciences are more than 10 to 1. This can undermine the quality of scholarly research as well as what students are learning. What is found in research and taught at universities will often be “left-shifted” from the truth. Social psychologist Jonathan Haidt explains this in greater detail in his book The Coddling of the American Mind.

Does this mean that we shouldn’t trust psychology or science in general? As a student of the field, I’d argue otherwise. As Stephen Jay Gould has pointed out, “In science, ‘fact’ can only mean ‘confirmed to such a degree that it would be perverse to withhold provisional assent.” In other words, it is irrational not to assume the truth of established knowledge until reproducible studies done by independent teams suggest otherwise.

Lee Jussim (et al) expand on this in their paper “A Social Psychological Model of Scientific Practices” which lists five conditions that should be met to confirm the validity of a phenomenon:

1. Something must be found
2. It must be subject to rigorous, pre-registered attempts at replication by researchers independent of the team who made the discovery.
3. Most of those rigorous pre-registered replication attempts must succeed.
4. Pre-registered meta-analyses of those pre-registered studies must reveal that the phenomenon exists even after attempts to assess and remove biases in the literature.
5. The conclusions and interpretations regarding the phenomenon, even when its evidentiary basis is sufficiently strong to be considered “valid new knowledge,” need to be subjected to long and intense skeptical scrutiny by the scientific community.

Take the well-established data on personality. The big five model of personality has been studied endlessly, both within and outside western culture. Cross-cultural research of this kind tends to be more robust because it can be said to apply to the human population universally. This personality model has also been replicated countless times. Personality psychology doesn’t care about making profound points: it relies on objectivity and on the admission that the truth is sometimes boring — and that’s OK.

Professors, science writers and the media as a whole ought to abstain from presenting interesting findings as objective facts, unless they are established knowledge. We can learn from the science wars, as well as from methodological errors in our scientific methods. Science is the most objective way of measuring reality that we have, and it works pretty well most of the time. But we should work to improve it, and hold it accountable when its findings are incorrect. We ought to increase the amount of viewpoint diversity among professors. And we ought to teach our students about the replication crisis. This will strengthen public trust in academia and improve our methods of scientific inquiry.