Many psychological studies rely on participants to give up their time to take part in experiments or complete questionnaires. They take part because they get paid or because they are required to as part of their university course. But, beyond this, not much is known about what motivates people to take part in these studies.

Some participants may be looking for help—perhaps seeking a diagnosis for a mental health issue they're struggling with. A team of researchers in Poland theorized that taking part in a psychological study might be "perceived as a cheap substitute or alternative to acquire some professional help". To this end, they set out to discover if participants in psychological studies were more likely to have a personality disorder or be experiencing depression or anxiety.

Their results are published in the open-access journal PLOS ONE.

"Researchers often take for granted that the way they advertise their studies and who they recruit do not appreciably affect their outcomes," the study authors write. "In our studies, we have shown that those who have more personality pathologies are more drawn to studies where they can express their trauma and may be simply more likely to volunteer for studies."

Izabela Kaźmierczak and colleagues at Maria Grzegorzewska University in Warsaw, Poland, conducted several studies, involving 947 participants in total (62% of whom were women), comparing people who had previously taken part in psychology studies with those who had never taken part in such studies.

They found that participants who had previously taken part in studies exhibited symptoms found in those with personality disorders, depression or anxiety. There are many different types of personality disorder—including borderline personality disorder and narcissistic personality disorder—but, in short, a person with a personality disorder thinks, feels, behaves or relates to others differently from those without it. They may, for instance, blame people for things, or behave aggressively and unpredictably.

Why it matters

What this new study has revealed is a potentially worrying issue of self-selection. Since participants in research choose which studies to take part in, the results of the research may be unduly influenced by a large number of participants of a particular type taking part. Study bias is a serious issue.

Like many other scientific disciplines, psychology research is designed and carried out mainly in universities. Unlike many disciplines, though, psychology requires human participation and, as such, students form a handy subject pool from which to draw. This has led many in the field to wonder how research carried out on predominantly 18 to 22-year-old western students can provide findings that are in any way relevant to any population other than 18 to 24-year-old western students.

Research needs to be valid, and if we cannot claim that our findings relate to the wider population (so-called "generalizability") we have a serious issue. What this new study shows is that our findings may well be influenced by the psychological nature of the very people we are testing.

We cannot, however, control the students who give their time to sit through our procedures. For instance, we cannot provide instructions on recruitment posters that say, "Those with symptoms of personality disorders need not apply." But we can and must be more careful in how we select our participants.

What we need to do is carry out research with large enough numbers of people, work that can be repeated, that can allow us to be more confident that our findings have relevance off campus.

Bumpy road

All sciences have their bumpy roads to travel, and psychology has certainly been traveling on one in recent years. Experiments that were once deemed to be groundbreaking, have failed to produce the same results when they were repeated by other psychologists. This is known as the "replication crisis" or "reproducibility crisis".

And the shockwaves caused by the scientific treason of Diederik Stapel, a Dutch psychologist who invented his data and even fabricated entire experiments, are still being felt. Psychology's reputation has certainly taken a battering.

But psychologists are working carefully on developing transparency and techniques we hope will help us regain the faith of the wider scientific community. What this latest paper has shown is that the participants themselves may well be self-selecting—and, as a result, our findings may again be called into question. We may think we are drawing from as general a population as possible to make the results generalisable to the wider population, but that may not be the case.

This finding will set alarm bells ringing in those working to develop the reliability and reputation of psychology. It needs to be taken seriously.

The results tell us more formally something we should have already known. Those of us involved in psychological research involving participants drawn largely from a pool of psychology students need to be very careful in our recruitment strategies. We might, for instance, need to take care to design research that may not be influenced by the personality or mood of the participant, or we may need to assess the participants taking part in our research. For example, the authors of this latest study suggest winnowing out participants who have taken part in previous psychology studies.

Most importantly, we need to be very careful in the grand claims we make after we publish how our "groundbreaking" research relates to the wider population we look to be investigating. Such a claim may not, it seems, stand up to scrutiny.