Many scientists have pointed out possible errors of analysis and interpretation in a high-profile study that suggested microbes can ease autism-like behaviors in mice. The backlash was swift, appearing on Twitter and the post-publication review site PubPeer within hours of the paper’s publication.

In the paper, published 30 May in Cell, Sarkis Mazmanian and his colleagues reported that fecal transplants from autistic boys can trigger autism-like behaviors in mice—and that molecules from certain microbes can ease these behaviors.

Minutes after the study was published, independent experts began raising concerns on Twitter about three graphs included in the paper. They noted that the “P-value”—a measure of statistical significance—accompanying each graph seemed implausible.

“The data didn’t pass what I call the eyeball test,” one of the tweeters, Kevin Mitchell, associate professor of genetics at Trinity College Dublin in Ireland, tells Spectrum. Others raised concerns about the study’s small sample size.

“The data that you can see in the paper is quite misleading because it makes it look like there’s a lot of data points, but they’re really coming from a small number of participants,” says Jon Brock, a former autism researcher. Brock wrote an article, published June 13 in Medium, critiquing the small sample size and the tests of behavior the researchers used. “Given the fanfare accorded to this study, the evidence for its central claim is remarkably weak,” he wrote.

Comments like these prompted Thomas Lumley, a statistician who wasn’t involved in the study, to reanalyze the data. On June 16, Lumley posted his reanalysis on his personal blog. He concluded that one of the study’s key findings—that the mice socialize less than controls—may have been caused by a flawed statistical analysis.

Inappropriate analysis

For the analysis in question, the researchers had transplanted stool from each donor—five boys with autism and three controls—into at least eight different mice. They found that, as a group, mice with stool from autistic boys bury more marbles than controls do, spend less time interacting with other mice, and travel shorter distances in an open chamber. These results appeared in the three graphs Mitchell and others raised concerns about.

But in their statistical analysis, the researchers appear to have treated the mice in each group independently, even though some had received stool from the same donor, Lumley says. This mistake artificially inflated the sample size in each group. After correcting the error, the only statistically significant difference is the number of marbles the mice buried, according to his reanalysis.

“The authors’ analysis wasn’t appropriate for their experimental design, and also it wasn’t clearly described, because there were conflicting descriptions in two different places in the paper,” says Lumley, chair of biostatistics at the University of Auckland in New Zealand.

The researchers stand by their original results: “As we sit here today, we’ve found no errors with the statistics we’ve done,” Mazmanian, professor of microbiology at California Institute of Technology in Pasadena, tells Spectrum.

Mazmanian says he has asked several statisticians to reanalyze the data. “We want to get this right,” he says. “We are going to do everything possible to make sure that we arrive at a conclusion that we can defend.”

Lumley says he struggled with the reanalysis because although Mazmanian’s team had made the data available online, the statistical method they described in their figure caption did not jibe with that in the paper’s Methods section.

“The one in the figure caption was clear, but it wasn’t what they had done,” Lumley says. “And the one in the Methods section wasn’t clear.”

Statistical flaw

Lumley began his reanalysis using the methods described in the caption. This approach accounts for random differences between all fecal transplant donors and then looks for differences associated with autistic donors.

“That, I think, is the right analysis to do, but it doesn’t give the results they gave; it gives much less evidence for differences,” Lumley says.

He then reanalyzed the data following the description he found in the Methods section and additional details the researchers provided on PubPeer. He was able to reproduce the P-values reported in the paper for each of the three graphs.

Still, he says it is odd that the researchers treated each mouse as if it had received stool from a different donor.

“That’s a strange analysis,” Lumley says. “I can’t see any reason why you’d do that deliberately.”

The problem may lie in the software, called SPSS, the researchers used, he says: “I don’t know whether that contributed to their problem, but it’s the sort of thing that easily could have.” (Mazmanian says the team’s statistician followed the software’s guidelines.)

Even if the statistical analysis does not turn out to be flawed, critics say the provocative findings need to meet a higher bar than P-values.

“The P-value just summarizes the data you have in the experiment,” Mitchell says. “If it’s an extraordinary claim, if it requires some new biological mechanisms that are really unknown and no foundation of research strongly supports it, then we should ask for a higher standard of evidence.”

One higher standard, Mitchell says, is replication. Another is a registered report, a type of peer-reviewed journal article in which researchers first describe their study methods before gathering their data.

Critics of the paper also noted that four of the team members have pending or approved patents related to the use of microbes or microbial molecules as treatments for autism and other conditions. Mazmanian is also co-founder and director of Axial Biotherapeutics, a company developing gut-based treatments for these conditions.