Hypothesis: peer review doesn't work and makes science worse

[Winehole23]

Here’s a simple question: does peer review actually do the thing it’s supposed to do? Does it catch bad research and prevent it from being published?

It doesn’t. Scientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this. In this study reviewers caught 30% of the major flaws, in this study they caught 25%, and in this study they caught 29%. These were critical issues, like “the paper claims to be a randomized controlled trial but it isn’t” and “when you look at the graphs, it’s pretty clear there’s no effect” and “the authors draw conclusions that are totally unsupported by the data.” Reviewers mostly didn’t notice.

In fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time. If reviewers were doing their job, we’d hear lots of stories like “Professor Cornelius von Fraud was fired today after trying to submit a fake paper to a scientific journal.” But we never hear stories like that. Instead, pretty much every story about fraud begins with the paper passing review and being published. Only later does some good Samaritan—often someone in the author’s own lab!—notice something weird and decide to investigate. That’s what happened with this this paper about dishonesty that clearly has fake data (ironic), these guys who have published dozens or even hundreds of fraudulent papers, and this debacle:

1597355324008108034

Why don’t reviewers catch basic errors and blatant fraud? One reason is that they almost never look at the data behind the papers they review, which is exactly where the errors and fraud are most likely to be. In fact, most journals don’t require you to make your data public at all. You’re supposed to provide them “on request,” but most people don’t. That’s how we’ve ended up in sitcom-esque situations like ~20% of genetics papers having totally useless data because Excel autocorrected the names of genes into months and years.https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review

[Winehole23]

(When one editor started asking authors to add their raw data after they submitted a paper to his journal, half of them declined and retracted their submissions. This suggests, in the editor’s words, “a possibility that the raw data did not exist from the beginning.”)

[Winehole23]

First: if scientists cared a lot about peer review, when their papers got reviewed and rejected, they would listen to the feedback, do more experiments, rewrite the paper, etc. Instead, they usually just submit the same paper to another journal. This was one of the first things I learned as a young psychologist, when my undergrad advisor explained there is a “big stochastic element” in publishing (translation: “it’s random, dude”). If the first journal didn’t work out, we’d try the next one. Publishing is like winning the lottery, she told me, and the way to win is to keep stuffing the box with tickets. When very serious and successful scientists proclaim that your supposed system of scientific fact-checking is no better than chance, that’s pretty dismal.

Second: once a paper gets published, we shred the reviews. A few journals publish reviews; most don't. Nobody cares to find out what the reviewers said or how the authors edited their paper in response, which suggests that nobody thinks the reviews actually mattered in the first place.

And third: scientists take unreviewed work seriously without thinking twice. We read “preprints” and working papers and blog posts, none of which have been published in peer-reviewed journals. We use data from Pew and Gallup and the government, also unreviewed. We go to conferences where people give talks about unvetted projects, and we do not turn to each other and say, “So interesting! I can’t wait for it to be peer reviewed so I can find out if it’s true.”

Instead, scientists tacitly agree that peer review adds nothing, and they make up their minds about scientific work by looking at the methods and results. Sometimes people say the quiet part loud, like Nobel laureate Sydney Brenner:


I don’t believe in peer review because I think it’s very distorted and as I’ve said, it’s simply a regression to the mean. I think peer review is hindering science. In fact, I think it has become a completely corrupt system.

[Winehole23]

(Only one of Einstein’s papers was ever peer-reviewed, by the way, and he was so surprised and upset that he published his paper in a different journal instead.)

[leemajors]

This is veering into Weinstein territory, tbh. I think Open Science is a step in the right direction, though

[Winehole23]

This is veering into Weinstein territory, tbhwho is Weinstein?

[leemajors]

who is Weinstein?

Eric and Bret

[Winehole23]

Eric and BretIs there a genetic link to the IDW here?

[leemajors]

Is there a genetic link to the IDW here?

Yup

[DarrinS]

Especially true for garbage academic papers in fields of grievance studies that infect today's university


https://www.youtube.com/watch?v=kVk9a5Jcd1k

[pgardn]

Conferences and networking with the right people who do good work.
My experience. Journals for specific areas will take garbage on ime. In my experience.
The journals themselves might want to become “popular” at the expense of accuracy.

In any event and off subject:

The government plays a central role in funding basic fundamental research that “for immediate profit” companies don’t.
I thought this should go under the MAGA banner. Then I came to the realization MAGA was interested in dictators and wanted nothing to do with basic science truths. Or any basic truth.

[leemajors]

Is there a genetic link to the IDW here?

Sorry, I didn't mean this author is part of the IDW, just that the perspective is similar to what Eric crows about, DISC etc. It's not perfect, but I think if an effort is made to check out the paper, who peer reviewed it, how the studies were conducted you can figure out what happened. Open Science is a great movement because all of the previous is not locked behind paywalls.

And, predictably, a Darrin rears its ugly head

[Winehole23]

Yupwhat's the link?

Is the author an ivermectin sorehead with an axe to grind? Seems to me one can arrive at a conclusion like this from many directions -- Paul Feyerabend comes to mind.

https://theanarchistlibrary.org/library/paul-feyerabend-against-method

[Winehole23]

Sorry, I see your reply now, leemajors.

[Tyronn Lue]

What's the alternative other than open source? I think collective review is fine as long as one can discern valid critique from noise. Eventually people not entrenched in the process are required to have faith in the review, or simply accept the risk of accepting/denying the research.

[pgardn]

What's the alternative other than open source? I think collective review is fine as long as one can discern valid critique from noise. Eventually people not entrenched in the process are required to have faith in the review, or simply accept the risk of accepting/denying the research.

What I wrote and Winehole had written a bit further on is huge imo.
Also when different labs have tried essentially the same thing and you read or correspond its easy to call BS.
Its kind of built in when a group finds something important because its very likely other groups are doing somewhat the same thing.
In my experience anyway.

Also State Health Dept play big roles in reviewing submission to start research. For example, some medical Doctors are notorious for doing a very shitty job of filing for permission for cancer research that involves people. They have a horrible submission. The PHDs/ medical doctors do a much better job and the health dept knows when they get submissions from these groups they are more likely to be very well done and get the ok. This comes to me from a friend who does this work in Austin. But this type of research is closely looked at because it involves people. Other types that dont involve people not so much. But they still have to submit for a grant of some sort. So that is another problem area, the review of submissions for a grant is laborious.

Specific science communities are likely to know the leaders in the field. But it still does not stop top notch groups from getting lazy or up and coming groups not getting the proper recognition as they are not well known yet but do good work. Physical face to face conferences are very important imo. Without the bragging bs. You go to these things with a purpose, not to get away for drinks and dinner.

[Winehole23]

Tangential


More than 40 leading scientists have resigned en masse from the editorial board of a top science journal in protest at what they describe as the “greed” of publishing giant Elsevier.

The entire academic board of the journal Neuroimage, including professors from Oxford University, King’s College London and Cardiff University resigned (https://twitter.com/chrisdc77/status/1647971370473607169?s=20) after Elsevier refused to reduce publication charges.

Academics around the world have applauded what many hope is the start of a rebellion against the huge profit margins (https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science) in academic publishing, which outstrip those made by Apple, Google and Amazon.

Neuroimage, the leading publication globally for brain-imaging research, is one of many journals that are now “open access” rather than sitting behind a subscription paywall. But its charges to authors reflect its prestige, and academics now pay over £2,700 for a research paper to be published. The former editors say this is “unethical” and bears no relation to the costs involved.

Professor Chris Chambers, head of brain stimulation at Cardiff University and one of the resigning team, said: “Elsevier preys on the academic community, claiming huge profits while adding little value to science.”

He has urged fellow scientists to turn their backs on the Elsevier journal and submit papers to a nonprofit open-access journal which the team is setting up instead.
https://www.theguardian.com/science/2023/may/07/too-greedy-mass-walkout-at-global-science-journal-over-unethical-fees

[Ef-man]

Tangential

https://www.theguardian.com/science/2023/may/07/too-greedy-mass-walkout-at-global-science-journal-over-unethical-fees

Eh, in the academic world, it is publish or perish.

However, they have the option of using less prestigious journals at lesser or no cost, but it is always up to them and/or their sponsor(s).