Over 120 Science Journal Papers Pulled for Being Total Gibberish

Second thought: The peer-reviewers of those papers have a lot of explaining to do.

Third thought: I hope one of my kids goes to MIT. It sounds like they have fun there.

Hey now! We had a good laugh at this at work.

I have NIH grant review on Wednesday. Spent all weekend writing critiques for a dozen applications, and, I have to say, for some of them random gibberish would have been an improvement.

Well that sounds like a fun weekend.  

If you look at the article the link is based on, it turns out all of the 120 papers were published in conference proceedings in China.  Even though they're supposed to be reviewed, not quite the same as getting them into a peer-reviewed journal.  Still, at least now I know where to turn if I'm desperate to publish a paper...

Not a fun weekend, unless you count staying in sweats, drinking large volumes of coffee, and not leaving the house as fun. But it's done, and I wasn't the last one submitting!

There's always the Journal of Irreproducible Results.

I don't know a lot about this story, but I do often have questions (and some skepticism) of the peer review process.

I think it's such a subjective term that too much weight is attributed to the term "peer reviewed".  I always like to follow the whole AGW discussion and one side throws out "peer reviewed" study proving X and the other side responds with "peer reviewed" study proving completely the opposite.

I'd say that the context of this article is to discredit the peer review process as a whole as "gibberish", but as Drew pointed out many of the articles were in a very obscure journal so it's really not proving that point at all.

Don't read me wrong, I'm not saying the peer review process is bad, but I think it gets used out of context a lot outside of scientific circles.

Peer review is only as good as the process and people in place to manage it. If you have reviewers who don't give a hoot, they're not checking references, data, or results. When peer review is set up properly it works quite well. As I said, I just finished my reviews for NIH study section on Wednesday. Each grant has 6 pages of technical material and background, and takes me about 3 hours to read and review. It's fairly thorough.They get scored, and the scores between reviewers are almost always consistent with each other. Because it's human-driven, people do make mistakes. You get tired or miss something, but that's why you assign multiple reviewers to an article or grant.

For conferences, it depends on the quality of the conference and the organizers oversight of the submitted manuscripts. Usually a paper is accompanied by a presentation or poster, so that part is pretty weird.The editors of the journals in question at IEEE and Springer are at least partially to blame, since they are ultimately responsible for content. And then there's the issue of page charges - most authors do have to pay to get articles published.

I've heard people say that peer review is like democracy - the worst system, except for all the others.  I actually agree with you that I think people sometimes put too much weight on the peer review stamp and it tends to make people less critical when reading a paper.  But at the same time, we all can't be experts in everything, and having experts in that particular subject review it before publication is one of the only filters on bad ideas, poor study designs, improper analysis, errors, etc.  Especially since it's the only time in the process when authors actually have to address criticisms of their work.  Yes, obviously the process isn't perfect and mistakes make it thru to publication, I'll be the first to admit there are flaws in the system.  But peer review is the gold standard for a reason.  It doesn't mean you should stop being critical when you read a paper, but knowing that it's been at least somewhat vetted should lend it a greater weight than say, something someone just publishes on their blog.

I think the other part about competing peer reviewed papers leading to confusion is more a legacy of how we use science than the peer review process itself.  Its perfectly fine to have two well designed and carried out, peer reviewed studies report conflicting results.  Peer review doesn't mean the reviewers go back to their lab and rerun the experiment to verify it.  It means it's (hopefully) been reviewed to make sure the experiment was designed correctly, the data were analyzed using the correct methods, etc.  In general, when you have 50% of studies saying one thing and 50% saying another it's usually a sign that 1) more research needs to be done to figure out the discrepancy and 2) not much research has been done on the subject overall.  As more and more research is done, unless it's something we have completely no idea about, usually evidence starts tipping the scales for a particular explanation.  That doesn't mean evidence can't come in against it, but it has to be taken in context.  When you have contradictory results to a large body of existing evidence, there is a very large burden of proof to show why this explanation is right but all the others are wrong.  

Couldn't agree more.

I like that too. 

The Economist ran a great article recently on the challenges of the scientific method, the limitations of peer review, and the unwillingness of scientists to try replicating experiments.  General point is that scientists are much more incentivized to publish their own research ("publish or perish") rather than "waste" time engaging in the review of others' work.

http://www.economist.com/news/leaders/21588069-scientific-research-has-changed-world-now-it-needs-change-itself-how-science-goes-wrong

Very true. Most NIH grant applications are scored on 5 criteria to give an overall Impact Score.

1. Significance

2. Innovation

3. Approach

4. Investigators

5. Environment (Facilities, equipment, etc0

Yep.  If your tenure and funding are based on what you publish, you don't repeat results.  You also only repeat your own experiments enough times that they work   Negative results don't get published even though they provide us with information.