Peer Review Veteran: “Peer Review Has Lots Of Problems …Tends To Stifle Non-Dogmatic Thinking”

What follows is a comment by a PhD (wishes to remain anonymous) on the subject of peer-review. I’ve decided to upgrade it to a post (some editing).
============================================

Peer Review Is Not Magic

By The indomitable Snowman PhD

I usually don’t comment. But I noticed that NTZ attracted a new troll, who is just taking talking points from the support group sites and regurgitating them.

Marching

Peer-review often serves to prop up dearly held dogma.

I just couldn’t let the so often cited “holy peer review” thing pass because it’s long been a mess. Since with publication on climate science, one is required to support the dogma, or the very least not dare to question it, or risk getting fired or having your funding pulled (same thing).

I would say the appropriate term is “FEAR review”.

It’s funny to see comments from alarmists and the links to the sites they provide. Those “sites” are very often not information sites, but support groups for zealots who need to believe and to go there to get belief-support and talking points – and who then leave their bubble and mindlessly regurgitate it all later.

The “peer review” thing is one of those babbling points. As Ed Begley (in)famously said, “Don’t take it from me folks – take it from people with a ‘Ph.D.’ after their name.” Peer review isn’t magic and has lots of problems, and I say this as someone who has had regular contact with peer review for more than two decades – as an author, reviewer, and editor.

Peer review serves only to provide some minimal quality control – screening out things like bad grammar, weak background work on references and prior art, etc. It’s also an iterative process – a paper is reviewed and the reviewers have comments/thoughts for the author(s), and the process iterates – it’s actually very unusual for any paper to be “outrighted” – either accepted as-is or simply rejected outright.

It’s funny seeing the inflated claims made (by no-doubt non-participants), since the system was never built to provide the claims of infallibility that the non-participants ascribe to it. Some problems with peer review:

  • It’s time consuming, and everyone is busy. Reviewers are volunteers, and usually they are short on time (like everyone) and a request for a prompt review is one more thing on the pile.  (The best place to do peer review is in the passenger compartment of a commercial aircraft.)  So peer review tends to be quite cursory, even when intentions are good.
  • Too many academics – and they too often treat manuscripts as being a student’s dissertation and continually demand more analysis and MORE analysis; it’s amazing how much useful material never gets into the journals simply because someone wants more and more and MORE analysis/measurements done to (in the reviewer’s view) nail down every last detail to the n-th degree.
  • The process can be abused with intent – and this is true in ALL fields. Peer review tends to stifle non-dogmatic thinking (and the more academic the field, the more this happens). This doesn’t even have to be malevolent – dogma can become so entrenched that anything that disagrees is instantly branded as incorrect; given the number of dogmatic points in a wide variety of fields that have turned out to be totally wrong, you’d think that there would be interest in reforming the process, but that hasn’t happened yet.  And, unfortunately, the process is regularly abused to suppress opposing views – or views/ideas that are merely “competitive” to the reviewer’s views (or “side”).

I’ve actually seen numerous instances where the same (as always, anonymous during the review process) reviewers who blocked a paper from being published then state publicly that those opposing ideas are obviously wrong because “they were unable to get through peer review.”

This general problem is finally getting some attention – along with other problems with the actual realities of “science.” There’s an excellent-and-timely article that came out this month that’s worth reading in detail, which discusses “peer review” and other issues that are undermining the quality and usefulness of “science.”

http://www.firstthings.com/article/2016/05/scientific-regress

Take the time to read that whole article; it’s lengthy but worth it.

 

18 responses to “Peer Review Veteran: “Peer Review Has Lots Of Problems …Tends To Stifle Non-Dogmatic Thinking””

  1. AndyG55

    I have done peer-reviewing.

    It is time consuming.. and since it is often on “new stuff” there is no way one can always say “this is wrong”, or this is right”

    That is where the whole idea of a peer reviewed article as being “correct science” falls down.

    ALL peer review means is that the reviewers think it is worth putting forward into the scientific literature.
    (although even that can be argued against in many cases)

    That is the SINGLE and ONLY purpose of peer review.

    And with the lock-out by the AGW peer-review guardians, peer-review in so-called climate science, has become all but meaningless.

    1. yonason

      Precisely!

      Or, as Frank Tipler writes:
      http://johndfenton.com/Documents/Tipler03-PeerReview.pdf

      “…prior to the Second World War the refereeing process, even where it existed, had very little effect on the publication of novel ideas, at least in the field of physics. But in the last several decades, many outstanding scientists have complained that their best ideas—the very ideas that brought them fame—were rejected by the refereed journals. Thus, prior to the Second World War, the refereeing process worked primarily to eliminate crackpot papers.

      Today, the refereeing process works primarily to enforce orthodoxy.”

      That’s bad enough. But when you add to that a deliberate enlistment of it in the service of advancing a flawed agenda, you take an annoyance and turn it into a recipe for disaster.

  2. Harry Dale Huffman

    Ordinary people need to understand that there are basically two kinds of scientist when it comes to the peer-review process: 1) Principal Investigators, and 2) Everyone else, including a) Research Associates, who are simply employees, without tenure, who can be “terminated”–fired–without cause, and b)”expert” (a meaningless word these days) consultants from other fields or other institutions (also known as “pals”, as in “pal review”, get it?).

    When the subject of peer review comes up these days, I trot out my 2003 letter on peer review to the OMB, when it requested feedback on peer review (see also all 2003 Public Comments on Peer Review that they received).

    My bottom line, from experience: “peer review … is a system akin to that of independent feudal lords or warring tribal chiefs”, i.e., those “Principal Investigators” whose main concerns are keeping their research grant money coming in year after year, and being taken as the unquestioned authorities in their fields (which mainly requires they never allow even the possibility they could be wrong about the “settled science”, as the “Emperor” must be peerless, in their own minds). As you can see, the fundamental mindset is that of “survival of the fittest and I must be the fittest”–and (LEARN THIS, people) that mindset (also known as “the struggle for scarce resources”) is NOT the way to the TRUTH in science (despite what you may think you have learned “the hard way” in life). True science–uncovering the truth–is NOT a competition (or a social construct, as in “postnormal science” or even a “consensus”), in any way, shape or form. It is, fundamentally, a single mind, learning something true (cause and effect) about the world–period.

  3. R2Dtoo

    Peer review is a very hit-and-miss practice. Many scholars take the process seriously and do as good a job as possible at being fair to the author, and maintaining the level of scholarship necessary in their field of study. I reviewed hundreds of papers as a reviewer and editor, and had about a hundred opt my own reviewed over 35 years. If all players do their jobs the process works reasonably well most of the time. It usually took more than a year to get through the process, and this could be followed by waiting in a queque for publication after acceptance. Many of my own papers were improved owing to good comments from reviewers. Although it generally wasn’t required, I always signed reviews that I did for others.

    The only downside I encountered was the common practice of what I termed academic arrogance. Papers often were rejected because of “who” submitted it, or “where” it originated. Certain departments in certain universities “protected” their reputations by favouring papers submitted by their resident colleagues and students. It was particularly difficult to have papers published if you were from a university that didn’t specialize in a particular subject area, or if you were not well known in the field. Here, the politics among universities could override the quality of the work.

    I have been following the climate science publications for about 10 years. I find it hard to believe the tripe that gets through peer review. Thirty years ago many of these missives would have been rejected outright. Universities in general have lost their way regarding scholarship, and, because they provide most of the reviewers for peer review, the standards have dropped accordingly. Fortunately, not all disciplines have succumbed to this practice. It is, however, far too common.

  4. Stephen Richards

    I have been peer reviewed. I found it a useful process for crossing Ts and dotting Is but not for correction of the science within.

  5. yonason

    “Universities in general have lost their way regarding scholarship…” – R2Dtoo

    After reading what Lubos Motl wrote, I am convinced that insanity is the new normal. It is so much worse than I thought. For what I’m referring to, scroll down to “Harvard student leaders must be bi-sexual or trans-sexual” here:
    http://motls.blogspot.com/2016/05/trump-isnt-responsible-for-passengers.html

    Also, he has another post on that topic here:
    http://motls.blogspot.com/2016/05/a-revolution-against-pc-tyranny-at.html

    S.E.s (social engineers) – ” Woo Hoo! Let’s party like it’s the end of the world!

    O (sane observer) – “But the world isn’t ending.”

    S.E.s – “Not if we can help it! Woo Hoo!”

  6. Reasonable Skeptic

    Not being a scientist but a layman (taxpayer/voter) I can’t ever truly comment on these topics directly, so I let others do that.

    However, it is very easy to find “sciency” things that alarmists stick to that fall apart under basic examination.

    Here is a good example of that I mean.

    The science is settled:
    – but Karl et al (2105) demonstrated that our very best information is still not reliable.

    I take their position, and their paper and demonstrate that you can’t believe the science is settled when such a cornerstone product, derived from the highest quality sources (spatial, temporal, precision and accuracy) can be radically altered by one paper.

    1. yonason

      Logic isn’t an activist’s strong suit. Or, as Judith Curry writes:

      “Color me ‘unconvinced’.”

      https://judithcurry.com/2015/06/04/has-noaa-busted-the-pause-in-global-warming/

  7. DirkH

    From the linked article
    “The Whiggish view of scientific history is so dominant today that this possibility is spoken of only in hushed whispers, but ours is a world in which things once known can be lost and buried. ”

    Sounds like the job description of Gavin Schmidt.
    Given that he is a government employee, and governments have used the services of propaganda men since long before Hearst and Bernays, government science should be mistrusted like any other government utterance.

    “All that the state has he has stolen, and everything he says is a lie” – Nietzsche

    1. yonason

      “Sounds like the job description of Gavin Schmidt.” – DrikH

      Speaking of things that need to be lost and buried… 🙂

  8. stan stendera

    Peer review is just (in my layman’s view) another excuse by alarmists to ignore truth. Ideas stand on their own ultimately. Blocking truth by appeals to authority (which is what peer review is) leads to a pent up demand for this truth. When the dam breaks and the truth becomes known heads may very well roll. This is why I believe when the truth about the global warming alarmism finally becomes known the political and academic fallout will be catastrophic. Fortunately the victims of the fallout will richly deserve their fate.

  9. Derek Colman

    There was the well reported case of the medical scientist who, in order to test the process, submitted a prank paper which duly passed peer review and was published. He then withdrew the paper and owned up to the prank. Also in medial sdience there have been several papers submitted with malicious intent which later were discovered to be faked. I think the point is that reviewers do not have the time or resources to replicate the work, so it’s a case of it seems alright, and the author is a fellow scientist, so give them the benefit of the doubt.

  10. Frederick Colbourne

    “It’s time consuming, and everyone is busy. … So peer review tends to be quite cursory, even when intentions are good.”

    This was impressed upon me earlier this week on reading a remark in a warmist blog that discounted the potential of GCRs(galactic cosmic rays)to affect climate via modulation of cloud formation and thus albedo. The writer claimed that the Laschamp geomagnetic excursion 40,000 years ago falsified the GCR theory because it was associated with a warming period (interstadial).

    It took me about three hours to track down a series of papers based on data with higher resolution than earlier studies of the Laschamp Event (LE). As it now appears, interstadials 9 & 10 (warm period) bracketed a brief stadial (cold period) that matches the time of occurrence of the LE.

    Thus, the Laschamp Event actually supports the GCR theory.

    But how many reviewers of claims and counterclaims have three hours to locate studies that update prior errors in received wisdom?

    To do so requires a skeptical turn of mind that is discouraged by the guardians of normal science. We have the science historian Thomas Kuhn to thank for explaining how Big Science works. Unfortunately, too many people believe that Kuhn was a philosopher who established norms to guide the proper imposition of consensus.

    When I did my first master degree, the head of the department criticized me for questioning accepted knowledge. He objected to my term paper on population trends in Spain because “Catholics don’t practice birth control” and docked me a grade, which made my GPA 3.9 instead of 4.0.

    In retrospect I regard it an honor to have been castigated by such a man.

  11. sod

    Let me start with a simple sentence: i agree with all there major points made in the original post above.

    But looking at the latest discussion, i also want to point out some benefits of peer review:

    people with some serious knowledge in a field, sort out papers and ideas, which are obviously and utterly false.

    In the latest post, multiple people present multiple theories, which in part contradict each other:

    CO2 effect is strongest in summer.

    Winter warming is mostly caused by heating houses.

    the angle of impact influences the CO2 effect.

    Land does not absorb short wave solar radiation and turn it into heat.

    greenhouse warming to be most evident in the moderation of low temperatures, not an increase in highs.

    I have an opinion on most of these, but i am not a climate scientist. Those i expect, could easily throw out those ideas, which are obviously false. and that is an important part of peer review.

    In mathematics, papers and professors are constantly flooded with laymen papers, who claim to have solved problems which are known to be unsolvable or which are unsolved for hundreds of years. without “peer review”, papers would constantly be filled with total garbage.

  12. BobC

    ‘The process of peer review has been described as “unjust”, “usually ignorant” and “frequently wrong” by no less than the editor of the eminent British journal The Lancet.’

    http://www.abc.net.au/science/articles/2000/02/23/102612.htm

  13. GM

    Another problem with peer review is it is easy to get around. Start a journal, get your buddies to review your work, and presto! Another way to demonstrate the folly of peer review is to point out that the Bible was peer reviewed. A committee decided what books would be included and which would be discarded.

  14. WMasters

    As the Catholic Church realized in the 5th Century, they needed a Professional corps of men to copy the older Bibles. Laypersons were using different interpretations of the ancient Greek texts that were frequently wrong and, they were also adding things. Science like-wise needs a Professional corps who only do Peer Review. Peer Review is vital to the Scientific Method. We cannot learn if new ideas are not presented and discussed. There needs to be a recognition that Peer Review is a right not a privilege. It is a right of the People to hear the views of those who may be targeted by the mainstream, or those in control.
    In my own case I’ve been blacklisted by the peer review process since I submitted my anti-AGW paper THE STEFAN-BOLTZMANN LAW, THE CONUNDRUM OF GREENHOUSE GAS WARMING. Which points out, as we all know, that Greenhouse Gas molecules are opaque to most Infrared radiation entering the Earth’s atmosphere. Ergo they block entering sun heat from reaching the earth (currently 67.8% according NASA satellite and ground stations). Increasing CO2 levels means more sun heat will be captured and blocked out by these additional molecules of CO2. This means the Earth will receive less heat with greater amounts of GHG molecules in the atmosphere. The S-B Law tells us that the emitted heat of any body is equal to it absolute temperature taken to its Forth Power. Ergo, a 1% drop in the Earth’s surface temperature will create a 4% drop in emitted heat. A 2% drop in temperature will create a 16% drop in emitted heat etc…
    Not only was this denied publication by my other papers were TAKEN DOWN!!!
    I’ve been denied publication in Peer Review ever since.

    This phoney AGW scare is not the first time Nazism has occurred in the sciences, but this is the first time it has reached this kind of proportion. Most grants offered today are tied in with proving AGW in some way. Science funding is literally being held for ransom.

    In hopes of putting an end to this new Nazism I’ve filed a suit at law in California Superior Court against some AGWers It is numbered “15K07031” Masters vs Uygur eta al. The Defendants have defaulted rather than fight the case. The did file a response but 25 days late. They did not challenge the science nor deny the mathematics, they instead unable to find a scientist who will testify to lies under oath in a court, they claim I was not harmed and have no standing. The case is set for this December. With the money from this case, I will file a second against President Obama, the UN’s POCC, as well as NASA and the NOAA, which have joined in the science Nazi movement. I will also include all the major leaders in the phony AGW movement as well.
    I will seek a court order forcing them to admit they were lying and ordering them to stop lying about AGW and to explain to the American People who the S-B Law proves that AGW is impossible.

    At this time, all major news agencies have been informed of the case but there is a complete blackout in the news media on this issue. I am now convinced that the Courts are Science’s only hope of ending this Nazi movement against truth in science. Politics is politics, and everything a politician does is political, but science cannot be come political and still serve science.

    I understand the funding requirements of science and am not criticizing my colleges for doing what they must to survive, but when politicians take over science we must march into the Courts and put them down like a dog with rabies.

    It is time to put an end to this, AGW nonsense, it is hurting science research and the advancement of science itself.
    We cannot become politicians, by doing so we are destroying the trust the American People have in science to improve their lives an help them. Every day I read or hear ordinary people say Scientists are lye and are politicians in white coats. This cannot continue, for Science sake we must go to war! The Nazis must be defeated.