There is much outcry about the increasing competition in scientific research. Yet, I do not hear comparable outcry about the increasing competition in the Olympic 100-meters dash. I see competition as a very powerful driving force; whether it drives positive or negative changes depends on our metrics and the system. Unlike the metrics for the Olympic 100-meters dash, the metrics for research performance seem rather poor. It is the metrics that make all the difference between competition driving better research or undermining the research enterprise.
The peer-review is the central metric in our current system for scientific research. Yet, sometimes PIs are embarrassed to show the reviews to their students who had done all the work. How can a magazine or journal be any good if it uses poor reviews to judge what to publish and what not to publish ? It cannot!
Evaluating research results will never be as simple as measuring the time for a 100-meters dash. However, I think that is not a justification for the sorry state of our current system and not a reason to despair. We can improve the quality of the peer-reviews by making their contents public, as some journals (such as molecular systems biology, eLife and others) already do. I cannot think of a good reason for other journals not to adopt this good practice. However, most journals refuse to do so. I think that publishing peer-reviews will help increase their quality and will provide useful background and additional (hopefully thoughtful) discussion of the research for interested readers. The content of the peer-reviews will also give us a useful indication for the quality of the journal, perhaps much more useful and meaningful than the summary statistics (first moment) of its visibility and the availability bias.
I would love to hear arguments for and against making the contents of the peer-reviews openly accessible for published papers. Stay tuned for a post on whether the reviewers should be anonymous. What are your thoughts ?
Slavov Lab Blog 
The PI who was embarrassed was me. I’ve thought about putting those reviews online (maybe in Niko Grigorieff’s rejection repository). Technically, I think we’re not allowed to as the journal owns the rights to the commentary.
After being in science a while, you get pretty inured to criticism, even when it’s rude and un-constructive. For our trainees though, this is really shocking. Especially after all their hard work doing the experiments and getting the first paper together. They deserve better.
Anyway, the criticism in question centred around the fact that this anonymous person had tried to do the method that we were using and had failed to get it to work. Therefore our paper must be wrong! Never mind that the journal had carried several papers per issue for the previous five years using the method. The review also had the knockout punch “this paper should not be published in ANY journal”.
As I mentioned in my comment, I complained to the journal about the quality of the review and suggested that they send the referee a copy of “Any Jackass Can Trash A Manuscript” by David Drubin (http://www.molbiolcell.org/content/22/5/525.full).
I also showed the review to several key people in my field, who all identified the same person as author. The paper was accepted after minor corrections at a different journal (of similar prestige) from the same family of journals. It’s been reasonably well-cited since. It just went to somebody who’d lost sight of what peer review is about.
LikeLiked by 1 person
Thanks for filling in the details for this unpleasant experience. It is a sad reflection of our current system that many peers write reviews as though they have lost sight of what peer review is about. I think that the most potent cure for this disease will be more transparency and accountability. These may come only from making the peer-reviews public.
LikeLike
I agree with you that journals should generally publish peer reviews. Some of the reasons are:
It increases transparency of the publishing process.
I think it would encourage reviewers to write more objective and reasoned reviews.
It could serve as a kind of qualitative article-level metric and part of an improved alternative to the IMO wrong practice of using JIF as an article-level measure of quality.
I have previously discussed open review here: http://thomas.arildsen.org/2013/08/01/open-review-of-scientific-literature/ and also thought about the anonymity issue: http://thomas.arildsen.org/2013/09/02/openness-and-anonymity-in-peer-review/ plus more thoughts on possibilities of open peer review: http://thomas.arildsen.org/2013/09/16/publishing-models-with-open-review/
LikeLiked by 1 person
At PeerJ we believe that transparency in peer review will improve the process ofr authors, readers, and (yes) reviewers.
We give reviewers the option to name themselves on their peer review (>40% do) and we give authors the option to then make their entire peer-review history public on their publisher paper (>80% do). A blog post with this data is at: http://blog.peerj.com/post/100580518238/whos-afraid-of-open-peer-review
In our opinion, making all peer review activity fully open is the ideal situation, and we are working towards the day when that will be the norm, not the exception. In the meantime, if you want to make your review process fully open then submit to PeerJ (and join thousands of satisfied authors: http://blog.peerj.com/post/103451626748/results-of-the-second-peerj-author-survey
In the ‘high quality’ journals you refer to, although we might assume they do ‘good’ peer-review, we have no evidence of that fact. Isn’t science all about evidence?…
LikeLiked by 2 people
PeerJ sets a great example, and I hope that it contributes both to the rise of PeerJ and the spread of the practices.
LikeLike
You are absolutely and entirely correct. Peer reviews are so important in our system that they should be published, visible for anyone (i.e., not behind a paywall, before Elsevier gets any funny ideas), and not anonymous, as I’ve argued in http://osc.centerforopenscience.org/2014/05/15/anonymous-peer-review/
The only way to keep reviewers focused and fair is when they know that they are accountable for their reviews. One of the reasons that journals are reluctant, I suspect, is that if reviews were visible (and/or non-anonymous) it will be even harder than it already is to find reviewers.
Finally, anyone coming up with the “whether transparent non-anonymous reviews are better is an empirical issue and you don’t have the data neener-neener”-argument: yes, you are right, and we should do studies on that, but in the mean time, have a look at open reviews, for example those in PeerJ or BBS, and compare them with the typical anonymous reviews. The difference is striking. Or do an experiment with yourself: after your next anonymous review of a paper you have strong opinions about, save it, and then write it again while imagining that this review is going to published together with the paper, with your name under it.
LikeLiked by 1 person
Every time I review a paper I ask myself whether I am willing to (i) make the review public and (ii) whether I am willing to sign my name. The answer of (i) is always without an exception yes, while the answer of (ii) is sometimes hesitant.
I do think that in general peer-reviews should be non-anonymous but I can see reasonable arguments against non-anonymous reviews in some cases. I agree with you that finding peers to write non-anonymous reviews can be harder. However, making the reviews open should not contribute to such reluctance. It is admittedly the smaller step and it is incomplete. However, I see it as being much easier to make with no reasonable arguments against it. Once we make public peer-reviews the norm, it will be easier to take the next step.
LikeLike
Pingback: Twitter Open Access Report – 2 April 2015 |
My experience with Molecular Systems Biology public open access peer reviews has been in so far great. I think all journals should adopt it. I want to add than, in addition to generating in average more constructive reviews, such reviews are also a great resource for readers. I find that peer reviews in MSB often contain additional valuable pieces of reasoning about the work and help substantially in putting the work in context. Since these are usually 3 or 4 authorities in the field spending considerable time evaluating the work, I do not see why we should throw away all this critical thinking instead of sharing it. Public peer reviews are similar a good chat over a paper during coffee break with the top experts in the field..
LikeLiked by 1 person
Pingback: Which publications are citable? | Forest Vista
As a nice bonus, open per review would also help end so-called predatory journals.
LikeLike
The Committee on Publication Ethics is running an open forum discussion topic on “Who “owns” peer reviews?”: Please weigh in. Folks are asking such basic questions, as for example, “who will benefit from open reviews”?
-F.
LikeLike