To Peer Review or not to Peer Review

from Grazia Ietto-Gillies

The debate on alternatives to the Peer Review system for the assessment of research has been going on for a little while and it is nice to see it has now hit the New York Times. It was highlighted in this blog by David Ruccio who raises the issue of how tenure and research funds can be allocated in the absence of a Peer Review system.

In  my 2008 paper  ‘A XXI-century alternative to XX-century peer review” real-world economics review, 45: 10-22, March www.paecon.net/PAEReview/issue45/IettoGillies45 I deal with similar issues and propose an alternative system to Peer Review, one that utilized the digital technologies while avoiding some of the pitfalls of Peer Review.   

Following a review of criticisms of the Peer Review (PR) system – labelled ex-ante top-down PR system – the paper proposes an alternative system: an Open Access system – labelled ex-post bottom-up Peer Comments (PC) system – characterized by the following:

• Use of Open Access sites categorized by fields of specialization for each subject. Research papers to undergo a first selection by an editor with the aim to (a) weed out crankish papers and (b) make sure that – as far as possible – they pertain to the right field of specialization. The latter point is designed to help readers as well as authors. Specialization will also ensure that real interested/expert people will contribute to it.
• For each paper published on Open Access the editor should open an electronic  ‘Comments Link’ inviting readers to send comments which – following a vetting to weed out crank or offensive contributions – will then be placed on the Link site. These open debates should be positively encouraged as a way of developing research; they are a way of recognizing that research is a social activity and the interaction of various researchers can aid progress.

The advantages of this Peer Comment system are the following.

1. It secures quick dissemination of research ideas and results.
2. It is very cost efficient because both the private and social costs associated with the Peer Review system – and discussed in the above paper – are very low.
3. The bottom-up approach is likely to give better assessment because of the large number of potential contributors against the few referees in the ex-ante top-down PR system.
4. A further advantage of the PCs system is that those who are prepared to read the relevant papers and write criticisms are likely to be people interested in the specific topic and thus their criticisms are likely to be relevant.
5. The wider dissemination of papers on e-sites has a major advantage: within a large readership and potential commentators we are more likely to have a few who can spot the occasional ground-breaking research than if we confine such a task to very few referees as in the present PR system.
6. From the reader’s perspective, there is evidence – given in the above paper – that the opportunity to read comments and debates is viewed positively.
7. As regards jobs and grants/funds allocation, the proposed system has the following advantages over the PR system: the allocators of grants and jobs – and their advisors – can rely on a wider number of potential commentators than the current system and thus will be better able to assess the impact of the paper. David Ruccio, be reassured: the Peer Comment system may lead to fairer system for the allocation of resources.
8. Lastly, the Link site for comments invites people to participate disclosing their identity rather than anonymously. The lack of anonymity has the advantage that, if someone has a brilliant idea following the reading of the original paper, s/he will not be tempted to hold it back for fear of losing attribution – as may happen under the current system of anonymous refereeing. Under the PC system they know that whatever comments they place on the site will be attributed to them. Moreover, openness is likely to lead to more positive developments and the process would strengthen the social character of research: further progress would emerge from critiques and discussions. It could, however, be claimed that the lack of anonymity discourages academics from making negative comments. This is possible; however, we should not forget that the internet interaction spans the whole globe; while someone in Britain may not want to offend co-researchers whom they are likely to meet often and/or who may have power over jobs allocation, they may be less worried about academics further afield. Moreover, academics are well prepared to stick the knife in when writing signed reviews of books why should they not do it when writing comments on other papers? It is partly a matter of culture. Once a culture of signed comments develops, then most academics will be prepared to write sober, reasonable comments.

My paper above is freely available and the editor of the journal, Edward Fullbrook, has opened a discussion link in line with the recommendation on the paper. There have been several comments; others are welcome either in that link or within this blog.

It is pleasing to see that some of the above recommendations in the paper – particularly with regard to the signed comments – seem to have been followed in the experiment by the Shakespeare Quarterly cited in the New York Times. We are told that it led to good outcomes.

It is to be hoped that more subjects, including economics, will follow the trend: the research community would greatly benefit from it. However, there are long term implications from the proposed system not least financial implications for the publishing industry. Resistance to change must be expected.

Grazia Ietto-Gillies
iettogg@lsbu.ac.uk