In the Research 2.0 workshop at EC-TEL, we had a lengthy discussion about new forms of academic peer review. Among other issues, blinding was one of the topics that came up. Some dismissed blinding in favour of a completely open process, but I am not so sure about that.
I agree that blinding does a bad job of protecting the author(s), especially in close-knit communities. References to earlier work, projects, or simply the focus point of the paper often give away the submittee(s). In combination with a closed process (i.e. the reviews do not get published with the paper), blinding leverages biased and ill-founded reviews. This is bad news in a world where it often comes down to a single review whether your paper or project gets accepted.
Still, blinding satisfies a very important social function: it protects the reviewers and allows to them be honest, without fearing repercussions. After all, the author of the paper you just reviewed could well be the reviewer of your next project proposal. Therefore, it plays an important role in academic quality control.
Nevertheless, we all get bad reviews from time to time; reviews which make it apparent that the reviewer just had a bad day, and that it was not the quality of our submission that caused a reject. The question is – how can we improve this situation and how can the capabilites of the web help us in this endeavour?
A middle way
You might have noticed the Call for Papers on the IJTEL Young Researcher Special Issue, where we are seeking to attract visionary articles of PhD students in the field of Technology Enhanced Learning. When we put together the CfP, we did not only want to broaden the scope of articles we would allow for submission. We also wanted to innovate on the reviewing process. One thing that we agreed upon very early was to have teams of junior and experienced reviewers, whereby the experienced reviewers will act as mentors for their younger colleagues.
Then I brought some new forms of peer review into the discussion. A completely open, unblinded peer review was off the table pretty fast: as a young researcher, you do not want your vision being crushed in public, possibly by an authority in the field. Still, there was a common understanding that we were not completely satisified with going down the road of a completely closed and blinded review. Therefore, after some heated discussions, we settled for a middle-way: we will have the authors submit abstracts first for a closed and double-blinded peer review. Then will be put both abstracts and reviews (still blinded) on TEL Europe and open them for comments by the community. This way we want to make the process more transparent and increase the quality of reviews, while at the same time protecting authors and reviewers.
Thomas posted links some links on TEL Europe to the Open Peer Review process in the Semantic Web Journal. They took a similar approach, although they made blinding for reviewers optional and completely dismissed it for authors.
What do you think? Is it enough to make corrections to the traditional peer reviewing process, or do we need a completely new approach in the light of the capabilites of the web?