Blinding reviewers
Just
because you are not paranoid doesn't mean that they are not out to get
you. People sometimes feel like that if they try to get their papers into
certain journals, or abstracts into large and important scientific meetings. At
the centre of it all is the peer review process. We accept it is the best we
can do, but a study from the US [1] has startling insight into how blinding can
affect abstract acceptance.
Study
The
study examined submissions for the American Heart Association meetings in 2000
and 2001, where review was open, and 2002-2004, when it was blinded. It was
really well done, and the outcome was whether or not a study was recommended
for acceptance using standard procedures. Acceptance rates were then judged
before and after blining was introduced according to different criteria: US or
non-US, English on non-English speaking country, Prestigious or non prestigious
institutions, sex of author, US government agency, industry, and by basic or
clinical study. Each abstract was evaluated by 8-10 reviewers.
Results
There
were over 13,000 abstracts submitted annually, with an average acceptance of
29%. About 40% were from the US, a quarter from highly prestigious
institutions. Most (85%) of reviewers were US-based. We might expect higher
acceptance from prestigious institutions compared to non-prestigious
counterparts, for instance, with open compared with blinded review: if bias
were a factor, the gap should narrow with blinding. Table 1 shows that with
blinding the gap narrowed by 9%. There were similar changes for abstracts from
US government institutions, for US, and for non-industry abstracts. Only sex of
author was unaffected.
Table 1: The effects of blinding reviewers on abstract acceptance rates - differences in pairs of separate criteria
| Criterion | |
| US govt vs non-government | |
| Prestigious vs non-prestigious | |
| US vs non-US | |
| Not industry vs industy | |
| English vs non-English |
Comment
However
balanced we like to think we are, the simple fact is that we are persuaded that
research from government agencies, or prestigious institutions, or not coming
from industry, is better. When we are deprived of that information, we make
different judgements.
Reference:
- JS Ross et al. Effect of blinded peer review on abstract acceptance. JAMA 2006 295:1675-1680.