- A Glimpse behind the Scenes of Scientific Practice
- About the Author
- Subscribe to our Newsletter!
A Glimpse behind the Scenes of Scientific Practice
Over the past couple of years, the term peer review has become increasingly popular. Whenever a claim has been “peer reviewed”, it’s received as the voice of the ultimate authority. As if the Olympian Gods themselves had declared it to be true. Yet little is known about how the peer review process works and what flaws it has. Because in fact, peer review is much more subject to the whims of humans than its reputation suggests.
I have been involved in scientific publishing for over a decade and often get the same five questions. Here are my answers to these questions, which also explain why I think it’s time to acknowledge that the glorious reputation of peer review itself needs a proper revision.
Question 1: How does the peer review process work?
Peer review was established in response to the need for having scientific claims backed by an authority. While in ancient Greece, people asked priests or oracles for the Gods’ advice, scientific practice nowadays relies on peer review for a claim to attain a comparable status of allegedly absolute truth.
The peer review process is a standardized procedure that assesses the quality of a claim on the basis of more than one expert opinion. In scientific research, peer review is the parallel evaluation of a scientific manuscript by two or more experts – so-called peers. Any kind of scientific manuscript can be submitted to peer review, like books, journal articles, or conference abstracts. The editor and the editorial board, somewhat comparable to Zeus and the Greek Gods, then take a decision on the quality and truthfulness of the claim proposed in the manuscript.
Let’s take journal articles as an example. Researchers who submit a manuscript to a journal usually pursue a strategic career goal with their paper, so they select the most prestigious outlet. Peer-reviewed journals are generally more prestigious than other outlets.
Upon submission, the editor examines whether its quality and topic match the purpose of the journal. If they don’t, the editor rejects the paper right away; that’s a desk rejection. This is the moment a paper should either be revised or else submitted to a more suitable outlet.
If the editor considers the paper to be suitable, they’ll select the most qualified reviewers and ask them to assess the manuscript. Depending on these reviews, the editor will then either reject or accept the paper expecting major, minor, or no revisions.
Once the paper has been accepted, it goes into production. The copyeditor ensures the text follows the journal’s style manual, which may require another round of revisions. Most journals, however, only accept papers that already comply with the manual. The journal’s typesetter then takes care of the layout.
If the paper is rejected or if the author doesn’t agree with the requested revisions, the author is free to withdraw the paper and submit it to another publisher. This is common practice, though it annihilates all the work that has gone into the review process so far. When experts are scarce, this may entail that the next editor will not be able to find a reviewer.
Question 2: How does the decision come about?
Reviewers scrutinize the paper roughly answering the following questions:
Does the paper discuss relevant findings and theories that have shaped the field so far?
Are methods and data appropriate to answer the research question?
Is the methodological procedure sound?
Does the paper contribute to advancing the field?
Is the language professional and factual?
After receiving the reviewers’ comments, the editor sends the final verdict together with a summary of the reviewers’ comments to the author. This process can be painfully lengthy and end in the study being entirely outdated by the time it’s published. Still, imagine that in disciplines as traditional as linguistics, manuscripts used to be shipped around the world to reach the right reviewer!
In “blind” peer review, neither the author’s nor the reviewers’ names are disclosed. When a field is small or a person is famous, however, it’s easy to guess who the involved people are.
When I submitted my first manuscript to a prestigious journal, my field was so small and my data so needed that it was hard but crucial for the editor to find qualified reviewers. In the end, I had two rounds of reviews with a total of six reviewers from different disciplines.
Some of the reviewers couldn’t be bothered about their review being anonymous and just left their name tags in the document. I was grateful for their benevolent and helpful comments, particularly because at the time, their expertise was invaluable to the unexperienced researcher that I was. Still, due to the idiosyncrasies of my field and the enormous backlog of the journal, it took four years to see my first peer-reviewed paper in print.
It is unusual for a paper to be accepted without any revisions. Almost every paper requires some kind of revision, even if it’s just to adjust the text to the publisher’s style manual. It’s essential to follow the style manual because in written academic language, everything has a specific function – from the correct meaning of each word to special symbols and fonts. The latter actually add information to the content. Italics, for example, are used to mark words under discussion or words in a foreign language, while bold font is used for emphasis. And hyphens and dashes mean different things and are often confused.
A vast majority of scientific papers across the world are published in English. Most of them were written by non-native speakers or speakers of non-standard varieties of English. Still, either British or American standard English is expected, and papers often get the comment that they “should be checked by a native speaker”. The comment is unprofessional; what is meant is that a copyeditor should revise the text.
Question 3: Who are the reviewers?
The reviewers are chosen by the editor and the editorial board. They should be established specialists with several years of research experience regardless of where they work; what counts is their expertise. The combined knowledge of all reviewers is expected to ensure that the research is relevant, methodologically sound, and factually correct.
There is, however, no superior body – nor Zeus and the Greek Gods – to guarantee that these criteria are met. This is what the editor and members of the editorial board have committed themselves to do.
Accordingly, the quality of the reviewers’ expertise varies, as does their willingness to share it. Benevolent experts can substantially contribute to improving a paper by adding important comments and suggestions. They are, however, not obliged to do so.
If an author does research in a field that is dominated by certain reviewers, it is smart to prominently cite their names in the paper. In my field, there used to be two main camps that had very strong feelings about each other, which meant that the choice of the journal and the names cited was part of the strategy to get the paper published. Peer review and academic publishing are a lot about navigating politics.
Question 4: Who pays for the review?
Reviewers don’t receive a single cent for their expertise and their work in this process. Indeed, taken together, the voluntary work put into peer review would sum up to over USD 100,000,000 a year. Yet voluntary peer review counts as “service”, and whatever goes under “service” is included in the job. In other words, unless reviewers are employed at a private institution or company, their salaries and “service” are paid by taxes.
This is why most reviewers are frustrated: They are barely paid for their expertise. And since frustration is high and payment low, it is understandably difficult to get high-quality reviews. That’s why an increasing number of voices claims that peer review is actually dead.
Question 5: Who is Reviewer Two?
Peer review often relies on two to three reviewers. If one of them is condescending, he will be called “Reviewer Two”. Among researchers, Reviewer Two is the buzz killer. He’s the one to tell you to add another study on an entirely different dataset to validate your claim. He’ll tell you that studying language contact in Johannesburg isn’t representative enough and that you should also conduct the same study in São Paulo and Manila.
Also, he’ll probably tell you your claim is neither novel nor worth anybody’s time, and most importantly, he’ll tell you to cite his paper because, of course, without his paper, yours is worthless (which is how you’ll figure out who he is). He may be right in some points, but his tone may also induce the author to withdraw the paper and submit it to another journal – which gets them back to square one.
Reviewer Two’s mischievousness may also induce some authors to bend over backwards just to see their paper published. This attempt at pleasing Reviewer Two is one of the mechanisms that has turned academic language into what it is today: an unspecific, abstract, all-encompassing-while-not-making-any-compromising-claim kind of language with a justifiably bad reputation.
In sum: Is peer-review stuck in a dead-end?
I think so. Peer review can only be as good as the people involved. Unfortunately, a substantial part of the peer review process is guided by interests that have little to do with scientific research, and as long as this situation persists, the decline will continue.
Certain disciplines, above all the most innovative fields of technology, have already started to shift away from this cumbersome publication process. They favor conference proceedings, which are published immediately after the most important, industry-funded conferences have taken place.
If peer review is to persist, it will need profound revisions. We could start by questioning its glorious reputation.