The necessity of peer reviewing

One question I often ask myself, which has sparked a lot of discussion among fellow researchers and academics, is the necessity of peer-reviewed publications. Do we actually need them? I still have the feeling that just by thinking of such a question, I automatically become the target of those who strongly suggest “yes, of course we do” without any second thought. I have already been shelled with heavy critique just for allowing this sacrilegious thought to penetrate my mind; therefore, the purpose of this article is not to prove the opposite. Furthermore, I do not wish to engage in an argument with people who support the idea that academic knowledge and scientific information should be reviewed, critiqued, filtered, and approved by the board of wise tribesmen. Given that the board would probably object if I were to present my own views in any conventional periodical, I decided to do so online, taking advantage of the freedom the web provides. For what is to be said further on, I need to apologise in advance.

Before I begin discussing my thoughts on the subject, I need to say that so far, mankind has managed quite well in chopping down as many trees as possible to produce paper for people to express their thoughts openly to the public. One may argue that human views and knowledge are worth more than the lives of trees, but, having read a lot of abstract academic thought and theory in my life, particularly in archaeology, I think the lives of trees are more valuable than we already believe, and trees should not be wasted just for producing paper. There have been many occasions when I have reminded fellow researchers and academics that every line of thought we produce is equal to the life of another tree. I guess I am a bit oversensitive on this matter, and remind people to act with consideration for the environment. To be honest, the unnecessarily complicated language, the blurry arguments and the abstract thoughts I have encountered in many books and articles in archaeology cannot match the value of the oxygen that would have been produced had that tree still been alive. Therefore, if I were to choose, I would vote for the circulation of scientific work through electronic media, which does not require paper or the destruction of valuable natural resources.

Even though electronic media are cheap, easily accessible, and eco-friendly for the circulation of scientific work, there is always a danger to the quality and credibility of such information when it circulates freely on the web. Many colleagues who have been asked to mark first-year undergraduate assignments know exactly what I am talking about: a major problem with websites is that the information they publish often has little (if any) credibility; it is published anonymously and, in many cases, is the product of amateur work. Utilisation of such information can often be dangerous: it is likely to initiate a discussion based on a 'fact' that is not necessarily factual, and lead to a conclusion that is equally false; therefore, many of us have advised university students in the past to avoid the web and to prefer working at libraries, where information has been filtered through peer reviewing strategies.

Despite having always favoured libraries over the web, I now see more clearly than before the need and necessity of electronic resources. There are undoubtedly many advantages, even though several problems need to be tackled along the way. First of all, the web can be easily updated with the latest information on a specific topic. Secondly, the web is easily accessible to the broader public and offers opportunities for people with specific interests but no prior expertise. Thirdly, information can be found through customised search engines, which save time, trouble, and money. Fourthly, dialogue is more direct and can be promoted in real time as the research unfolds. Unfortunately, a major drawback of the web is that information can often be manipulated, become misleading and untrustworthy. When using the web, one must exercise great caution and train oneself to distinguish credible information, which is usually published on websites that meet certain quality standards. This brings me to the necessity of peer reviewing.

For those unaware of the term, peer review is the assessment of scientific work to be published in an academic journal or book (usually in print) by established researchers, called referees. Electronic publications require a similar peer-review process for publication online. In any case, an external, experienced researcher evaluates the quality of a person's work and ensures that scientific and methodological integrity is maintained throughout the text. This process is important because it separates scientific arguments from undocumented information that is often found online. In my opinion, however, this important process can become highly problematic, particularly when online peer review follows the standards of conventional peer review for academic journals. In this article, I plan to highlight a few problems with conventional peer review, primarily based on my personal experience in Classical Archaeology. Secondly, I plan to discuss possible solutions that could be applied online to make peer review of electronic articles easier, without compromising the quality required for the publication of scientific work.

To begin, I would like to state the obvious for any type of human interaction, including peer reviewing: the author of an article and the referees are all human beings! This means that despite their high level of education, their experience and their training in dealing with complex situations, they are still humans; they are imperfect; they are still in the process of learning; and finally, their personalities consist of a mixture of intellectual, spiritual and emotional elements, which do not necessarily communicate with each other all the time. Mistakes are not only possible, but most likely expected and acceptable. If all participants are aware of their human nature from the beginning, any friction or clashes of personality in such interactions can be easily resolved. I strongly believe that all those working on a publication do not only work on the actual text but also on various aspects of their personalities. And in my opinion, any authors or referees who think they possess the divine charisma of perfection and have the right to judge the work of others against their supposed perfection definitely need to find a dictionary and look up the word “humility”, as they have probably never heard of it before.

So, why is Humility so important? At the time of writing this article, I came across a book that is unusual for an archaeologist living and working in a totally secular environment. The book was written by Archimandrite Sophrony, founder of the Patriarchal Stavropegic Monastery of St. John the Baptist at Maldon, Essex. The book is titled 'St. Silouan the Athonite' and presents some episodes from the life and teachings of Staretz Silouan, an Orthodox monk who practised asceticism in the Russian monastery of St. Panteleimon on Mt. Athos. To me, the book was extremely difficult to follow, and I definitely need to go back to it at some point; however, one of the things I remember from it is the wonderful sense of peace and fulfilment I found in its words. Even though my mind was incapable of grasping the deeper meaning of what I was reading at that time, there was one thing that stroke me and I am still thinking about it ever since: Staretz Silouan attributed war, violence, clash, rage and a lot of other bad practices to the false perception of human beings that they confidently know what is right and true; and this is the direct result of lack of Humility. Putting this observation into perspective, we need to ask ourselves two questions: Is there someone out there who confidently knows the difference between right and wrong? Is there someone out there who confidently believes that he or she possesses the one and only knowledge while others do not?

At this point, one could argue that peer review is not about proving who is knowledgeable and who is still working on it, but rather about maintaining the quality required for the circulation of scientific knowledge. Staretz Silouan would probably have challenged everybody by asking whether there is such a thing as scientific knowledge, but I am not planning to go that way. I will, however, challenge the mechanism for ensuring such quality and the extent of its intervention. If put down in simple terms, peer reviewers or referees act as guardians, so, quis custodiet ipsos custodes? In other words, how do we know that the criteria applied by the referees to a person's work are the correct and most reliable ones? My immediate answer to this question is that we do not know. Referees are human beings, and they will assess an author's work based on their own belief systems and perceptions of truth, and they will probably abuse their position of authority to make their judgment clear. This is not necessarily a bad idea if the person writing the article does not know what he or she is talking about; however, it becomes annoying if the author actually does. To explain my argument, I will produce a ridiculous story, which will point out the problem in the simplest and most effective way:

“A researcher who specialises in mustard decides to produce a scientific article, which discusses the gastronomical effects of his favourite sauce on potatoes. His article was happily accepted by the Journal of Gastronomical Pleasures, and the journal appointed two referees to review it and suggest changes. The first referee, who has conducted research on the effects of ketchup on barbecued meat, points out that the author has omitted to compare mustard to ketchup. Furthermore, the author has omitted discussing the fact that potatoes normally pair with meat; therefore, the gastronomic effects of mustard must be discussed in relation to meat as well. The second referee, who has conducted research on the effects of mayonnaise on fish, suggests that some mention should be made of the significance of tartar sauce in potato consumption. At some point during the review, the editor makes clear to the author that, although he does not need to follow the referees' suggestions, failing to do so will only delay publication of his article for another couple of years. The only option the author has is to discuss the combined effects of mustard and ketchup on meat and potatoes; point out the disadvantages of the two sauces when used on fish as opposed to mayonnaise; and leave reasonable space to suggest that the solution to the universal gastronomical problem lies in tartar sauce. In other words, if he wishes to publish his article, he has to make sure he pleases everybody, regardless of whether his original idea was simply about mustard and potatoes. By the time the article is ready and all sides are pleased, the publisher emails the author to inform them that the article has exceeded the publication's word limit and needs to be cut. By the time the article is to be published, nothing in it reflects the original ideas of the author”.

I need to admit that this story is definitely not the most representative of what happens during peer review, even though I have already witnessed something similar; however, it does show how different dynamics work and how the final result may have nothing to do with the author's original idea. In my opinion, if there is one thing that needs to be respected from the beginning in online articles, it is the author's original point. A huge advantage of online work is that comments and modifications can be added at any time after an article is published, and, if required, without forcing the author to change their work before it is released. This situation does not generate the same amount of pressure noted in conventional publications, where once the text is out, there is little that can be done to amend it; and therefore, I think it has become a practice for peer reviewers to amend everything beforehand, often including the author's original point.

If this is the case, then one may wonder why authors cope with such significant changes to their work. The answer is that very often, authors do not really care about the amendment of their original views. They seem quite happy to have the press get everything and anything related to their work, which adds points to their publication record in case they need to pursue an academic career. The original contribution of their work is normally sacrificed so that they can gain the most of what is offered in a pre-established academic system. This attitude has a significant impact on the work of other researchers as well, who must do the same to remain competitive. In general, what is nowadays regarded as an ideal publication is brief and focused on a single point, which does not really challenge anything but just adds a few more sentences to an overall discussion and a few more credits to the author's publication record. In my own field, Classical Archaeology, I often read articles where the point the author is making is so simplistic and obvious that I wonder whether they offer anything new. I have a feeling it's become common practice among researchers to write brief articles stating the obvious, just because no other researcher has said it before. In the real world, however, the obvious is so obvious that it does not require any academic analysis to be addressed; it is simply obvious!

The structure rules and strict word limits imposed by publishers force the publication of brief articles, often lacking the quality and originality a paper has to offer. So, how can someone publish something 'big', which requires plenty of ink and paper? An author can either go for a book or a monograph, or even better, publish his or her own work online. If an author wishes to write a book or a monograph on a topic that is 'big' and requires thorough analysis, the book may not necessarily reflect the publisher's understanding of the 'big picture', which is normally translated to 'big sales'. In this case, the author will need to present their work in a way that satisfies the broader interests of the widest possible audience; failing to do so will likely lead to a series of rejections from publishers. This pursuit of big sales pushes humility aside, as the author is expected to present a scientific argument as the central focus of the entire universe, even though, in reality, nobody outside a small, specific group of academics will care. For example, if I were to publish my own doctoral thesis in the form of a monograph under the title 'The Correlation of Technological and Stylistic Changes, and Society, in the Production of Attic Geometric and Orientalising Finewares', it would have sold a limited number of copies to people and institutional libraries specialised in Classics. Furthermore, each copy would have cost the equivalent of a month's minimum wage, probably because this would have been the only way for the publisher to make a profit out of limited sales. However, if I had modified the title of my monograph to 'Style, Technology and Social Change: A Theoretical Manual in Pottery Analysis for Students and Specialists', the situation would have been different, as it would have targeted a larger audience. This is not necessarily a bad practice, but such a monograph would have displeased the audience when they realised that its content had nothing to do with its title. In my opinion, going with an online publication saves a lot of paper and trouble, while the author's thoughts and views can retain their originality and be introduced to the widest possible audience for free. In archaeology, in particular, there are fantastic examples of work published online that are accessed and read every day by large numbers of professionals and students. Two of my favourites are the Atlas of Roman Pottery in Britain by Paul Tyers and the National Roman Fabric Reference Collection by Roberta Tomber and John Dore.

Another problem with publishers and conventional peer review is that the review does not necessarily target the article; it often targets the author. In other words, it does not matter what is written in the paper, but who the author is. I am not entirely sure whether this strategy is followed to guarantee scientific integrity or to drive big sales; however, denying a person the right to present his or her views because of who this person is is, in my opinion, completely unacceptable. Several of my colleagues have tried to publish scientific work in a couple of well-established, respectable journals. While reading endless pages of rules and requirements for submitting articles to them, they realised they needed to present them with a ridiculously large number of referees, who also had to be active academics associated with institutions different to the ones they were studying in. This made me think that a new researcher who has not yet built a circle of useful acquaintances may be condemned to anonymity and rejection. In any super politically correct utopia that academics and other intellectuals dream of living in, such an act would have been seen as racism and discrimination. For the sake of preserving the integrity of scientific truth, however (presuming that there is one), peer review can become a form of acceptable discrimination, which still allows referees to claim, in the most socialist of manners, that they remain humanists and respect the diversity of opinions.

I believe the phrase 'preserving the integrity of scientific truth, presuming that there is one' strikes a sensitive chord with many people reading this page. It probably feels like throwing salt on an open wound; therefore, I need to explain what I mean. First of all, as someone who has used statistics in his own research, I have come to believe that there is no such thing as scientific truth, particularly in archaeology and the humanities more generally. What we normally have are trends presented using different scientific methods, one of which is statistics, which are then used to support an argument and offer possible explanations. What is produced at the end is not a full explanation but a proposed one, which contributes only to a broader discussion. Nothing guarantees that this explanation will not be challenged in the future under the light of new evidence. Furthermore, the use of statistical methodologies, which nowadays dominate archaeology, is, by definition, problematic: although they can point to a trend, they can also be manipulated and often wrong. People who vote in the elections have probably seen how often statistics fail to predict the winner. The reason is very simple: statistics are based on calculations that do not account for the unpredictable nature of human beings and the complexity of their behaviours. For example, on a table for two where a plate holds the leftovers from roast chicken, statistics will show that the two participants ate half a chicken each. Statistics will deliberately fail to notice plates with other leftovers on the table and will not consider that one of the two participants in that meal was vegetarian. In a similar way, any scientific approach that utilises such quantification methods can easily lead to the wrong conclusion. So, how can we know what is true or false?

When it comes down to peer reviewing, we definitely need to consider which 'truth' is being reviewed. Based on the work I have read so far in archaeology, I have concluded that no methodology is 100% correct. Every methodology can be easily challenged, especially if it is a new and untested tool; so, is it a good idea for referees to attack new methodologies? And to what extent is the referee's critique constructive to the author? Based on some reviews I have read along my own line of research, the appreciation of a certain methodology depends solely on the character and kind intentions of a referee. In other words, there are referees who realise there is no such thing as perfect methodology, and others who measure their own supposed perfection against the methodology of another individual. In my opinion, such peer reviewing is unrealistic and non-constructive. Particularly when dealing with theoretical questions, which are commonly encountered in the humanities, it may be worth skipping the review entirely and allowing the ideas to circulate freely, so the broader research community can decide whether to assimilate them into their own work or reject them. Online publications enable this and offer the advantage of constructive dialogue, which might lead an author to revise their methodology after discussing it openly with colleagues.

Having explained some reasons why peer review is non-constructive in theoretical disciplines, I will now move to peer review in practical disciplines. The main difference between the two is that an essay on a practical subject must be based on factual evidence, axioms, or tested data analysis models; therefore, the reader needs to see a clear distinction between true and false or right and wrong. In this case, referees check if the rationale followed by an author is indeed rational, or in other words, if it makes any practical sense. For example, if an author claims that 4+5=72, the referees are expected to return the paper to the author and also slide a calculator in the envelope. If this calculation is essential for the author to claim that a bridge made of toothpicks can hold the weight of twenty lorries, then the referees have every right to refuse the publication of such work. If they do not, whoever decides to build such a bridge in the future will become responsible for the deaths of many people, and this must be stopped. Regardless of how simplistic this example is, it shows that peer review of scientific research in practical disciplines should be strict, as it often involves matters of public safety.

Archaeology is one of these sciences that lies between extreme philosophy and abstract theory practised in lecture theatres and manual work, digging, hands-on artefact handling and recording, which is practised on site. When it comes to writing a piece of work related to the practical side of archaeology, peer reviewing can offer very little. For example, if the excavators of an archaeological site claim they recovered 50 kg of burnt flint from a specific pit, what is there for the referees to review? Fifty kilograms of burnt flint are just fifty kilograms of burnt flint; a plain fact. Even if the referees wanted to prove that the quantity was wrong or that the stones were different, there would have been no way for them to prove the opposite. So, is peer review necessary for publications that present quantified data? Probably not.

At this point, some of my colleagues may argue that peer review of archaeological work aims to check whether the rationale followed by an archaeologist is indeed rational, in the same way it can be applied to an engineer or a spine surgeon. Without meaning to underplay the importance of archaeological research, however, I must say that I completely disagree with this opinion for two reasons. Firstly, in archaeology, the subject of our study is already dead; our research will not cause any loss of human life that has not already occurred. Furthermore, it is a great shame that many archaeologists are obsessed with the dead, in a way that they have nothing interesting to offer to the living; and this is perhaps the reason why our profession is held in low regard, and our research funding is constantly facing cuts. Secondly, we really need to sit down and discuss what 'scientific' and 'factual' archaeology is. Personally, I feel deeply worried every time I read some new theoretical approach that turns out to be a highly imaginative study, which discusses abstract concepts such as symbolism, agency, hierarchy, nationalism, personhood, entanglement, gender, feminism, diversity and transvestitism in periods that fall way into the prehistory, where practical evidence is virtually absent. And having said this, I am even more worried that some of these studies have been peer-reviewed as sound pieces of academic work.

After expressing my views on the topic, I will now finish this article by presenting three practical ways in which online peer reviewing can be more effective and useful than conventional peer reviewing. First of all, the main problem in conventional peer reviewing is that articles take too long to be published. By the time a publication is published, the information presented in an article can be completely out of date. If the work is produced online, however, this is not a problem as information is available in real time. Secondly, reviewing needs to take place after the article has been published online and not before. The researchers who read the article can act as referees and offer an online discussion of the author's ideas. Their comments and suggestions will be read by the public in real time, and new ideas will be incorporated into the author's work in the article's sequence. Even if the author decides to take the article down because it is clearly inadequate, this can happen without much hassle. And if the author wishes to defend their views against public critique, this is possible without rejecting the article before it is even published. Thirdly, there is no need for the reviewers to be accredited academics or known researchers to review an article. For example, the work published on the Archaeological Data Service (ADS) website has been reviewed by accredited professionals in the field. Often, the nature of such articles, which aim to present raw data, does not require any sophisticated peer review beyond conventional proofreading, ensuring that the text and images are coherent and present the data in the best possible way.

To conclude, online publications need to be practical and to convey the message of academic research simply and quickly, even to members of the public. Conventional peer-reviewing strategies not only make things difficult for authors but also structure a system that limits information to a few knowledgeable and highly specialised individuals. This aspect definitely undermines the role of scientific research and projects an image of a community restricted to a small group. Furthermore, it often sets unrealistic standards for well-established professionals who have important ideas and knowledge to offer, but fail to comply with the requirements of academically established peer reviewing. Technological evolution and the widespread use of website resources have not only enabled the presentation of data in real time but also the publication of strictly academic ideas to a wider public. To me, conventional peer review belongs to an era when class and status defined the integrity of research. Entering the twenty-first century, I strongly feel that we need to abandon practices that restrain the circulation of information and accept the terms that technological evolution has brought into the game. If there are researchers who still worry about the quality of online work, we first need to reconsider the quality of conventionally published work and think twice about its peer-reviewing strategies: have they been designed to guarantee quality, or to promote exclusion?