The necessity of peer reviewing

One of the questions I often ask myself, which has produced a lot of discussion among fellow researchers and people working in academia, relates to the necessity of peer reviewed publications. Do we actually need them? I still have the feeling that just by thinking of such question, I automatically become the target of those who strongly suggest “yes, of course we do” without any second thought. I have already been shelled with heavy critique just for allowing this sacrilegious though to penetrate my mind; therefore, the purpose of this article in not to prove the opposite. Furthermore, I do not wish to engage in an argument with people who support that academic knowledge and scientific information need to be reviewed, critiqued, filtered and approved by the board of wise tribesmen. Given that the board would probably object if I was to present my own views in any conventional periodical, however, I decided to do it online by taking advantage of the freedom provided by the web. For what is to be said further on, I need to apologise in advance.

Before I begin discussing my thoughts on the subject, I need to say that so far mankind has managed quite well in chopping down as many trees as possible to produce paper for people to express their thoughts openly to the public. One may argue that human views and knowledge worth more than the lives of trees; but, having read a lot of abstract academic thought and theory in my life, particularly in archaeology, I think the lives of trees are more valuable than we already believe and trees should not be wasted just for producing paper. There have been many occasions that I reminded fellow researchers and academics that every line of thought we produce, equals to the life of another tree. I guess I am a bit over-sensitive on this matter and remind people to act with consideration to the environment. To be honest, the unnecessarily complicated language, the blurry arguments and the abstract thoughts I have encountered in many books and articles in archaeology, cannot match the value of the oxygen that would have been produced had that tree been still alive. Therefore, if I was to choose, I would vote for the circulation of scientific work through electronic media, which do not require wasting paper and destroying valuable natural resources.

Even though electronic media are cheap, easily accessible and eco-friendly for the circulation of scientific work, there is always danger in relation to the quality and credibility of such information when circulating freely though the web. Many colleagues that have been asked to mark first-year undergraduate assignments know exactly what I am talking about: a major problem with websites is that information often has little (if any) credibility; it is published anonymously, and in many cases is the product of amateur work. Utilisation of such information can often be dangerous: it is likely to initiate a discussion based on a 'fact' that is not necessarily factual, and lead to a conclusion that is equally false; therefore, many of us have advised university students in the past to avoid the web and to prefer working at libraries, where information has been filtered through peer reviewing strategies.

Despite the fact that I have always favouritised libraries over the web, I am now seeing more clearly than I did before the need and necessity of electronic resources. There are undoubtedly many advantages, even though several problems need to be tackled along the way. First of all, the web can be easily updated with the latest information on a specific topic. Secondly, the web can be easily accessed by the broader public and offer the opportunity of researching to people with specific interests but no previous expertise. Thirdly, information can be detected through customised search engines, which save a lot of trouble, time and money. Fourthly, dialogue is more direct and can be promoted in real time as the research unfolds. Unfortunately, a major drawback of the web is that information can often be manipulated, become misleading and untrustworthy. When using the web, one must exercise great caution and train oneself to detect creditable information, which is usually published on websites that follow some quality standards. This brings me to the necessity of peer reviewing.

For those that are unaware of the term, peer reviewing relates to the assessment of scientific work, which is to be published in an academic journal or book (usually in paper), and is conducted by established researchers called referees. Electronic publications require a similar process of peer reviewing, which is applied on pieces of work that are to be published online. In any case, an external and experienced researcher evaluates the quality of a person's work and makes sure that a level of scientific and methodological integrity is maintained throughout the text. This process is important as it separates scientific arguments from undocumented pieces of information that are normally found online. In my opinion, however, this important process can become highly problematic, particularly when online peer reviewing follows the standards noted in conventional peer reviewing for academic journals. In this article, I first plan to highlight a few of the problems noted in conventional peer reviewing, based primarily on my personal experience from Classical Archaeology. Secondly, I plan to discuss some possible solutions, which could be applied online in order to make the peer reviewing of electronic articles easier, without endangering the level of quality required for the publication of scientific work.

To begin, I would like to state the obvious for any type of human interaction, including peer reviewing: the author of an article and the referees are all human beings! This means that despite their high level of education, their experience and their training in dealing with complex situations, they are still humans; they are imperfect; they are still in the process of learning; and finally, their personalities consist of a mixture of intellectual, spiritual and emotional elements, which do not necessarily communicate with each other all the time. Mistakes are not only possible, but most likely expectable and acceptable. If all the participants are aware of their human nature from the beginning, then any friction or clash of personalities noted in such interactions can be easily resolved. I strongly believe that all those working over a publication do not only work on the actual text but also on different elements their personality. And in my opinion, any authors or referees who think they possess the divine charisma of perfection and they have the right to judge the work of others against their supposed perfection, they definitely need to find a dictionary and look up the word Humility as they have probably never heard of it before.

So, why is Humility so important? At the time of writing of this article, I came across a book that is unusual to be read by an archaeologist living and working in a totally secular environment. The book was written by Archimandrite Sophrony, who was the founder of the Patriarchal Stavropegic Monastery of St. John the Baptist at Maldon, Essex. The book is titled 'St. Siloun the Athonite' and it presents some episodes from the life and the teachings of Staretz Silouan, an Orthodox monk who practised asceticism in the Russian monastery of St. Panteleimon on Mt. Athos. To me, the book was extremely difficult to follow and I definitely need to go back to it at some point in time; however, one of the things I remember while reading it is this wonderful sense of peace and fulfilment I found in its words. Even though my mind was incapable of grasping the deeper meaning of what I was reading at that time, there was one thing that stroke me and I am still thinking about it ever since: Staretz Silouan attributed war, violence, clash, rage and a lot of other bad practices to the false perception of human beings that they confidently know what is right and true; and this is the direct result of lack of Humility. Putting this observation into perspective, we need to ask ourselves two questions: Is there someone out there who confidently knows the difference between right and wrong? Is there someone out there who confidently believes that he or she possesses the one and only knowledge while others do not?

At this point one could argue that peer reviewing is not about proving who is knowledgeable and who is still working on it, but the whole procedure is about maintaining a level of quality that is required for the circulation of scientific knowledge. Staretz Silouan would have probably challenged everybody by asking if there is such thing as scientific knowledge, but I am not planning to go that way. I will, however, challenge the mechanism of assuring such quality and the degree of its intervention. If put down in simple terms, peer reviewers or referees act as guardians, so, quis custodiet ipsos custodes? In other words: how do we know that the criteria applied by the referees on a person's work are the correct and most reliable ones? My immediate answer to this question is that we do not know. Referees are human beings and they will assess an author's work based on their own belief systems and perceptions of truth, and they will probably abuse their position of authority to make their judgement clear. This is not necessarily a bad idea if the person writing the article does not know what he or she is talking about; however, it becomes annoying if the author actually does. To explain my argument, I will produce a ridiculous story, which will point the problem out in the simplest and most effective way:

“A researcher who specialises on mustard decides to produce a scientific article, which discusses the gastronomical effects of his favourite sauce on potatoes. His article is happily accepted by the Journal of Gastronomical Pleasures and they appoint two referees to go through the article and suggest changes. The first referee, who has conducted research on the effects of ketchup on barbecued meat, points out that the author has omitted to compare mustard to ketchup. Furthermore, the author has omitted discussing the fact that potatoes normally assort meat, and therefore, the gastronomical effects of mustard must be discussed in relation to meat too. The second referee, who has conducted research on the effects of mayonnaise on fish, suggests that some mention needs to be made in relation to the significance of tartar sauce in potato consumption. At some point along the review, the editor makes clear to the author that although he does not need to follow the referees' suggestions, failing to do so will only postpone the publication of his article for another couple of years. The only option the author has is to discuss the combined effects of mustard and ketchup on meat and potatoes; point the disadvantages of the two sauces when used on fish as opposed to mayonnaise; and, leave reasonable space to suggest that the solution of the universal gastronomical problem lies on tartar sauce. In other words, if he wishes to publish his article he has to make sure he pleases everybody, regardless if his original idea was simply about mustard and potatoes. By the time the article is ready and all sides are pleased, the publisher emails the author back to point out that the article has exceeded the word limit required for the publication and it needs to be cut down. By the time the article is to be published, nothing in it reflects the original ideas of the author”.

I need to admit that this story is definitely not the most representative of what is happening during peer reviewing, even though I have already witnessed something similar; however, it does show how different dynamics work and how the final result may have nothing to do with the author's original idea. In my opinion, if there is one thing that needs to be respected from the beginning in online articles, this is the author's original point. A huge advantage of online work is that comments and modifications can be added any time after the publication of an article, and if this is required, without forcing the author to change his/her work before it is released. This situation does not generate the same amount of pressure noted in conventional publications, where once the text is out, there is little that can be done to amend it; and therefore, I think it has become a practice for peer reviewers to amend everything beforehand, often including the author's original point.

If this is the case, then one may wonder why authors cope with such significant changes of their work. The answer is that very often, authors do not really care about the amendment of their original views. They seem to be quite happy getting out of the press everything and anything possible related to their work, which adds points on their publication record in case they need to pursue an academic career. The original contribution of their work is normally sacrificed so that they can gain the most of what is offered in a pre-established academic system. This attitude has a significant impact on the work of other researchers too, who have to do the same if they wish to be competitive. In general, what is nowadays regarded as ideal publication is brief and related to an isolated point, which does not really challenge anything but just adds a few more sentences to an overall discussion, and a few more credits to the author's publication record. In my own field, which is Classical Archaeology, I often read articles where the point an author is so simplistic and obvious, that I wonder if the article has anything new to offer. I have a feeling that it has become a common practice among researchers to write brief articles stating the obvious, just because the obvious has not been stated by another researcher earlier. In the real world, however, the obvious is so obvious that it does not require any academic analysis to be addressed; it is simply obvious!

The structure rules and strict word limits demanded by the publishers force the circulation of brief articles often lacking the quality and originality a paper has to offer. So, how can someone publish something 'big', which requires plenty of ink and paper? An author can either go for a book or a monograph, or even better, publish his or her own work online. If an author wishes to write a book or a monograph on a topic that is 'big' and requires thorough analysis, the book may not necessarily reflect the publisher's understanding of the 'big picture', which is normally translated to 'big sales'. In this case, the author will need to present his or her work in a way that satisfies the broader interests of the widest audience possible, while failing to do so, will probably lead to series of rejections from the publishers. This pursue of big sales pushes humility aside, as the author is expected to present a scientific argument as being the central focus of the entire universe, even though in reality nobody will care outside a small and specific group of few academics. For example, if I was to publish my own doctorate thesis in the form of a monograph under the title 'The Correlation of Technological and Stylistic Changes, and Society, in the Production of Attic Geometric and Orientalising Finewares', it would have sold a limited number of copies to people and institutional libraries specialised in Classics. Furthermore, each copy would have cost the equivalent of a month's minimum wage, probably because this would have been the only way for the publisher to make a profit out of limited sales. However, if I was to modify the title of my monograph to 'Style, Technology and Social Change: A Theoretical Manual in Pottery Analysis for Students and Specialists', the situation would have been different as it would have targeted a larger audience. This is not necessarily a bad practice, but such a monograph would have displeased the audience when they were to realise that its content had nothing to do with its title. In my opinion, going for an online publication saves a lot of paper and trouble, while the author's thoughts and views can preserve their originality and become introduced to the widest audience possible for free. In archaeology, in particular, there are fantastic examples of work published online, which are accessed and read every day by a large number of professionals and students. Two of my favourites are the Atlas of Roman Pottery in Britain by Paul Tyers and the National Roman Fabric Reference Collection by Roberta Tomber and John Dore.

Another problem with publishers and conventional peer reviewing is that the review does not necessarily target the article but it often targets the author writing it. In other words, it does not matter what is written in the paper, but who the author is. I am not entirely sure if this strategy is followed in order to guarantee scientific integrity or big sales; however, denying a person to present his or her views because of who this person is, is in my opinion completely unacceptable. Several colleagues of mine have tried publishing pieces of scientific work in a couple of well-established and respectable journals. While reading endless pages of rules and requirements for submitting articles to them, they realised they needed to present them with a ridiculously large number of referees, who also had to be active academics associated with institutions different to the ones they were studying in. This made me think that a new researcher, who has not managed to generate a circle of useful acquaintances yet, may be condemned to anonymity and rejection. In any super politically correct utopia that academics and other intellectuals dream on living in, such act would have been seen as racism and discrimination. For the sake of preserving the integrity of scientific truth, however, (presuming that there is one) peer reviewing can become a form of acceptable discrimination, which still allows referees to claim in the most socialist of all manners that they remain humanists and respect the diversity of opinions.

I believe the phrase 'preserving the integrity of scientific truth, presuming that there is one' strikes a sensitive chord to many people reading this page. It probably feels like throwing salt on an open wound; therefore, I need to explain what I mean. First of all, being a person that has used statistics in his own research, I have developed the perception that there is no such thing as scientific truth, particularly in archaeology and the humanities in general. What we normally have are trends presented with different scientific methods, one of which is statistics, which are then used to discuss an argument and some possible explanations. What is produced at the end is not a full explanation but a proposed explanation, which only contributes to a broader discussion. Nothing guarantees that this explanation will not be challenged in the future under the light of new evidence. Furthermore, the use of methodologies based on statistics, which nowadays dominate archaeology, is by definition problematic: although they can point out a trend, they can also me manipulated and they can often be wrong. People who vote in the elections have probably seen how often statistics fail to predict the winner. The reason is very simple: statistics are based on calculations that do not take into account the unpredictable nature of human beings and the complexity of their behaviours. For example, on a table for two where a plate carries the leftovers of roast chicken, statistics will point out that two participants ate half chicken each. Statistics will purposely fail to notice plates with other leftovers on the table, and will not consider the fact that one of the two participants of that meal was vegetarian. In a similar way, any scientific approach that utilises such quantification methods can easily lead to the wrong conclusion. So, how can we know what is true or false?

When it comes down to peer reviewing, we definitely need to consider which 'truth' is being reviewed. Based on the work I have read so far in archaeology, I have concluded that no methodology is 100% correct. Every methodology can be easily challenged, especially if this is a new and untested tool; so, is it a good idea for referees to attack new methodologies? And to what extent is the referee's critique constructive to the author? Based on some reviews I have read along my own line of research, the appreciation of a certain methodology depends solely on the character and kind intentions of a referee. In other words, there are referees that realise there is no such thing as perfect methodology, and others who measure their own supposed perfection against the methodology of another individual. In my opinion, such peer reviewing is unrealistic and non-constructive. Particularly when dealing with theoretical questions, which are normally encountered in the humanities, it may worth skipping the review completely and allowing the ideas to circulate freely, and let the broader community of researchers decide if they wish to assimilate them in their own work or reject them. Online publications allow this to happen, and also offer the advantage of constructive dialogue, which might lead to the revision of an author's methodology after discussing it openly online with other colleagues.

Having explained some of the reasons why peer reviewing is non-constructive when it comes down to theoretical disciplines, I will now move to the peer reviewing of work in practical disciplines. The main difference between the two is that an essay on a practical subject needs to be based on factual evidence, axioms or data analysis models that have already been tested in the past; therefore, the reader needs to see a thick line between true and false or right and wrong. In this case, referees check if the rationale followed by an author is indeed rational, or in other words, if it makes any practical sense. For example, if an author claims that 4+5=72, the referees are expected to return the paper back to the author and also slide a calculator in the envelope. If this calculation is essential for the author to claim that a bridge made of toothpicks can hold the weight of twenty lorries, then the referees have every right to refuse the publication of such work. If they do not, whoever will decide to build such a bridge in the future will become responsible for the death of many people, and this must be stopped. Regardless how simplistic this example is, it shows that the peer reviewing of scientific research on practical disciplines should be strict as it often involves matters of public safety.

Archaeology is one of these sciences that lies between extreme philosophy and abstract theory practised in lecture theatres and manual work, digging, hands-on artefact handling and recording, which is practised on site. When it comes to writing a piece of work related to the practical side of archaeology, peer reviewing can offer very little. For example, if the excavators of an archaeological site claim that they recovered fifty kilograms of burnt flint from a specific pit, what is there for the referees to review? Fifty kilograms of burnt flint are just fifty kilograms of burnt flint; a plain fact. Even if the referees wanted to prove that the quantity is not right, or that the stones are different, there would have been no way for them to prove the opposite. So, is there a need for peer reviewing when is come to publications that are meant to present quantified data? Probably not.

At this point, some colleagues of mine may argue that the peer reviewing of archaeological work aims to check if the rationale followed by an archaeologist is indeed rational in the same this can be applied to an engineer or a spine surgeon. Without meaning to underplay the importance of archaeological research, however, I must say that I completely disagree with this opinion for two reasons. Firstly, in archaeology the subject of our study is already dead; our research will not cause any loss of human lives that have not been lost already. Furthermore, it is great shame that many archaeologists are obsessed with the dead, in a way that they have nothing interesting to offer to the living; and this is perhaps the reason why our profession is held in low regard and our research funding is constantly facing cuts. Secondly, we really need to sit down and discuss how 'scientific' and 'factual' archaeology is. Personally, I feel deeply worried every time I read some new theoretical approach that turns out to be a highly imaginative study, which discusses abstract concepts such as symbolism, agency, hierarchy, nationalism, personhood, entanglement, gender, feminism, diversity and transvestitism in periods that fall way into the prehistory, where practical evidence is virtually absent. And having said this, I am even more worried that some of these studies have been peer reviewed as sound pieces of academic work.

After having expressed my views on the topic, I will now finish this article by presenting three practical ways, in which online peer reviewing can become more effective and useful as opposed to conventional peer reviewing. First of all, the main problem in conventional peer reviewing is that articles take too long to be published. By the time a publication is out, the information that is presented in an article can be completely out of date. If the work is produced online, however, this is not a problem as information is available in real time. Secondly, reviewing needs to take place after the article has been published online and not before. The researchers reading the article can act as referees and offer an online discussion on the ideas of the author. Their comments and suggestions will be read by the public in real time and new ideas will be accumulated in the author's work, following the sequence of the article. Even if the author decides to take the article down because it is clearly inadequate, this can happen without much hassle. And if the author wishes to defend his/her views against public critique, this is possible without the rejection of the article before it is even published. Thirdly, there is no need for the reviewers to be accredited academics or known researchers to review an article. For example, the work published on the website of the Archaeological Data Service (ADS) has been reviewed by accredited professionals in the field. Often, the nature of such articles, which aim in the presentation of raw data, does not require any sophisticated peer reviewing other than a conventional proof reading, which guarantees that the text and images are coherent and present the data in the best possible way.

To conclude, online publications need to be practical and to convey the message of academic research simply and quickly, even to members of the public. Conventional peer reviewing strategies not only make things difficult for an author, but also structure a system that limits information to few knowledgeable and highly specialised individuals. This aspect definitely undermines the role of scientific research and projects the image that is restricted to a small community. Furthermore, it often sets unrealistic standards to well-established professionals who have important ideas and knowledge to offer, but fail to comply to the requirements of academically established peer reviewing. Technological evolution and the wide spread of website resources have not only allowed the presentation of data in real time, but they have also allowed the publication of strictly academic ideas to a wider public. To me conventional, peer reviewing belongs to an age where class and status defined the integrity of research. Entering the twenty-first century, I strongly feel that we need to abandon practices that restrain the circulation of information and accept the terms that technological evolution has brought into the game. If there are researchers out there who still worry about the quality of online work, we first need to reconsider the quality of conventionally published work and think twice of its peer reviewing strategies: have they been there to guarantee quality, or have they been there to promote exclusion?