Editorial

DEVELOPING PEER REVIEW SYSTEM FOR A SMALL JOURNAL – THE KJP EXPERIENCE

Shahul Ameen1*, Samir Kumar Praharaj2
1Consultant Psychiatrist, Puthujeevan Trust Hospital for Psychological Medicine, Kottamury; St. Thomas Hospital, Chethipuzha.
2Associate Professor, Department of Psychiatry, Kasturba Medical College, Manipal.
*Correspondence: Mise en Scene, Behind Anandashramam, Changanacherry PO, PIN – 686101. Email: shahulameen@yahoo.com

This is the 30th year of publication for Kerala Journal of Psychiatry. The Journal is being regularly published since 1985, and we begin this editorial with a huge applause for the former office bearers and editors of Kerala state branch of Indian Psychiatric Society (IPS) for this achievement. This is a tremendous feat, especially when we consider that the neighboring state of Karnataka, home to many prestigious psychiatric institutions in both the government and private sector, is yet to have a state psychiatric journal. List of KJP’s past editors include some names which are well-known in national and international circles: like Dr KA Kumar, who subsequently went on to become the President of Indian Psychiatric Society, and Dr E Mohandas, who recently won the Excellence in Mentorship Award from World Federation of Societies of Biological Psychiatry; and renowned researchers like Dr S Shaji whose epidemiological studies on dementia have been published in British Journal of Psychiatry.1,2 With a glorious history like this behind it, the time had come for KJP to take another step that will further improve its quality and legitimacy — to implement the peer review system.

Peer review — the use of experts, or peers, to help judge the value of submitted work — is nowadays generally regarded as an essential step before biomedical publication. Science works best in an environment of unrestrained criticism, and peer review, which guarantees provision of such criticism in a structured way, is considered “one of the glories of science”.3 External peer review helps the journal editors, and therefore the authors, and science in general, to select and improve the best manuscripts for publication, and has been reliably proven to improve the quality and readability of manuscripts.4,5,6

Many changes that happened in the past about a decade did make the task of introducing a peer review system for KJP much easier:

Another facet of the aforementioned burgeoning of postgraduate training centers, though indirectly, also created an urgent need that KJP should now implement a system of peer review: Medical Council of India insists that trainees should have published one research paper to be eligible to appear at the postgraduate degree examination, and that faculty members should have a specified number of publications to be eligible for promotion. These clauses are likely to lead to an increase in the number of submissions to KJP, necessitating that the Journal develop a system to scrutinize and sieve through them. Besides, it was also realized that, in a scenario like this, KJP is uniquely placed to serve as a state-level platform where the new generation of psychiatrists can learn and master the techniques of scientific writing, publishing, peer reviewing, etc.

When SA, the current editor, presented the plan to introduce the peer review system in one of the Executive Committee (EC) meetings, Dr Harish M Tharayil, a past editor of KJP and one of the strongest advocates for quality psychiatric research in the state, suggested that we should first provide adequate training to prospective reviewers. Considering the efforts and expenses involved, the eventual decision was to wait and see if the initial reviews are of adequate quality. As the first set of reviews poured in, however, one fact came to the fore: Though most reviewers submitted highly insightful comments indicative of their deep understanding of and critical perspectives on the subject matters involved, many of the reviews were not structured properly and revealed certain other scopes for improvement. Besides, at a more macro level, it was noticed that most reviewers were handling the process with an approach that would have been more suitable for larger,international journals. Marušic′ et al. had noted that, in small journals, unlike in large prestigious journals, the reviewers look for valuable elements even in weak manuscripts and make great efforts of their own to see that potentially interesting articles get published.7 Such an approach was found lacking in many reviews.

At this juncture, the need for a formal training for our reviewers was again seriously thought about. A series of discussions — spanning nearly 200 emails — ensued in the e-group of EC members, on areas like what format the training program should use, who all can attend it, how to screen and select the participants, what all topics should be covered, etc. Eventually, it was decided to conduct a one-day workshop for 25 participants willing to review 1-2 articles an year. It was also decided that Kerala branch of IPS will fully fund the program. Members interested in joining the program, and thereafter the reviewer team, were requested to submit a list of their publications. Intimation about the workshop was sent to all the 16 postgraduate training centers in the state and to the secretaries of nine local guilds of psychiatrists, and was also spread through many online fora. A committee was constituted — with the Editor, an Associate Editor and a past Editor as members — to screen the applications and select the workshop participants-cum-prospective reviewers. A total of 28 applications were received, and the committee decided to approve all of them.

Another issue was the selection of resource persons. After considering many names, the EC eventually assigned that responsibility to the two of us. SKP, with his experience of having published more than a hundred articles in national and international journals, and of being a peer reviewer for reputed journals like Journal of Clinical Psychopharmacology, Psychiatry Research and Journal of Affective Disorders; and SA, with his experience of having assessed nearly 25 submissions to three issues of KJP in his capacity as Associate Editor and Editor, were deemed by the EC to be capable of covering the entire spectrum of information, from sensitization about international standards to reminders about local particularities, needed by our reviewers.

In accordance with the suggestions made by Callahan in his “format of an effective workshop on peer review”, we decided to send the participants some manuscripts beforehand, which they were expected to review in detail and return to us before the workshop.8 A randomized controlled trial, two case reports and a systematic review, already published in some open access journals, were handpicked by SKP through Google searches. We also introduced many new errors to those manuscripts before forwarding them to the workshop participants. All participants were also sent a 3-page document detailing the “Plans and Policies” of KJP.

Three of the 28 applicants did not return their reviews as they realized that they will not be able to attend the workshop due to some previously unforeseen problems. Sixteen of the remaining 25 participants who attended the workshop duly submitted their reviews to us.

The first presentation at the workshop (by SKP) was titled “Peer review: an introduction”, and touched upon history and future of peer review, limitations of and alternatives to the conventional peer review process, evidence base for its effectiveness, advantages in being peer reviewers, etc. The next presentation (by SA) was “KJP: Policies & Procedures”. It explained the differences in the roles of reviewers in small journals like KJP and larger, more reputed journals; and reminded the prospective reviewers to remember, before making any decisions, that the ultimate responsibility of KJP will be towards its readers. The workflow through which manuscripts submitted to KJP currently pass was described, and the reviewers were informed that the time they take to return their reviews and the quality of the comments they provide are being rated and monitored.

The third presentation (by SA) was the most detailed one. It was titled “Responsibilities of reviewers”, and covered what all the reviewers are supposed to do in different stages of the review process. Three factors the reviewers should consider before accepting a review invitation —if they have any conflict of interest related to the submission, if they have sufficient expertise to assess it, and if they have sufficient time to do a review — were explained. Ethical issues related to the peer review process, like confidentiality, biases and scientific misconduct, were described. Steps involved in doing a review, instructions on preparing an effective reviewer report, and many do’s and don’ts for peer reviewers were discussed. This presentation was mostly based on the suggestions provided in two wonderful books on peer review and the reviewer guidelines of many leading journals.9,10

The next presentation (by SKP) was on review of research reports. It covered the recommendations of Consolidated Standards of Reporting Trials (CONSORT) and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) statements.11,12 After this presentation, the participants were divided into five groups and each group was asked to review, in lines of the CONSORT statement, a certain section of the randomized controlled trial which was mailed to all the participants for pre-workshop review. Reviews done on the spot by each group were discussed in detail. Feedback was also provided on the reviews of that manuscript the participants had submitted before the workshop.

The fifth presentation was titled “How to review a review article” (by SA). Though a lion’s share of submissions to KJP currently belong to the category of non systematic reviews, we were unable to find any formal guidelines on reviewing such manuscripts. Hence this presentation rather focused on the common errors detected in the manuscripts submitted to KJP in last about three years, categorized under headings like errors in references, problems in supplementary material, etc. Examples were shared on how reviewers can help the authors improve their manuscripts by suggesting other important references or additional information, by pointing out unnecessary diversions, by reminding the authors to target their writing at the Journal’s primary audience, and by helping them to improve the language of their manuscripts with suggestions for better clarity, specificity and structure. The presentation also included some slides that summarized the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.13 However, this portion was not presented due to time constraints, and rather the slides were given to the participants. SA had also prepared a set of slides compiling the good comments handpicked from the reviews submitted by the participants on the systematic review mailed to them beforehand, categorizing those comments as per different sections of PRISMA. Those slides too were not presented and were distributed to the participants.

The final presentation was on reviewing case reports (by SKP). It covered the recommendations of CARE guidelines and Naranjo ADR Probability Scale.14,15 After the presentation, the five groups of participants were asked to review different portions of the case report which was mailed to them earlier, and their reviews were discussed in detail. The reviews they had sent us beforehand were discussed too.

Towards the end of the workshop, feedback was collected from all participants. It was found to be generally positive. Following a request from the participants, it was decided to do review and online group discussion of some more sample manuscripts in the future months. All participants were subsequently emailed another case report and a research report for review.

An overall improvement has been noted in the quality of subsequent reviews done for KJP by the workshop participants. Their reviews have become more structured, neatly divided to confidential comments to the editor, comments for both the editor and the authors, general comments, specific comments, etc. They have also started assessing the submissions against guidelines like CARE and Naranjo ADR Probability Scale.The authors who received such reviews too opine that the reviews are insightful, have a friendly tone, and have helped them make their manuscripts more scientifically accurate and easier to read and comprehend.

Further training of the reviewer team too is being planned. All reviewers are being provided with the comments of the other reviewers of the same manuscript. Some booster sessions, on more advanced topics, by more expert faculty, are planned to be organized in the near future, not as independent events, but in the sidelines of other state-level academic programs. KJP also intends to add an element of post-publication peer review, where the entire reviewer team, or even those who are not a part of it, can provide their comments on all our published manuscripts within 7-10 days of online publication.

REFERENCES

  1. Shaji S, PromoduK, AbrahamT, Roy KJ, VergheseA. An epidemiological study of dementia in a rural community in Kerala, India. Br J Psychiatry 1996; 168(6):745-9.

  2. Shaji S, Bose S, Verghese A.Prevalence of dementia in an urban population in Kerala, India. Br J Psychiatry 2005; 186:136-40.

  3. Rennie D. Editorial peer review: its development and rationale. In: Godlee F, Jefferson T, editors. Peer Review in Health Sciences. 2nd ed. London: BMJ Publishing Group; 2003. pp. 1-13.

  4. Armstrong JS. Peer review for journals: evidence on quality control, fairness, and innovation. Sci Eng Ethics 1997; 3:63–84.

  5. Kassirer JP, Campion EW. Peer review. Crude and understudied, but indispensable. JAMA 1994; 272:96–7.

  6. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med 1994; 121:11–21.

  7. Marušic′ A, Lukic′ IK, Marušic′ M, McNamee D, Sharp D, Horton R. Peer review in a small and a big medical journal: case study of the Croatian Medical Journal and The Lancet. Croat Med J 1992; 43:286–9.

  8. Callaham M. The evaluation and training of peer reviewers. In: Godlee F, Jefferson T, editors. Peer Review in Health Sciences. 2nd ed. London: BMJ Publishing Group; 2003. pp. 164-82.

  9. Godlee F, Jefferson T, editors. Peer Review in Health Sciences. 2nd ed. London: BMJ Publishing Group; 2003.

  10. Hames I. Peer review and manuscript management in scientific journals. Malden MA: Blackwell Publishing; 2007.

  11. Schulz KF, Altman DG, Moher D, for the CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c332.

  12. von Elm E, Altman DG, Egger M, Pocock SJ, Gřtzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol 2008; 61(4):344-9.

  13. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 2009; 6(6): e1000097. doi:10.1371/journal.pmed1000097

  14. Gagnier JJ, Kienle G, Altman DG, Moher D, Sox H, Riley D, et al.The CARE guidelines: consensus-based clinical case reporting guideline development. BMJ Case Rep 2013; doi:10.1136/bcr-2013-201554

  15. Naranjo CA, Busto U, Sellers EM, Sandor P, Ruiz I, Roberts EA, et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther 1981; 30:239–45.