Open Peer Review in Scientific Publishing: A Web Mining Study of PeerJ Authors and Reviewers

Peiling Wang 1 , Sukjin You 2 , Rath Manasa 1  and Dietmar Wolfram 2
  • 1 School of Information Sciences, University of Tennessee, TN 37996-0332, Knoxville, USA
  • 2 School of Information Studies, University of Wisconsin-Milwaukee, WI 53201, Milwaukee, USA
Peiling Wang, Sukjin You, Rath Manasa and Dietmar Wolfram
  • Corresponding author
  • School of Information Studies, University of Wisconsin-Milwaukee, Milwaukee, WI 53201, USA
  • Email
  • Search for other articles:
  • degruyter.comGoogle Scholar

Abstract

Purpose

To understand how authors and reviewers are accepting and embracing Open Peer Review (OPR), one of the newest innovations in the Open Science movement.

Design/methodology/approach

This research collected and analyzed data from the Open Access journal PeerJ over its first three years (2013–2016). Web data were scraped, cleaned, and structured using several Web tools and programs. The structured data were imported into a relational database. Data analyses were conducted using analytical tools as well as programs developed by the researchers.

Findings

PeerJ, which supports optional OPR, has a broad international representation of authors and referees. Approximately 73.89% of articles provide full review histories. Of the articles with published review histories, 17.61% had identities of all reviewers and 52.57% had at least one signed reviewer. In total, 43.23% of all reviews were signed. The observed proportions of signed reviews have been relatively stable over the period since the Journal’s inception.

Research limitations

This research is constrained by the availability of the peer review history data. Some peer reviews were not available when the authors opted out of publishing their review histories. The anonymity of reviewers made it impossible to give an accurate count of reviewers who contributed to the review process.

Practical implications

These findings shed light on the current characteristics of OPR. Given the policy that authors are encouraged to make their articles’ review history public and referees are encouraged to sign their review reports, the three years of PeerJ review data demonstrate that there is still some reluctance by authors to make their reviews public and by reviewers to identify themselves.

Originality/value

This is the first study to closely examine PeerJ as an example of an OPR model journal. As Open Science moves further towards open research, OPR is a final and critical component. Research in this area must identify the best policies and paths towards a transparent and open peer review process for scientific communication.

1 Introduction and Literature Review

Scientific research has entered a new era of Open Science, which aims to make all aspects of the scientific research process open and available by promoting new models for the dissemination of findings and the peer review process. Peer review is one of the cornerstones of scholarly inquiry, where peers assess the merits of research to determine if the claims made, methods used, findings and conclusions adhere to accepted practices in conducting scholarship. Assessment by peers also performs an important gatekeeping function, where poorly conceived research is not recommended for publication. The gold standard for peer review has been blind pre-publication review, where reviewers do not know the identities of the authors and vice versa. The longstanding rationale has been that blind review will limit bias in the review process if all parties remain anonymous. There has been growing concern with this traditional approach to peer review. Woosen (2015), for example, reports in an article in The Chronicle of Higher Education that some publishers are now making changes to the peer review process to fix what some call a broken system. Criticisms of the closed peer review process include biases, failure to weed out flawed manuscripts or the rejection of innovative ideas that do not fit believed norms, and long delays in publishing. However, not all scientists see the need for radical changes in the peer review process.

To date, there has been much research that has investigated different aspects of Open Science such as Open Access (OA) to documents that report research findings in OA journals or publicly accessible repositories, and open data, which make the data collected, generated, or processed for research studies openly available to others. Open Access to scientific journals is growing steadily (Laakso & Björk, 2012). Although many OA journals have adopted single or double blind Closed Peer Review (CPR), a number of more recent OA journals launched in this century have not passed the scrutiny of quality control (Bohannon, 2013), and may be considered predatory journals (Bartholomew, 2014). Open Science cannot succeed if the peer review process is flawed or compromised. One aspect of Open Science that has not been widely adopted and investigated is that of Open Peer Review (OPR), which makes pre- or post-publication reviews of scholarly works publicly available. In recent years, several OA journals have pioneered the OPR process to vet manuscripts for publication. Today, the notion of OPR is still evolving; diverse models of OPR have been implemented from the most open and interactive process, such as by F1000Research, to blind peer review followed by optional publishing of review history, such as by PeerJ. A dual model of peer review as used by Papers in Physics allows the author to choose, at the time of submission, either CPR or OPR. Despite the lack of standards or de facto models, OPR journals are growing fast. The number of OPR journals indexed by the Directory of Open Access Journals1 increased notably from 20 in January 2016 to 77 in September 2016 (285%).

The idea of OPR is not new, but it has not been widely adopted to date. In an early investigation of this issue, McNutt et al. (1990) examined the quality of submitted reviews to a medical journal based on blind and signed reviews. The authors found there was no association between review quality and signing, but concluded blinding improved the quality of reviews based on human judgments. This latter conclusion has changed over time. Proposals for OPR extend back at least to the early days of Web-based Open Access journals. Sumner and Shum (1996) proposed pre- and post-publication OPR (which they called computer-supported collaborative argumentation) for a newly created electronic OA journal, arguing that OPR was central to the journal’s operation and for opening up scholarly debate. The British Medical Journal (now BMJ) began experimenting with OPR in the late 1990s to determine if peer review quality was different when referees identify themselves. van Rooyen et al. (1999) in studying OPR in BMJ with blind reviews noted that there was no difference in quality between the open and blind reviews based on human judgments. They concluded there was no evidence to support that the traditional blind approach to peer review resulted in superior reviews. They also posited that if reviewers were identifiable, they might put greater effort into their reviews. Although these experiments with BMJ began more than 15 years ago, it was not until September 2014 that the journal fully adopted pre-publication OPR2. In a similar study, Walsh et al. (2000) conducted a randomized trial where reviewers were randomly assigned to an anonymous or signed group for their reviews. The authors found open peer review was feasible based on the percentage of reviewers who were willing to forgo anonymity. The quality of reviews was no different than for anonymous reviews, which took longer to complete and were more courteous.

The debate whether the traditional peer review model should be modernized to promote transparency in the referee process or to accelerate dissemination of scientific discoveries has been a topic of continued interest (e.g. Taylor & Francis Group, 2015). Scientists, such as Nobel Laureate Harald zur Hausen, openly endorse the OPR model3. Editors and researchers promote OPR, arguing that making the contents of the peer reviews openly accessible can improve the quality of peer review and accountability of both authors and reviewers (Groves, 2010; Hunter, 2012; Pöschl & Koop, 2008; Soergel, Saunders, & McCallum, 2013). Scientists in favor of CPR believe it to be the most trusted approach to evaluating research reports and that the benefits outweigh the problems or inherent limitations (Khan, 2010; Nicholas et al., 2015; Rennie, 2016; Taylor & Francis Group, 2015).

Critics of CPR have argued that the blind peer review process is, at a minimum, flawed (Smith, 2006) or, at worst, broken (McCook, 2006). Aside from opinions (e.g. Kriegeskorte, Walther, & Deca, 2012; Whither Science Publishing, 2012) or theoretical discussion (e.g. Lee et al., 2013), there is a lack of empirical research examining the complex peer review process to provide fact-based insights. It is obvious that in the blind peer review process the original review reports are only accessible by the editors. As OPR journals are opening the peer review process and sharing review reports, researchers now have the opportunity to study the process and review the discourses. Furthermore, Web 2.0 technology has also provided useful tools to collect data to help in the analysis of these processes.

Led by the European Union, Open Science is moving towards the ambitious goal “to open access to scientific publications as the default option by 2020” (Council of the European Union, 2016, p. 12; Enserink, 2016). Publications for Open Access will still need peer review. In a recent article in Nature, Rennie (2016) states that OPR provides a more ethical approach to peer review and calls for rigorous studies to compare various models (open, blind, pre- and post-publication, portable and so on).

2 Framework and Research Questions

We developed a descriptive model to frame the research questions and guide data collection (Figure 1). PeerJ uses a single blind, CPR process. As the established model for journal publishing, CPR assigns an editor to handle a submission and solicit peers to review the manuscript before making a decision. During the review process, the editor communicates with the corresponding author on review results and the decision. A manuscript may be resubmitted after revisions and go through one or more rounds of peer review before a final decision. In a published article, two types of review history can be accessed: for all articles, a summary of the processing dates (submission, revision, and acceptance), and for some, if the author chose this option, a full peer review history. The full peer review history includes: (1) all previous versions of the article; (2) all decision letters corresponding to the versions; (3) all review reports corresponding to the versions (the reviewers may have signed the reports or remained anonymous), and (4) all rebuttal letters as attachments (See the dashed objects depicted in Figure 1).

Figure 1
Figure 1

Model of OPR implemented in PeerJ.

Citation: Journal of Data and Information Science 1, 4; 10.20309/jdis.201625

As literature reports, OPR may adopt different levels of openness. The most transparent peer review model opens the entire review process, during which both reviewers and authors are known to each other. Several BMJ journals moved from CPR to this OPR model; F1000Reserch adopted OPR with a post-publication peer review implementation. Transparent peer review has one more decision element: whether to make the signed review reports public alongside the articles. A less rigid model allows reviewers the option of anonymity or signing their identity, and authors the option of publishing their reviews or keeping them private. A variable level of transparency exists between CPR and OPR. It is the combination of a CPR model with an OPR model that is of particular interest in the current research. Do authors choose to make their reviews available? How willing will reviewers be to identify themselves? Answers to these questions will inform decisions on how to adopt OPR.

To study the complex OPR process and understand the nature of OPR as it stands in current practice, this research examines the peer review process adopted by the OPR journal PeerJ. PeerJ4 was chosen because it serves as an example of an OA journal that straddles the world of CPR and OPR by allowing authors to decide if the review histories will be published alongside their papers and allowing reviewers to sign their reports or remain anonymous. PeerJ’s publishing model made the news in The Scientist at its inception on February 12, 2013 (Zielinska, 2013). However, PeerJ is not the first to adopt an OPR policy. It is not on the list of the 77 OPR journals in the DOAJ; PeerJ is listed among the 1,617 [single] blind peer review journals (Search performed on September 10, 2016).

Nevertheless, PeerJ has succeeded as a fast growing OPR journal for research articles since its inception and gained its first (partial) Impact Factor of 2.183 (2015). The journal is of great interest because of its peer review policies and processes. PeerJ consists of three publications: PeerJ (Life, Biology, & Health Science), PeerJ Computer Science since May 27, 2015, and PeerJ Preprints (not peer reviewed). This research collected data from PeerJ (Life, Biology, & Health Science) and addresses the following specific research questions:

  1. How have scientists accepted principles of Open Peer Review as represented in PeerJ?
    • From which countries do the authors who participate in Open Peer Review originate?
    • Has there been a change over time in the proportion of articles with full peer review histories appearing alongside published articles?
    • Has there been a change over time in the proportion of reviewers who identify themselves (i.e. signed their reports) as represented in the full peer review histories?
  2. What does the peer review history reveal about the review process?
    • What is the time frame for completion of peer review and for publication of accepted articles?
    • What is the time taken to submit reports in the first round of review by referees?
    • Is there a difference in the number of rounds of reviews prior to acceptance based on signed and anonymous reviews?
  3. Is there a difference between anonymous and signed review reports:
    • in the effort measured by the number of sentences and number of tokens per review?
    • in the time taken to submit the first round of review?
  4. Is there a correlation between the length of the report and the time taken to submit the report?

3 Research Methods

This study adopts a Web mining approach to collect OPR data from PeerJ and applies several analytical techniques and tools. The use of Web analytics to study OA online journals faces unique challenges due to the complexity of the articles and associated objects as well as the lack of standards for OPR platform design. No well-developed computational tools exist for data collection, and quantitative and qualitative analysis except for a few project-based experimental tools that need to be modified for use. This Section describes the tools used to manage data collection, extraction, cleansing, restructuring, and analytical mining for the present study.

3.1 Web Scraping and Data Extraction

PeerJ publishes all articles as Web pages with a right sidebar for hyperlinks to a downloadable PDF file, a peer review history page, article level metrics, and sections of the article (Figure 2).

Figure 2
Figure 2

PeerJ articles and meta sidebar link to peer review history.

Citation: Journal of Data and Information Science 1, 4; 10.20309/jdis.201625

For the articles with a full review history, the review page has a summary of the publication process (dates on initial submission, initial decision, revision, acceptance, etc.) followed by the peer review history, including the editor’s decision (accept, major revision, or minor revision) and review reports (signed or anonymous), all versions of the article, and rebuttals. For the articles whose authors choose not to publish the full peer review history, the review page provides only a summary of the publication process. Because the data relevant to our research purpose were scattered across different pages and in diverse formats, we divided the data collection into two processes: (1) the use of Google Chrome SelectorGadget to collect each article’s data, and; (2) the use of a PHP program developed by one of the authors to crawl the full peer review history Web pages. Fortunately, the URLs for PeerJ articles are well structured with a domain and a sequential article number.

We extracted three sets of data:

  1. The article data include the article identification (articleID is the sequential number of the URL), authors, publication dates (submission, acceptance, and publication), grants, and affiliations (author’s organization and country). Affiliations are associated with each article instead of individual authors in this study due to difficulties in definitively assigning authors to specific affiliations.
  2. The Peer Review History: Summary: All published articles have a summary that includes the article’s identification (articleID is the same sequential number of the URL) and dates for submission, revisions, and acceptance; the summary history does not include the editor’s identity.
  3. The Peer Review History: Review Reports: These are available only if the authors have opted to make their reviews public. The full history page includes the summary data as described in 2) plus the dated and named editor’s decision letters (each for a specific version), referees (either signed or anonymized as Reviewer 1, Reviewer 2, etc.), and referee’s review reports (each report is for a specific version). Some review reports also referred to marked manuscripts as being attached, etc. (Figure 3).

Figure 3
Figure 3

PeerJ review history screenshot.

Citation: Journal of Data and Information Science 1, 4; 10.20309/jdis.201625

The data collected represent accepted papers published by PeerJ between its first published article in 2013 and February 4, 2016. This resulted in 1,643 articles with ID numbers from 1 to 1,676. The missing numbers were likely due to cancelled manuscripts or retracted papers. Of the 1,643 articles, 1,214 (73.89%) published full review histories. Ten of the 1,214 articles were accepted by the academic editors based on portable reviews submitted by the authors with the manuscripts. These portable reviews were review reports from other journals that had rejected the manuscripts. Excluding these ten articles with portable peer reviews, there were 3,569 PeerJ peer review reports for the 1,204 articles’ full review histories. Each review history page (Figure 3) includes time stamps, versions, the editor’s letters, reviewer reports, and author rebuttals as attached PDF files. Each report includes four sections: basic reporting, experimental design, validity of the findings, and comments for the author. Of the 3,569 review reports, 85 were submitted as attachments in various formats (comments inserted to the manuscripts, PDF or Word DOC) or the reviews contained no substantive content such as “no further comments.” There were 3,484 valid reviews after removal of the reviews without any content.

The available data that were not extracted include: (1) author rebuttals, which are shown in the full peer review history as downloadable files (DOC or PDF); (2) affiliations of the signed referees, which is available only if the referee’s profile is public (but most reviewers set their profiles to private), and; (3) files which reviewers attached as review reports or marked manuscripts.

3.2 Data Cleaning and Restructuring

The data required substantial cleaning and restructuring, for example, where reviews provided no substantive comments (e.g. “N.A.”, “See above”). We did not correct misspellings. The cleaned data were restructured using Java and Python to produce tab-delimited files. Additional data processing consolidated variations in country names. For example, country names were not standardized in PeerJ, thus records showed PR China, Peoples Republic of China or China; United States, US, or USA. Structured data were imported into a relational database (ACCESS and SQL server) to ensure data integrity, from which analytical outputs were exported to MS Excel, SPSS, SAS, and WordStat for quantitative and qualitative analyses.

3.3 Data Analysis

Quantitative data were tabulated for statistical analysis to characterize PeerJ’s authors and reviewers, review process, and comparisons of the set of articles with the full peer review history (such as the rounds of reviews, the versions, and the editor’s decisions). For the review reports, the length of the reviews was measured by tokens and sentences to compare signed vs. anonymous review reports.

4 Results & Discussions

This project collected 1,643 peer-reviewed articles, which PeerJ published from February 12, 2013 to February 4, 2016. The time period was further grouped into 12 data periods consisting of three-month intervals in some analyses. As noted above (see Section 3.1), 10 of the 1,643 articles were found to be special cases of portable reviews. The ten articles were included in the descriptive analysis (such as article’s country of origin, versions, and author’s decisions) but their external review reports were not included in text analysis of the PeerJ review reports.

4.1 Countries of the OPR Participating Authors

The affiliation of authors’ organizations and countries is complicated by the journal format, in which one author may be associated with multiple organizations and several authors may be associated with one organization. Each country, therefore, was counted once regardless of how many author affiliations were from a given country. The contributing authors of the 1,643 articles are affiliated with organizations in 107 countries. The top five countries (Table 1) for the number of articles contributed, which include the USA, United Kingdom, Australia, and Canada, are also the top contributing countries of articles that published full peer review histories. China and Germany switched order in ranking for the number of articles and the number of articles with published full review histories. The top five countries’ articles with full peer reviews are similar to the study of a post-publication OPR journal based in the United Kingdom, F1000Research, as reported in Wang et al. (2016), although the ranking order is slightly different in that Germany was ranked 4th instead of 5th or 6th in PeerJ.

Table 1

Ranking and percentage of the countries whose authors participating in OPR.

RankingCountry of originNumber of articlesCountry of originNumber of articles with published review reportsPercentage
1USA700USA53676.57%
2UK254UK20279.53%
3Australia136Australia9872.06%
4Canada100Canada6969.00%
5China96Germany6773.63%
6Germany91China5860.42%
7Spain61Spain4980.33%
8France59France4576.27%
9Netherlands50Netherlands4182.00%
10Japan49Japan3163.27%

4.2 The Proportion of the Articles with a Full Peer Review History

Of the 1,643 articles, 1,214 articles (73.89%) published a full peer view history (including 10 portable reviews). Figure 4 shows the trend of the published articles over three years as 12 data points. The published articles have been steadily increasing over time, but the proportion of the articles with a published full peer review history ranges between 64.79% and 83.46% without an obvious increase or decrease overall.

Figure 4
Figure 4

A comparison of articles with and without full peer view history.

Citation: Journal of Data and Information Science 1, 4; 10.20309/jdis.201625

4.3 Comparison of Signed with Anonymous Peer Review Reports

The 1,214 articles’ full review histories include 2,855 decision letters by the academic editors (including the 10 special cases with portable review histories) and 3,569 peer review reports (excluding 10 portable review reports). Of the 3,569 review reports alongside the 1,204 articles, 1,543 (43.23%) were signed reports and 2,026 (56.77%) were anonymous reports. As depicted in Figure 5, all data points except for one show more anonymous reports than signed reports. Although the 8th point for November 2014 to January 2015 had 27.08% more signed review reports, the subsequent data points have not followed this increase. If this data point and the last data point are excluded as outliers, the proportion of signed reports over the three-month periods ranged from 40.74% to 48.86%. No obvious trend is observed.

Figure 5
Figure 5

Comparison of signed or anonymous review reports over three-month intervals 2013–2016.

Citation: Journal of Data and Information Science 1, 4; 10.20309/jdis.201625

4.4 Peer Review Process — Time Delays

Himmelstein (2015) ranked 16 journals by measuring acceptance delays and publication delays. PeerJ had the shortest period for peer review with 74 days median time from receipt of a manuscript to acceptance for publication based on 777 articles, but ranked only 8th with a median value of 22 days from acceptance for publication to actual publication. A time series analysis was performed to depict the time lags for acceptance delay and publication delay. Figure 6 plots the means and medians for the delays over the three years in a three-month interval. Publication delays remain close to 25 days over time, whether measured by mean or median number of days. Acceptance delays, that is the peer review time, have increased from a mean of 55 days to a mean of 106 days over the past three years. The mean and median values both show the same trend, although the median value is the measure used by publishers because the data usually are skewed and may contain outliers. Following a pre-publication CPR process, PeerJ faces the same challenges as most print CPR journals: time for peer review increases as the number of submissions increases. Our results are similar to the findings of Björk and Solomon (2013). They investigated scholarly peer reviewed journals for publishing delays using a stratified random sample of 2,700 papers from 135 journals. They found that delays ranged from nine months to 18 months. They also found that the online OA journals showed a shorter time from acceptance to publication. The various factors contributing to the journal publishing delays include disciplines, size of the journal, and authors’ revision time.

Figure 6
Figure 6

Delays in peer review and delays in publication.

Citation: Journal of Data and Information Science 1, 4; 10.20309/jdis.201625

4.5 Time Taken to Submit First Round of Peer Reviews

The first round of peer reviews was mostly conducted on the first version of the manuscripts, except for four articles where it was conducted on the second version. The editors reviewed the original submissions, which the authors revised accordingly. The times taken were calculated based on the difference between the time stamp of the review report and the manuscript submission date. The reviewers may have received the manuscript review request on a later date. The receipt date is not recorded by PeerJ, so it is unknown. The descriptive statistics for the first round of review reports (N = 2,720) are Mean = 20.36 days (SD = 11.57) and Median = 18.00 days. This set was partitioned into two sets to compare differences between anonymous reports (Na = 1,537) and signed reports (Ns = 1,183). The descriptive statistics are as follows: Meana = 20.70 days (SD = 11.75), Mediana = 18.00 days, ranging between 0 to 77 days; Means = 19.90 days (SD = 11.31), Medians = 18.00 days, ranging between 0 to 69 days (Figure 7). Although no significant statistical differences were shown across the two sets, the means and ranges show differences. We found that 15 review reports were submitted the same day or next day. In contrast, 280 review reports were submitted after more than 30 days. Further analysis is needed to compare the distributions of the time taken and the review text lengths (Section 4.8).

Figure 7
Figure 7

Boxplot of the first round of review time taken to submit report.

Citation: Journal of Data and Information Science 1, 4; 10.20309/jdis.201625

4.6 Peer Review Process — Acceptances Related to Versions

PeerJ reports that the manuscript acceptance rate is between 60% and 70%5. Our data show that the version of the accepted articles ranges from the first version up to the fifth version. Of the 1,214 published articles with full review histories, 13 (1.07%) were accepted without revision and 382 (31.47%) underwent multiple revisions; the majority (67.46%) underwent one revision. The distribution of the 1,214 articles in three groups based on identities attached to the review reports (signed, anonymous, or a mix of signed and anonymous names) was compared using a chi-square analysis to determine if the number of versions submitted before acceptance is associated with the anonymity of the reviewers. A summary of the distribution of the frequency data appears in Table 2. To provide sufficient observations within each cell to calculate the chi-square outcome, tallies for versions 1 and 2 were collapsed, as were tallies for versions 4 and 5. The resulting chi-square value is 1.67 with df = 4 and p = 0.796. Therefore, there is no significant relationship between the number of manuscript versions authors submit prior to article acceptance and the reviewer anonymity (signed, mixed or anonymous).

Table 2

Acceptance of the types of reviews for articles by versions.

VersionAnonymousBothSignedExternal*Total
1250613
22474231454819
393182560331
4161811045
506006
Total359633212101,214
Percentage29.8252.5717.61

Note.

4.7 Text Features of Signed vs. Anonymous Review Reports

The lengths of reviews were measured by the number of sentences and the number of tokens (i.e. words). The Python text processing module was used to calculate the numbers of sentences and tokens. This analysis focused on the first round of review reports because the versions affect the availability of review reports and the first review reports are presumably the most thorough and detailed. The subsequent reviews were either brief or not submitted. There were a total of 2,720 valid first round of review reports for the 1,204 articles after excluding the reports submitted as attachments or no substantive content (Table 3).

Table 3

Review length of the first round of all articles (Na = 1,537; Ns = 1,183).

SentencesTokens
AnonymousSignedAnonymousSigned
Mean28.0928.81477.16508.41
Std. Dev.23.7424.07389.87427.25
Median2222368394
Min1187
Max2751672,6043,747
T test Prob.0.4320.050

There is no significant difference in review lengths using a t test with unequal variances assumed. However, the average length of the reports by signed reviewers was longer than those by anonymous reviewers based on the number of tokens. The reported probability value of 0.05 is not sufficient for us to conclude there is a significant difference in the number of tokens between anonymous and signed reviews, noting recent concerns expressed with the use of p-values to assess significance (Baker, 2016).

There is evidence that the same referee reviewing different articles is likely to submit reports of different lengths. For example, three reviews by one referee for three different articles have lengths of 213, 1,185, and 286 tokens, respectively. Another referee’s two reports for two different articles were 274 and 995 tokens, respectively. The differences in the lengths of the reviews and the reviewer decisions to sign their reviews may be influenced by a number of factors related to the manuscripts such as the nature of the research, the topics undertaken and the quality or rigor of the research being evaluated. It is, therefore, also necessary to analyze the set of articles in which each article had both signed and anonymous review reports. We identified 633 such articles (Na = 775 and Ns = 749). The results (Table 4) show that there are no differences in tokens or sentences between the anonymous and signed reports. Therefore, we cannot conclude from the analysis that reviewers’ decisions to sign their reviews are associated with the length of the reviews.

Table 4

First round of review of the articles (each had both signed and anonymous reports).

SentencesTokens
AnonymousSignedAnonymousSigned
Mean27.7127.90478.93495.25
Std. Dev.21.8324.16385.02431.15
Median22.0021.00382.00372.00
Min1197
Max1431672,6043,435
T test Prob.0.8740.436

4.8 Comparisons of Time Taken to Submit Anonymous vs. Signed Reports

As mentioned in Section 4.5, the time taken to complete the first round of review shows a wide range. Figure 8 plots the 1,534 review reports from the set of 633 articles that each had at least one anonymous and one signed review report. For this set of articles, we can observe if the distributions of times differ between anonymous and signed review reports. Figure 8 plots the percentage of reports submitted over days. The two plots show similar patterns with a spike for signed reports at 13 days and for anonymous reports at 14 days. For anonymous reports, the plot shows 4 cases at 70 days or more. There were 15 review reports (0.98%) submitted the same day or next day and 280 review reports (18.22%) submitted after more than 30 days. This raises the question if there is an association between the time taken and the length of the reviews (See Section 4.9).

Figure 8
Figure 8

Time taken to submit review reports.

Citation: Journal of Data and Information Science 1, 4; 10.20309/jdis.201625

4.9 Correlation between Time-taken to Submit and Length of Report

For the 15 quickest turnaround reviews (same or next day), the mean report length is 17.60 sentences or 284.73 tokens. For the much delayed reviews (more than 60 days), the mean report length of 25.75 sentences or 458.41 tokens. Spearman’s rho procedures were applied to test correlations between time taken and the length of the report. There was no significant correlation between the time taken and report length for both the number of sentences (p = 0.386) and the number of tokens (p = 0.944) for all first round of reviews. Similarly, for all first round of reviews for submissions containing both signed and anonymous reports, there was no significant correlation between the time taken and the number of sentences (p = 0.452) and the number of tokens (p = 0.830). This indicates that longer submission times for reviews do not lead to longer or shorter reviews.

It must be pointed out as well that the longer turnaround could be influenced by longer periods of time between the manuscript submission date and the date when a reviewer receives her or his invitation. Therefore, additional stamps are needed to provide accurate assessments of the time taken to submit a review.

5 Limitations

Although the current study provides insights into author and reviewer acceptance of OPR in PeerJ, the study of a single OA journal over a relatively short period of time does not permit the prediction of long-term acceptance of OPR or to generalize our findings to other OPR journals. Given the recent adoption of different OPR models by current or new OA journals, there is a lack of substantial datasets across large numbers of journals or over long periods to analyze and compare. Furthermore, the data are not always readily accessible using current computational tools. In addition, the peer review history data are not available for the articles for which authors chose not to publish full peer review histories. Finally, the anonymity of reviewers prevents an accurate count of how many reviewers contributed to the review process. To reduce the effect of these limitations, the analysis in this study has been at the level of the articles or review reports rather than at the level of authors or reviewers when appropriate.

6 Conclusions and Further Research

This study has presented a detailed and systematic analysis of an OPR model as implemented in PeerJ, an OA journal that straddles the world of CPR and OPR. PeerJ article Web pages were systematically crawled to collect relevant article metadata and peer review history data. PeerJ is an international journal, with contributed articles by authors from 107 countries, but is still dominated by contributions from North America and Western Europe, with some representation from countries in East Asia and Oceania. A majority of articles (73.89%) have publicly available full peer review histories alongside the articles. The percentage of published full peer review histories has remained stable over the three-year period analyzed; the ratio of anonymous/signed reviewers has also remained stable during this period of time. Given a choice, the majority of authors publishing in PeerJ will make the peer review histories of their articles public. However, when it comes to reviewers signing their reports, the majority of reviewers still prefer to remain anonymous. As a CPR OA journal, the delay in peer review for PeerJ submissions has been increasing, from less than 50 days (Median) to 88 days (Median), despite the publication delay from acceptance being stable and less than 30 days. The volume of the journal has also grown, having experienced a four-fold increase in the number of articles published between the first time period and the most recent time period studied.

The vast majority (98.93%) of submissions have undergone at least one round of revision before acceptance, indicating that there is rigor in the review process. The lack of a significant difference between the number of rounds of reviews by signed and anonymous authors provides an indication that reviewer anonymity does not significantly influence when a submission is accepted. The finding that reports by signed reviewers could be longer than by anonymous reviewers based on the number of tokens suggests that signed reviewers may be putting more effort into their reports. However, this simple analysis does not reveal whether there are qualitative differences between the reviews of signed vs. anonymous reviewers and the statistical outcome was not definitive. This result is similar to the findings in Bornmann, Wolf, and Daniel (2012) that the comments in public peer review are much longer than the comments in closed peer review.

There is no significant difference in the time taken to submit reviews or the lengths of reports between the signed or anonymous review reports. This finding suggests that reviewers conduct their reviews in similar manner whether or not they sign their reports. Our results corroborate the findings of the experiment by van Rooyen, Delamothe, and Evans (2010). In their experiment, the participants were randomly assigned to a control or intervention group. The participants in intervention group were told that their review reports would be published alongside the paper. The authors found there was no significant difference in review quality between the two groups, although the intervention group took longer to review (mean difference 25 minutes).

The current research represents a first attempt to observe the characteristics of OPR and the peer review process using a Web mining approach to collect all the articles published in PeerJ up to February 4, 2016. One aspect of the reviews that the current research did not assess was the quality of the reviews based on semantic analysis of the reviews. Although text length provides an indication of the efforts by reviewers, a higher level of analysis is needed to focus on concepts that reviewers have identified to be of concerns or need additional work. Some of the texts show affective responses in the review process as well. For example, in a comment in the Section of “Comment for the author” the reviewer wrote “Cool paper.” As we move forward with more in-depth semantic analysis, we will investigate the conceptual and affective level of the reviews. Specifically, the analyses will focus on a comparison of the texts of signed and anonymous review reports using sentiment analysis and other natural language processing techniques, along with other text mining tools. The research will also be expanded to include additional OPR journals that use different models of OPR (e.g. pre-publication, post-publication, multistage) to provide further insights into OPR trends. Are scientists ready to accept a completely transparent peer review model? Which models will be adopted widely by different scientific disciplines? Our future research will address these important questions.

Author Contributions: P.L. Wang (peilingw@utk.edu) initiated and planned the project including database design. D. Wolfram (dwolfram@uwm.edu, corresponding author) and P.L. Wang jointly designed the conceptual framework and methodology, analyzed data, and wrote the majority of the manuscript. S. You (yous@uwm.edu) collected, cleaned, and structured data on peer review process and review reports, developed Python and Web scraping programs, and conducted preliminary text and numeric analysis. R. Manasa (rath@utk.edu) collected, cleaned, and structured data on published articles using SelectorGadget and R, and conducted preliminary text analysis using WordStat.

References

  • Baker, M. (2016). Statisticians issue warning over misuse of P values. Nature, 531(7593), 151.

  • Bartholomew, R.E. (2014). Science for sale: The rise of predatory journals. Journal of the Royal Society of Medicine, 107(10), 384–385.

  • Björk, B.C., & Solomon, D. (2013). The publishing delay in scholarly peer-reviewed journals. Journal of Informetrics, 7(4), 914–923.

  • Bohannon, J. (2013). Who’s afraid of peer review? Science, 342(6154), 60–65.

  • Bornmann, L., Wolf, M., & Daniel, H.D. (2012). Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics, 91, 843–856.

  • Council of the European Union. (2016). Outcome of the council meeting, 3470th council meeting: Competitiveness (internal market, industry, research and space), Brussels, 26 and 27 May 2016. Retrieved on July 16, 2016, from http://www.consilium.europa.eu/en/meetings/compet/2016/05/st09357_en16_pdf/.

  • Enserink, M. (2016). In dramatic statement, European leaders call for ‘immediate’ open access to all scientific papers by 2020. Science, News, May 27, 2016. Retrieved on July 16, 2016, from http://www.sciencemag.org/news/2016/05/dramatic-statement-european-leaders-call-immediate-open-access-all-scientific-papers/.

  • Groves, T. (2010). Is open peer review the fairest system? Yes. BMJ, 341, c6424.

  • Himmelstein, D. (2015). Publication delays at PLOS and 3,475 other journals. Satoshi Village. Retrieved on April 16, 2016, from http://blog.dhimmel.com/plos-and-publishing-delays/.

  • Hunter, J. (2012). Post-publication peer review: Opening up scientific conversation. Frontiers in Computational Neuroscience, 6, 63.

  • Khan, K. (2010). Is open peer review the fairest system? No. BMJ, 341, c6425.

  • Kriegeskorte, N., Walther, A., & Deca, D. (2012). An emerging consensus for open evaluation: 18 visions for the future of scientific publishing. Frontiers in Computational Neuroscience, 6, 94.

  • Laakso, M., & Björk, B.C. (2012). Anatomy of open access publishing: A study of longitudinal development and internal structure. BMC Medicine, 10, 124.

  • Lee, C.J., Sugimoto, C.R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.

  • McCook, A. (2006). Is peer review broken? Submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. What’s wrong with peer review? The Scientist, 20(2), 26–35.

  • McNutt, R.A., Evans, A.T., Fletcher, R.H., & Fletcher, S.W. (1990). The effects of blinding on the quality of peer review. A randomized trial. Journal of the American Medical Association, 263(10), 1371–1376.

  • Nicholas, D., Watkinson, A., Jamali, H.R., Herman, E., Tenopir, C., Volentine, R., Allard, S., & Levine, K. (2015). Peer review: Still king in the digital age. Learned Publishing, 28, 15–21.

  • Pöschl, U., & Koop, T. (2008). Interactive open access publishing and collaborative peer review for improved scientific communication and quality assurance. Information Services & User, 28, 105–107.

  • Rennie, D. (2016). Make peer review scientific. Nature, 535 (July 7), 31–33.

  • Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182.

  • Soergel, D., Saunders, A., & McCallum, A. (2013). Open scholarship and peer review: A time for experimentation. Retrieved on April 17, 2016, from http://tinyurl.com/h3jbkdz/.

  • Sumner, T., & Shum, S.B. (1996). Open peer review & argumentation: Loosening the paper chains on journals. Retrieved on July 17, 2016, from https://www.ariadne.ac.uk/issue5/jime/.

  • Taylor & Francis Group. (2015). Peer review in 2015: A global view. Retrieved on July 17, 2016, from http://authorservices.taylorandfrancis.com/wp-content/uploads/2015/10/Peer-Review-2015-white-paper.pdf/.

  • van Rooyen, S., Delamothe, T., & Evans, S.J.W. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the Web: Randomised controlled trial. BMJ, 341, c5729.

  • van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. British Medical Journal, 318(7175), 23–27.

  • Wang, P., Rath, R., Deike, M., & Wu, Q. (2016). Open post publication peer review: An innovation in scientific publishing. Retrieved on July 17, 2016, from https://www.ideals.illinois.edu/handle/2142/89432.

  • Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: A randomised controlled trial. The British Journal of Psychiatry, 176(1), 47–51.

  • Whither Science Publishing. (2012). As we stand on the brink of a new scientific age, how researchers should best communicate their findings and innovations is hotly debated in the publishing trenches. The Scientist, August 1. Retrieved on July 17, 2016, from http://www.the-scientist.com/?articles.view/articleNo/32378/title/Whither-Science-Publishing/.

  • Woosen, P. (2015). Journal publishers rethink a research mainstay: Peer review. The Chronicle of Higher Education, October 11. Retrieved on July 17, 2016, from http://www.chronicle.com/article/Academic-Publishing-Toward-a/236526.

  • Zielinska, E. (2013). Open-review journal launched. The Scientist, February 13. Retrieved on July 17, 2016, from http://www.the-scientist.com/?articles.view/articleNo/34367/title/Open-Review-Journal-Launched/.

Footnotes

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Baker, M. (2016). Statisticians issue warning over misuse of P values. Nature, 531(7593), 151.

  • Bartholomew, R.E. (2014). Science for sale: The rise of predatory journals. Journal of the Royal Society of Medicine, 107(10), 384–385.

  • Björk, B.C., & Solomon, D. (2013). The publishing delay in scholarly peer-reviewed journals. Journal of Informetrics, 7(4), 914–923.

  • Bohannon, J. (2013). Who’s afraid of peer review? Science, 342(6154), 60–65.

  • Bornmann, L., Wolf, M., & Daniel, H.D. (2012). Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics, 91, 843–856.

  • Council of the European Union. (2016). Outcome of the council meeting, 3470th council meeting: Competitiveness (internal market, industry, research and space), Brussels, 26 and 27 May 2016. Retrieved on July 16, 2016, from http://www.consilium.europa.eu/en/meetings/compet/2016/05/st09357_en16_pdf/.

  • Enserink, M. (2016). In dramatic statement, European leaders call for ‘immediate’ open access to all scientific papers by 2020. Science, News, May 27, 2016. Retrieved on July 16, 2016, from http://www.sciencemag.org/news/2016/05/dramatic-statement-european-leaders-call-immediate-open-access-all-scientific-papers/.

  • Groves, T. (2010). Is open peer review the fairest system? Yes. BMJ, 341, c6424.

  • Himmelstein, D. (2015). Publication delays at PLOS and 3,475 other journals. Satoshi Village. Retrieved on April 16, 2016, from http://blog.dhimmel.com/plos-and-publishing-delays/.

  • Hunter, J. (2012). Post-publication peer review: Opening up scientific conversation. Frontiers in Computational Neuroscience, 6, 63.

  • Khan, K. (2010). Is open peer review the fairest system? No. BMJ, 341, c6425.

  • Kriegeskorte, N., Walther, A., & Deca, D. (2012). An emerging consensus for open evaluation: 18 visions for the future of scientific publishing. Frontiers in Computational Neuroscience, 6, 94.

  • Laakso, M., & Björk, B.C. (2012). Anatomy of open access publishing: A study of longitudinal development and internal structure. BMC Medicine, 10, 124.

  • Lee, C.J., Sugimoto, C.R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.

  • McCook, A. (2006). Is peer review broken? Submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. What’s wrong with peer review? The Scientist, 20(2), 26–35.

  • McNutt, R.A., Evans, A.T., Fletcher, R.H., & Fletcher, S.W. (1990). The effects of blinding on the quality of peer review. A randomized trial. Journal of the American Medical Association, 263(10), 1371–1376.

  • Nicholas, D., Watkinson, A., Jamali, H.R., Herman, E., Tenopir, C., Volentine, R., Allard, S., & Levine, K. (2015). Peer review: Still king in the digital age. Learned Publishing, 28, 15–21.

  • Pöschl, U., & Koop, T. (2008). Interactive open access publishing and collaborative peer review for improved scientific communication and quality assurance. Information Services & User, 28, 105–107.

  • Rennie, D. (2016). Make peer review scientific. Nature, 535 (July 7), 31–33.

  • Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182.

  • Soergel, D., Saunders, A., & McCallum, A. (2013). Open scholarship and peer review: A time for experimentation. Retrieved on April 17, 2016, from http://tinyurl.com/h3jbkdz/.

  • Sumner, T., & Shum, S.B. (1996). Open peer review & argumentation: Loosening the paper chains on journals. Retrieved on July 17, 2016, from https://www.ariadne.ac.uk/issue5/jime/.

  • Taylor & Francis Group. (2015). Peer review in 2015: A global view. Retrieved on July 17, 2016, from http://authorservices.taylorandfrancis.com/wp-content/uploads/2015/10/Peer-Review-2015-white-paper.pdf/.

  • van Rooyen, S., Delamothe, T., & Evans, S.J.W. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the Web: Randomised controlled trial. BMJ, 341, c5729.

  • van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. British Medical Journal, 318(7175), 23–27.

  • Wang, P., Rath, R., Deike, M., & Wu, Q. (2016). Open post publication peer review: An innovation in scientific publishing. Retrieved on July 17, 2016, from https://www.ideals.illinois.edu/handle/2142/89432.

  • Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: A randomised controlled trial. The British Journal of Psychiatry, 176(1), 47–51.

  • Whither Science Publishing. (2012). As we stand on the brink of a new scientific age, how researchers should best communicate their findings and innovations is hotly debated in the publishing trenches. The Scientist, August 1. Retrieved on July 17, 2016, from http://www.the-scientist.com/?articles.view/articleNo/32378/title/Whither-Science-Publishing/.

  • Woosen, P. (2015). Journal publishers rethink a research mainstay: Peer review. The Chronicle of Higher Education, October 11. Retrieved on July 17, 2016, from http://www.chronicle.com/article/Academic-Publishing-Toward-a/236526.

  • Zielinska, E. (2013). Open-review journal launched. The Scientist, February 13. Retrieved on July 17, 2016, from http://www.the-scientist.com/?articles.view/articleNo/34367/title/Open-Review-Journal-Launched/.

OPEN ACCESS

Journal + Issues

Search