Specialized Workbenches provide access to additional datasets not included automatically in the ICSR Lab. For access to these datasets, during the application process you will be prompted to download and complete a Word document with specific questions relating to your planned use of the Specialized Workbench datasets. Please note that applications which include access to a Specialized Workbench, such as the Peer Review Workbench, are reviewed by external experts in the relevant field. This process may therefore take longer than for regular ICSR Lab applications.
Peer Review Workbench
This Workbench provides access to summarized metadata of the peer reviews for over one million proprietary Elsevier journal manuscripts submitted between 2018 and 2021 (updated annually), enabling systematic analysis of the peer review process across different disciplines, at scale.
The datasets in the Peer Review Workbench are transparently pre-processed to pre-filter and aggregate along dimensions required for each specific project. The attributes available in the PRW dataset before project-specific filtering are:
- Manuscript metadata: journal title, manuscript number, article type, submission date, status of submission, decision, decision date, keywords, abstract (only for accepted manuscripts upon further agreement), number of co-authors, inferred anonymized author gender*
- Review data: journal title, manuscript number, reviewer comment to author (only for accepted manuscripts), key dates such as reviewer invite-accept and completion dates, review status, decision recommendation and inferred anonymized reviewer gender*, geographical location
- Editorial decision data: decision date, anonymized editor gender *
- Reviewer data includes journal title, manuscript number article type, author position, board member, institution, city, state, country, keywords, classification
No author, reviewer, and editor identity will be available in the dataset. The Peer Review Workbench data does not include personal identifiers, nor does ICSR Lab host publication full text.
In order to ensure that the peer review data remains anonymous and can only be viewed at an aggregate level, each project will receive access to a pre-processed subset of this dataset with the fields required to answer their research question, as determined in conjunction with the Peer Review Workbench advisory board. Through pre-processing the subset we try to minimize the risk of individual person’s identification.
Please note that we do not encourage studies solely aiming at measuring journal publication speed or acceptance rates as these are often available on journal home pages (e.g. Current Applied Physics and Water Research).
We describe the dataset in a whitepaper that can be accessed on SSRN.
*Gender assignation metadata was derived using an AI-driven, inferred binary genderization methodology that is appropriate for bibliometrics or other large-scale analyses because such studies focus on trends at scale. The methodology cannot be used to unambiguously infer an individual’s gender, thus the gender metadata cannot be used for individual level or small group analyses as an alternative to self-reported data.
The advisory board members of the Peer Review Workbench are:
Professor Ana Marušic, Chair of the Department of Research in Biomedicine and Health University of Spit School of Medicine (editorial research in health) Professor Francisco Grimaldo, Vice Dean and Associate Professor of the School of Engineering University of Valencia (data science in peer review) Professor Francesca Dominici, Professor of Biostatistics, Population and Data Science, Harvard T.H. Chan School of Public Health (statistics) Dr. Mario Malički, META researcher at Stanford (ethics in peer review)
How to apply for Peer Review Workbench access
Interested groups should use the ICSR Lab application form to submit information about their project team and research plan, and will be prompted to enter extra information specifically related to their planned use of the PRW dataset. This extra information can be found in this Word document.
The procedure for evaluating these applications for our initial cohort of projects is as follows:
- Call for proposals opens on 8 September 2022 with the deadline being 10 January 2023.
- Proposal evaluation will take approximately 4 weeks and by March 2023 the applicants will receive the decision letter.
- PRW access will be provided to the accepted and finalized proposals as of 2 April 2023.
Recognizing that after the initial release of the dataset multiple groups will likely approach similar research questions, we will endeavor to connect groups to promote collaborative research by mutual consent.