Reviewer Guidance on Rigor and Transparency

Last Updated March 18, 2019 1 | Page

NIH Peer Review

Reviewer Guidance on Rigor and Transparency:

Research Project Grant and Mentored Career

Development Applications

The goal of this initiative is to enhance reproducibility of research through rigor and

transparency. Several years ago, NIH updated application instructions and review language for

research grant (NOT-OD-16-011) and mentored career development award (NOT-OD-16-012)

applications. Recently, NIH further clarified the language to replace the term “scientific

premise” with the term "rigor of the prior research" for applications submitted for due dates of

January 25, 2019 and beyond (NOT-OD-18-228 and NOT-OD-18-229). In addition, applicants will

describe plans to address any weaknesses in the rigor of prior research within the Research

Strategy and reviewers will assess this plan. Implementation of rigor and transparency for

individual fellowship, institutional career development, and institutional training grant

applications will be announced in advance, on a different timeline that allows for training in

rigor and transparency to be developed (NOT-OD-16-034).

The four areas of the current rigor and transparency initiative are explained below.

• The rigor of the prior research concerns the quality and strength of the research being

cited by the applicant as crucial to support the application; this is distinct from the

hypothesis or justification.

o The applicant should discuss the strengths and weaknesses of the prior research

used to support the application and describe how the proposed research will

address weaknesses or gaps identified by the applicant. This may include the

applicant’s own preliminary data, data published by the applicant, or data published

by others. The NIH expects this consideration to include attention to the rigor of the

previous experimental designs, as well as relevant biological variables and

authentication of key resources.

o Reviewers will evaluate the rigor of the prior research as part of the Significance and

Approach criterions for research grant applications or the Research Plan criterion for

mentored career development award applications.

 Consider whether the prior research that serves as the key support for the

proposed project is rigorous.

 Consider whether the investigators included plans to address weaknesses or

gaps identified in the rigor of prior research.

Last Updated March 18, 2019 2 | P a g e

 Weaknesses or gaps in the rigor of the prior research that serves as the key

support for the proposed project, or the failure to address those weakness or

gaps, may affect criterion and overall impact scores.

• Scientific Rigor is the strict application of the scientific method to ensure robust and

unbiased experimental design, methodology, analysis, interpretation and reporting of

results. Whereas rigor of the prior research pertains to key supporting data, scientific

rigor pertains to the proposed research.

o The applicant should describe experimental controls, plans to reduce bias (blinding,

randomization, inclusion and exclusion criteria, etc.), power analyses, and statistical

methods, as appropriate.

o Reviewers will assess scientific rigor as part of the Approach criterion for research

grant applications and the Research Plan criterion for mentored career development

award applications, as well as the overall impact score.

 The Vertebrate Animal Section no longer requires a justification of animal

numbers (NOT-OD-16-006). Inadequate vertebrate animal numbers should

be reflected in the score and will not result in a block to funding.

 Reviewers will assess information concerning numbers of animals according

to the section where it is included in the application.

Rigor of the prior data

Scientific Rigor

Pertains to:

Key supporting data

Proposed research

Review Criterion – Research Grants

Significance and Approach

Approach

Review Criterion – Mentored

Career Development Grants

Research Plan Research Plan

• Consideration of Sex and Other Biological Variables includes the critical factors

affecting health or disease in vertebrate animals or human subjects. Biological variables,

such as sex, age, weight, and underlying health conditions, are often critical factors

affecting health or disease.

o Applicants are expected to factor Sex as a Biological Variable (SABV) into research

designs, analyses, and reporting in vertebrate animal and human studies.

 Consideration of SABV does not necessarily mean sex differences research.

See Figure 1 in “Studying both sexes: A guiding principle for biomedicine” for

further detail.

 A justification is expected if the application proposes to study one sex, for

example in the case of a sex-specific condition or phenomenon (e.g., ovarian

or prostate cancer), acutely scare resources, or sex-specific hypotheses when

there are known differences between males and females.

 Cost and absence of known sex differences are inadequate justifications for

not studying both sexes.

Last Updated March 18, 2019 3 | P a g e

o Reviewers will assess the applicant's plans to address relevant biological variables,

such as sex, as part of the Approach (or Research Plan) criterion score and the

overall impact score, and comment on the adequacy of those plans in their written

critiques and in meeting discussions.

 Reviewers will assess information according to the section where it is

included in the application.

 See additional reviewer guidance for evaluating sex as a biological variable:

https://grants.nih.gov/grants/peer/guidelines_general/SABV_Decision_Tree_

for_Reviewers.pdf.

• Authentication of Key Biological and/or Chemical Resources. Key biological and/or

chemical resources are those that 1) may differ from laboratory to laboratory or over

time; 2) may have qualities and/or qualifications that could influence the research data;

and 3) are integral to the proposed research. These include, but are not limited to, cell

lines, specialty chemicals, antibodies, and other biologics, not standard laboratory

reagents. Please see NOT-OD-17-068 for more details.

o Applicants should provide a brief plan (one page or less).

 The plan should not include authentication data or any other data.

 The plan may reflect existing guidelines or standards for authentication of a

resource when such standards exist.

o Reviewers will discuss the authentication plan after scoring; comments on key

resource authentication should not affect scores.

 Reviewers will comment in their written critiques and discussion at the meeting

on the adequacy of the plan for key resource authentication; comments can be

addressed by the applicant prior to award for meritorious applications.

 Reviewers should note if the authentication plan is missing from the application.

Not all activity codes are included in the rigor and transparency initiative. Therefore, reviewers

need to follow the correct review criteria and use the appropriate and current critique template

for each application. Your Scientific Review Officer (SRO) should provide or direct you to the

appropriate templates and guidance.

Page limits have not changed. SROs and reviewers need to be alert for over-stuffed

applications.

You may submit your comments/questions about the NIH policy to reproducibility@nih.gov.

OVERVIEW: RESEARCH PROJECT GRANT (RPG) APPLICATIONS

Element of Rigor and

Transparency

Section of

Application

Criterion

Score

Additional

Review

Consideration

Contribute to

Overall Impact

Score?

Last Updated March 18, 2019 4 | P a g e

Rigor of the Prior

Research

Strategy

Significance

and

Approach

Yes

Scientific Rigor

Research

Strategy

Approach

Yes

Consideration of

Relevant Biological

Variables, such as Sex

Research

Strategy

Approach

Yes

Authentication of Key

Biological and/or

Chemical Resources

New

Attachment

Yes

OVERVIEW: MENTORED CAREER DEVELOPMENT AWARD (K)

APPLICATIONS

Element of Rigor and

Transparency

Section of

Application

Criterion

Score

Additional

Review

Consideration

Contribute to

Overall Impact

Score?

Rigor of the Prior

Research

Strategy

Research

Plan

Yes

Scientific Rigor

Research

Strategy

Research

Plan

Yes

Consideration of

Relevant Biological

Variables, such as Sex

Research

Strategy

Research

Plan

Yes

Authentication of Key

Biological and/or

Chemical Resources

New

Attachment

Yes

Additional Resources

• NIH Extramural website on Rigor and Reproducibility

• Nature Perspectives: "A call for transparent reporting to optimize the predictive value of

preclinical research" Landis, et al., 10/10/2012

• Nature Commentary: "Policy: NIH plans to enhance reproducibility" Collins & Tabak,

01/27/2014

Last Updated March 18, 2019 5 | P a g e

• Nature Commentary: "Policy: NIH to balance sex in cell and animal studies" Clayton &

Collins, 05/14/2014

• Science Editorial: "Journals Unite for Reproducibility" 11/07/2014

• Science Perspectives: "Fixing problems with cell lines" Lorsch, Collins & Lippincott-

Schwartz, 12/19/2014

• FASEB Journal Life Sciences Forums: "Studying both sexes: a guiding principle for

biomedicine" Clayton 10/29/2015

• Narrated overview of the NIH policy and why it’s important:

https://grants.nih.gov/reproducibility/module_1/presentation.html

• The FASEB Journal: "Studying both sexes: a guiding principle for biomedicine" Clayton

2/30/2016

• The FASEB Journal: “Considering sex as a biological variable in preclinical research”

Miller 9/28/2016