Check Session Proposal Review Scores for Conflicts, Thin Coverage and Lopsided Reviewers Before the Committee Meets

Scores from a call for session proposals reach a committee as one ranked list, and the list hides which scores should not count. This tutorial turns three exported files into a committee list with conflicted scores set aside, thin and split proposals marked and unusual reviewers noted, so a named person can follow up each flag before the meeting.

Try it now: the session review-score check reads three CSV files in your browser and lists what a person should look at. Nothing is uploaded or stored.

Where the limits come from

ASHE’s review page describes three reviewers with diverse expertise for each proposal, says their scores will sometimes vary, and lets a reviewer return a proposal over a conflict of interest. It tells chairs to strongly consider scores of 4.5 and higher on its own scale, says there is no cut-off score, and has staff review outliers such as proposals above 4.5 that were rejected or below 3.5 that were accepted. ASAE’s page says proposer names are omitted from what its peer reviewers see and that its committees read aggregate ratings and comments.

NIH guidance says reviewers who give high ratings to every application weaken their ability to convey its impact, and records a conflicted reviewer’s score under a conflict code. A Communications of the ACM review by a Carnegie Mellon researcher says some reviewers are lenient and others strict. A PNAS study by Wisconsin and West Chester researchers replicated NIH review with 43 reviewers and 25 grant applications and found no agreement among reviewers on the same application, in a sample of only high-quality applications.

Only the band edges and the three-review minimum echo ASHE, and ASHE says it has no cut-off. The matching rules, the range that marks a split, the distance that marks a reviewer, the flat-reviewer rule and the free-mail list are house choices with no cited source. None of these pages tests the checks.

Check one call in twelve steps

Export the proposals, reviewers and scores as three CSV files. Wait until scoring has closed.
Match the first row of each file to the sample files’ column names. Other columns are ignored.

Diagram: three files for proposals, reviewers and scores go into checks with your limits, and a committee list of ids and flags comes out.

Open the proposals file in the lab. Its first-row column names then show under the three files.
Open the reviewers file in the lab.
Open the scores file in the lab.
Set each number under “Limits you choose”. The starting values are placeholders.
Press Check scores.

Diagram: each score row is tested for a shared organization, email domain or copresenter listing, and a match sets the score aside and lists it while no match counts it as eligible.

Compare each score under “Scores set aside” with the reviewer’s disclosure.
Read each reviewer with a note under “Reviewers” beside the proposals that reviewer scored.

Ask for another review on each proposal with too few eligible reviews.
Read the comments on each proposal marked split.
Press Download committee-list.csv.

Write the person who follows up each flagged proposal in the empty decision column before the committee meets.

Thirty invented proposals, read once

The sample files are made up, so this teaching example shows what the checks do and not how a real call behaves. With the starting limits, 91 score rows give 87 eligible scores, 4 set aside, 4 proposals below three eligible reviews, 9 split proposals and 7 proposals tied at 4.00 for rank 5. P20 leads at 4.67 with no flag, yet reviewer V03, marked high, gave one of its three scores. P19 and V05 share a free-mail domain, so that score stays.

Check the result

Compare “Score rows read” under “Counts” with the rows in your scores file, then add “Scores set aside for conflicts” to “Eligible scores” and confirm the total. Raise “Score range that marks a split” above the width of the scale and confirm that “Proposals with a split” falls to 0. Ask a second person to recompute one proposal’s mean from the original file.

Common mistakes when reading the list

A match on organization, email domain or copresenter is a floor: the Carnegie Mellon author notes that colluders can avoid co-authoring to stay off conflict lists. A reviewer note shows distance from the pool, and NIH tells reviewers not to assume their assigned set covers the whole range, so a high reviewer may have drawn strong proposals. A band is not a decision, since ASHE says highly scored proposals may be declined for space.

Clean organization names first with the duplicate-records tutorial if they arrive spelled several ways. The flat-reviewer note applies the idea behind the survey-screen tutorial to scores.

What runs in the browser and what needs the script

The lab runs every check in your browser on three files at a time, within 1 MB each, 1,000 proposals, 500 reviewers and 20,000 score rows. Only the kit’s command-line script can check several folders of files in one run. The kit adds the sample files, a settings worksheet, a pre-meeting checklist and a decision log, free once you submit your email in the lab page’s form. The committee list holds ids, ranks and flags and copies no name, title or email column, so use ids that are not email addresses.