Steven Jecmen

Making Peer Review Robust to Undesirable Behavior Degree Type: Ph.D. in Computer Science
Advisor(s): Nihar B. Shah, Fei Fang
Graduated: February 2024

Abstract:

Scientific peer review is a critical part of the academic publication process, used across disciplines and venues in various forms. Peer review generally relies on the good-faith participation of many reviewers and authors. However, the peer review process must also deal with different kinds of undesirable behavior from participants, including both malicious attempts to cheat the system and non-malicious cases of unreliability. In this thesis, I describe several practical methods that we have proposed for handling different forms of undesirable behavior in peer review.

First, we consider the problem of reviewer-author collusion, in which malicious reviewers manipulate the paper assignment in order to get assigned to each others' papers so that they can give them positive reviews. We provide efficient algorithms for finding high-quality randomized assignments that limit the probability that a colluding reviewer-author pair succeeds at manipulating the paper assignment. These randomized assignments also mitigate attempts by malicious reviewers to "torpedo" a disliked paper and attempts by malicious authors to de-anonymize their reviewers.

Second, we provide an in-depth analysis of the cost of deploying a randomized assignment in terms of the resulting review quality. We propose methods that leverage the randomness introduced by these randomized assignments in order to evaluate alternative paper assignment policies, and apply these methods to estimate the quality of various potential changes to the assignment policy.

Third, we address the issue of unresponsive reviewers. We provide a simple procedure for finding high-quality paper assignments in a two-phase review process, which allows replacement reviewers to be assigned for any missing or low-effort reviews in the first-phase.

Fourth, we tackle the problem of strategic reviewing, in which reviewers give low scores to their assigned papers in the hopes of increasing their own paper's chances of acceptance. We provide algorithms for finding high-quality assignments that are strategyproof to this form of strategic reviewing.

Finally, we analyze other approaches to addressing the manipulation of paper assignments, which we categorize into mitigation-based and detection-based approaches. We compare the tradeoffs between various proposed approaches to mitigating the impact of manipulated bidding. We also empirically analyze the problem of explicitly detecting reviewer-author collusion rings from the manipulated paper bidding, and furthermore release a dataset on this kind of bidding.

Thesis Committee:
Nihar B. Shah (Co-chair)
Fei Fang (Co-chair)
Christos Faloutsos
Yiling Chen (Harvard University)
Ashish Goel (Stanford University)

Srinivasan Seshan, Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Keywords:
Peer evaluation, peer review, strategic behavior, paper assignment, matching, randomization, strategyproofness