Computer Science Thesis Proposal

Friday, April 8, 2022 - 9:00am to 10:30am

Location:

In Person and Virtual - ET Gates Hillman 8102 and Zoom

Speaker:

STEVEN JECMEN, Ph.D. StudentComputer Science DepartmentCarnegie Mellon University https://sjecmen.github.io/

Making Peer Review Robust to Undesirable Behavior

Scientific peer review is a critical part of the academic publication process, used across disciplines and venues in various forms. Peer review generally relies on the good-faith participation of many reviewers and authors. However, the peer review process must also deal with different kinds of undesirable behavior from participants, including both malicious attempts to cheat the system and non-malicious cases of unreliability. With this motivation, the goal of this thesis is to design algorithms to make peer review more robust to undesirable behavior. In this thesis proposal, I will first describe several practical methods that we have proposed in past work for handling different forms of undesirable behavior in peer review. •    Reviewers may collude with authors, manipulating the paper assignment in order to get assigned to their friend’s paper so that they can give it a positive review. We provide efficient algorithms for finding high-quality randomized assignments that limit the probability that a colluding reviewer-author pair succeeds. Our main algorithm has been used at several venues and is implemented at openreview.net.

  • Reviewers may give low scores to their assigned papers in the hopes of increasing their own paper’s chances of acceptance. We provide efficient algorithms for finding maximum-quality assignments that are strategyproof to this form of strategic reviewing.
  • Reviewers may turn out to be unresponsive or provide low-quality reviews. A two-phase review process allows replacement reviewers to be assigned for any unresponsive first-phase reviewers, but it’s not clear which reviewers should be saved for the second phase. We show that randomly saving reviewers for the second phase gives near-optimal assignments both on conference data and under natural theoretical conditions.

I will then outline several directions for future work that can further improve the robustness of peer review. Thesis Committee: Nihar Shah (Co-chair) Fei Fang (Co-chair) Christos Faloutsos Yiling Chen (Harvard University) Ashish Goel (Stanford University) In Person and Zoom Participation. See announcement.

For More Information, Contact:

Keywords:

Thesis Proposal