Ivan
Stelmakh,
PhD
candidate
Machine
Learning
Department,
School
of
Computer
Science
Carnegie
Mellon
University
Peer review is the backbone of scholarly research and fairness of this process is crucial for the successful development of academia. In this talk, we will discuss our two recent works on fairness of peer review. In the first part of the talk, we will focus on the automated assignment of papers to reviewers in the conference setup. We will show that the assignment procedure currently employed by NeurIPS and ICML does not guarantee fairness and may discriminate against some submissions. In contrast, we will present the assignment algorithm that simultaneously ensures fairness and accuracy of the resulting allocation.
In the second part of the talk, we will continue the long-standing debate on single- vs. double-blind peer review. We will discuss the methodology of prior works on testing for biases in single-blind peer review and show that strict assumptions made therein put their tests at risk of being unreliable. We will then present our novel test that accounts for various idiosyncrasies of peer review and provably controls for the Type-I error while having non-trivial power.
This talk is based on joint work with Nihar Shah and Aarti Singh. No specific background is required to enjoy the talk.
His research interests lie at the intersection of statistics and the field of learning from people with a current focus on studying and improving the process of scholarly peer review.
The goal of his research is to design tools and methods that help to assess and improve the fairness, reproducibility, and accuracy of peer review. Before coming to CMU, he received a B.S. in Physics from the Moscow Institute of Physics and Technology.