CS885 Winter 2025 - Reinforcement Learning

Paper Critiques

• If you present a paper: write one critique of a different paper than the one you will present (10% of final mark)
• If you do not present a paper: write three critiques (10% of final mark for each critique)
• To be done individually (i.e., no team)
• Due date: at 11:59 pm the day before the paper is scheduled for presentation and discussion (see schedule)
• Submission: via LEARN

Format

• At most 2 pages
• If you don't understand something, explain what is unclear about the paper and how clarity could be improved

Suggested questions to structure your critique

Overview

In your opinion:

• What is the paper about?
• What are the contributions?
• What are the claims of the paper?
• What is the take home message?

Clarity

In your opinion:

• Is the paper well written?
• Is there some missing background needed to understand the paper?
• Are there confusing concepts that should be explained in more details?
• Is the notation defined and consistent throughout the paper?
• What aspects of the paper would you explain differently to improve clarity?

Significance and originality

In your opinion:

• Are the ideas novel/original?
• Are the contributions significant?
• Does the paper simply combine existing ideas?
• Is the paper simply applying an exiting algorithm to a new problem?
• Does the paper contribute a new technique that solves a problem for the first time?
• How does the paper advance the state of the art?

Evidence to support the claims of the paper

In your opinion:

• What theoretical evidence does the paper provide to support its claims?
• What empirical evidence does the paper provide to support its claims?
• Is the evidence provided sufficient to support the claims

Theoretical Analysis

In your opinion:

• Are the proposed ideas technically sound?
• Is there a theoretical analysis for scalability, convergence, optimality, etc.?
• Does the theoretical analysis make some assumptions? Are those assumptions realistic and verifiable?

Empirical Evaluation

In your opinion:

• Are there enough testbeds and are they representative of realistic applications?
• Are the empirical improvements statistically significant?
• Are there missing metrics that should have been reported?
• Is there an ablation study? If yes, does it confirm that all components of the proposed approach are useful?

Limitations

In your opinion:

• What are the limitations of the paper?
• Are there assumptions that retrict the application of the work?
• Does scalability restrict the application of the work?