CS885 Fall 2022 - Reinforcement Learning

Overview

• 40% of final grade
• To be done individually
• To search for papers on a topic, use Google scholar. Most papers available on the web can be found via Google scholar. Furthermore, most links in Google scholar are free. As a student you should never have to pay to download a paper. If Google scholar does not provide a free link, you should be able to access the paper freely via the University's internet or the University's library website.

Options

• Option A (Literature survey):
  • Pick a problem that interests you
  • Search the literature for reinforcement learning approaches to tackle this problem
  • Survey and discuss the relative strengths of each approach
  • If you'd like to see examples of survey papers in AI, have a look at the IJCAI-2021 survey track
• Option B (Empirical evaluation):
  • Pick a problem that interests you.
  • Implement and experiment with several reinforcement learning techniques to tackle this problem.
• Option C (Algorithm design):
  • Identify a problem for which there are no satisfying approaches.
  • Develop a new reinforcement learning technique to tackle this problem.
  • Analyze theoretically and/or empirically the performance of your technique.
• Option D (Dataset/Simulator/Benchmark design):
  • Identify a problem for which there is a lack of datasets, simulators or benchmarks to evaluate reinforcement learning algorithm.
  • Collect a dataset, design a new simulator or design a new benchmark to evaluate reinforcement learning algorithms.
  • Demonstrate how some baseline reinforcement learning algorithms perform with your dataset, simulator or environment.
  • If you'd like to see examples of papers describing datasets, simulators or benchmarks, have a look at the NeurIPS-2021 datasets and benchmarks track
• Option E (Theoretical analysis):
  • Identify a problem or reinforcement learning technique for which the properties (e.g., complexity, performance) are not well understood.
  • Analyze the properties of this problem or technique.

Proposal (no mark)

• Submit electronically via LEARN by October 26 (11:59 pm)
• At most one page (excluding references)
• Use the JMLR format: https://www.jmlr.org/format/format.html
• Which option did you pick?
• Option A (Literature survey):
• What is the problem?
• Cite 8 to 12 papers that you plan to survey.
• Option B (Empirical evaluation):
  • What is the problem?
  • What reinforcement learning techniques do you plan to experiment with?
  • Cite 4 to 8 related papers that you plan to review.
• Option C (Algorithm design):
  • What is the problem?
  • Why are there no satisfying approaches?
  • What is the intuition behind the new reinforcement learning technique that you plan to develop?
  • Cite 4 to 8 related papers that you plan to review.
• Option D (Dataset/Simulator/Benchmark design):
  • What is the reinforcement learning problem for which there is a lack of datasets, simulators or benchmarks?
  • What dataset, simulator or benchmark do you plan to design?
  • Cite 4 to 8 related papers that you plan to review.
• Option E (Theoretical analysis):
  • What is the reinforcement learning problem or technique that you plan to analyze?
  • What properties would you like to analyze/prove about this problem or technique?
  • Cite 4 to 8 related papers that you plan to review.

Report (40% of final grade)

• At most 8 pages (excluding references)
• Use the JMLR format: https://www.jmlr.org/format/format.html
• Explain the big picture and any necessary detail
• Submit electronically via LEARN by December 12 (11:59 pm)

Suggested Structure for the Report

• Option A (Literature survey):
  • Introduction
    • What is the problem?
    • Why is it an important problem?
  • Survey
    • Summarize the range of techniques by highlighting their strengths and weaknesses (i.e., the 8-12 papers that you read)
    • Tip: this summary should not be a laundry list of techniques with an independent paragraph for each technique
    • Suggestion: organize your summary based on desirable properties of the techniques
  • Analysis:
    • What is the state of the art?
    • Any open problem?
  • Conclusion
    • What have you learned?
    • What future research do you recommend?
• Option B (Empirical evaluation):
  • Introduction
    • What is the problem?
    • Why is it an important problem?
  • Techniques to tackle the problem
    • Brief review of previous work concerning this problem (i.e., the 4-8 papers that you read)
    • Brief description of the techniques chosen and why
  • Empirical evaluation
    • Compare empirically the techniques for complexity, performance, ease of use, etc.
  • Conclusion:
    • What is the best technique?
    • Is any technique good enough to declare the problem solved?
    • What future research do you recommend?
• Option C (Algorithm design):
  • Introduction
    • What is the problem?
    • Why can't any of the existing techniques effectively tackle this problem?
    • What is the intuition behind the technique that you developed?
  • Techniques to tackle the problem
    • Brief review of previous work concerning this problem (i.e., the 4-8 papers that you read)
    • Describe the technique that you developed
    • Brief description of the existing techniques that you will compare to
  • Evaluation
    • Analyze and compare (empirically or theoretically) your new approach to existing approaches
  • Conclusion:
    • Can your new technique effectively tackle the problem?
    • What future research do you recommend?
• Option D (Dataset/Simulator/Benchmark):
  • Introduction
    • What is the problem?
    • Why aren't existing datasets, simulators or benchmarks sufficient to evaluate RL techniques for this problem?
  • Proposed Dataset, Simulator or Benchmark
    • Describe the proposed dataset, simulator or benchmnark
    • Describe the properties of the dataset, simulator or benchmark that are unique
  • Evaluation
    • Brief description of some baseline RL techniques that you plan to evaluate
    • Compare empirically the baseline RL techniques with your dataset, simulator or benchmark
  • Conclusion:
    • What are the most important weaknesses of existing baselines that your dataset, simulator or benchmark highlighted
    • What future research do you recommend?
• Option E (Theoretical analysis):
  • Introduction
    • What is the problem or technique?
    • What properties did you analyze/prove about this problem or technique?
  • Analysis
    • Brief survey of previous work concerning this problem (i.e., the 4-8 papers that you read)
    • Describe the analysis performed
  • Conclusion:
    • What have you discovered about the technique analyzed?
    • What future research do you recommend?