Textbooks

There are no textbooks required for this course. Students are expected to read the lecture notes and attend class. However, the following textbooks/websites are recommended for reference:

• Koji Yatani, Statistical Methods for HCI Research
• Jonathan Lazar, Jinjuan Heidi Feng, Harry Hochheiser, Research Methods in Human-Computer Interaction
• Judy Robertson, Maurits Kaptein, Modern Statistical Methods for HCI

Grading

• Participation and Preparation
  • Teams
  • Ethic Training
• Experimental Design Exercises (ED)
  • ED1: Questionnaires and Surveys
  • ED2: Diaries and Interviews
• Experimental Design Reviews (EDR)
• Micro Paper Reviews (MR)
• Class Seminar
• Class Project
  • Project Proposal
  • Project Presentation
  • Final Paper and Video Figure

The format of the course is mainly a seminar style. It will contain lectures by the instructor, paper discussion seminars by students, and presentations of group projects. Students will be evaluated by the quality of their in-class participation, assignments, paper presentations, and class projects.

This course consists of a mixture of individual and group work. Students will work individually or in pairs to complete a term-long class project. In addition, students will work individually on a set of experimental design exercises, weekly paper readings and reviews, as well as a seminar of paper presentation and discussion. The philosophy of this seminar based course is learning by doing and learning together.

The grading components of the course include:

Students’ work is graded based on a 10-point scale with a 0.5-point increment, summed and rescaled based on the below details, where in general 0 = not participate, 2 = poor, 5 = fair, 8 = good, 10 = excellent.

Participation and Preparation

Attendance, team building, ethics training, and in-class activities such as group discussion, paper critiques, and feedback to other students’ work. Students are required to attend every class and actively participate in discussions. As a rule of thumb, you are expected to contribute to class discussion at least two times each class: ask a question, comment on a topic, clarify a point, etc. Your attendance and participation level will be recorded during the class.

Teams

While students can work individually or in pairs (i.e., a team) for the group work, a maximum of 14 teams can be formed in this term due to the capacity. At the beginning of the term, students will be given time to find a partner. Students who propose to work individually may not always be satisfied. The expectation for working individually or in pairs remains the same.

No grade is assigned for this sub-component. After finalizing a team, register your team (even though just yourself) using the Team Information form (link will be shared on MS Teams).

Ethic Training

The TCPS 2 Tutorial Course on Research Ethics (CORE), also known as the TCPS2 tutorial, is mandatory for all researchers who intend to engage in research with human participants. In this course, you will conduct user experiments. Before contacting users and conducting the studies, you are required to complete the ethics tutorial. Each student needs to complete the ethics tutorial individually. If you have taken this tutorial previously in other courses or activities, you do not need to take it again.

From the Welcome Page, click on the “Login to Core / Create Account” button on the right. Click “Create new account here” and fill in the required fields; make sure you register using your uwaterloo.ca email address. A confirmation email will be sent to the email address that you provide (check spam if you do not see it in your mailbox). Click on the activation link only once to activate your account. Once your account is activated, you can log in and begin the tutorial. It can take up to 3 hours take to complete CORE, depending on how many examples and activities you explore. You can go through the modules at your own pace; your progress is automatically saved, and you can log out and in again to resume your session. If you experience any difficulties, refer to TCPS 2: CORE Frequently Asked Questions.

No grade is assigned for this sub-component, but all students must complete this requirement (if you have not done this before); otherwise, you cannot continue the course. Submit the certificate of completion with your name on it to the corresponding LEARN Dropbox.

Experimental Design Exercises (ED)

Two (2) experimental design exercises of creating study protocols for concrete research problems. Using required HCI methods, students need to define suitable research questions and design study protocols to answer these questions. The research questions must NOT be overlapped with or closely related to their class project topics. An experimental design document (less than 1000 words; around 1-3 pages, excluding references) should include the following parts:

1. Background: A brief description of the relevant contextual information for understanding the domain and the problem that you are investigating.
2. Research Question(s): One to three clearly defined, concrete, and measurable questions and/or objectives that your study is focusing on.
3. Study Design: A detailed description of what kind of participants you will recruit, what apparatus/systems/tools you will use (if any), what the study procedure will look like, what types of data you will collect and measurements you will employ, and justify how the study protocol and materials could address the research questions you ask.
4. Study Materials: Designed study materials (see ED1 & 2 below) that you will use in the study, as attachments (i.e., does not count towards the page limit).

ED1: Questionnaires and Surveys

In ED1, you are required to design a survey study that consists of the following study materials: 1) pre-screening questions that help recruit suitable participants for your study, 2) demographic questions that help collect essential information from participants, and 3) at least 10 actual questions that aim to address the research problem. A recommended process is: for each of the research questions and/or study (sub-)objectives you define, design a couple of actual questions concerned to that specific research question and/or objective, which form a complete list of questions. Please refer to what you learn during the class to craft a high-quality study protocol.

ED2: Diaries and Interviews

In ED2, you are required to design a diary study that consists of the following study materials: 1) pre-screening and demographic questionnaires that help set up the study, 2) a simple questionnaire that participants need to fill in during the diary study, and 3) interview questions that help collect insights in the middle and/or the end of the study. Similar to ED1, you could design the questionnaire and interview questions based on specific research questions and/or (sub-)objectives of the study. Please refer to what you learn during the class to craft a high-quality study protocol.

Each experimental design exercise is graded using the following marking scheme:

The writing must use the ACM Latex or Word template with the \documentclass[acmsmall,screen]{acmart} option. The study materials can be in any format (e.g., Google Form, Qualtrics, or plain text), as long as they are clear.

Submit your study design description and materials in a single zip file, named [your last name]_[your student ID]_ED[n].zip, to the corresponding LEARN Dropbox by the posted deadlines.

Experimental Design Reviews (EDR)

Two (2) critical reviews of fellow students’ experimental design exercises. As a follow-up of the experimental design exercises, students need to review others’ submissions and provide critics/recommendations. Students will be randomly assigned a submission to review. You must give at least five concrete critics or actionable recommendations based on what you learn in the class regarding the HCI methods.

Each critical review is graded using the following marking scheme:

Submit your critics/recommendations in a single PDF file, named [your last name]_[your student ID]_EDR[n].pdf, to the corresponding LEARN Dropbox by the posted deadlines.

Micro Paper Reviews (MR)

Seven (7) weekly reading assignments of papers provided by the instructor. Students are required to submit micro reviews for the assigned papers every week (two per week). A micro paper review (around 500 words) must include the following four parts:

1. Summary: One or two sentences to summarize the paper’s key contributions.
2. Strengths: Two or three specific strengths of the paper, with rationales.
3. Weaknesses: Two or three specific weaknesses of the paper, with rationales.
4. Future work: One or two sentences to state any work that could be done to extend or build on this.

You should consider the following aspects when writing your micro reviews:

• Be specific and provide details.
• Do not simply paraphrase the paper content: your goal is to demonstrate your read the paper and thought about it.
• Do not comment on the quality of writing or paper presentation: your goal is to focus on the research described in the paper.
• Use grammatically correct English writing.

Each micro paper review is graded using the following marking scheme:

Submit your micro paper reviews as PDF files with a predefined template or a similar format, named [your last name]_[your student ID]_MR[n].pdf, under the appropriate directory in the shared folder on MS Teams by the posted deadlines. All micro reviews will be made public to the class prior to that week’s class to foster discussions; late submission of micro reviews will NOT be accepted.

Class Seminar

One (1) seminar of discussing a designated research topic from the list provided by the instructor. Each team must lead a seminar on a specific topic during one week of the class, including presenting a designated paper and moderating the discussion of the paper (see the schedule for details regarding the papers and topics). Students must fill in the Seminar Schedule form (link will be shared on MS Teams) at the beginning of the class, to select which topics/weeks they will present for the seminars. All the team members need to contribute to the paper presentation, although not every student needs to present. When working in pairs, please include a contribution slide in the end to indicate the task distribution among team members.

The paper presentation needs to clearly demonstrate the problem, approach, and key results of the paper. More importantly, you should use the paper content and examples as a way to introduce and address problems beyond the paper, related to the week’s topic and other papers. You could consider the following questions:

• What is the problem area? What is the relevant background? Why do we care?
• What is the idea and what are the main contributions?
• What are the key related works? What are the differences and similarities?
• What are the core technical approach and results?

In addition, you are responsible for answering questions regarding the paper and leading a discussion after the presentation. To facilitate the discussion, you will have access to all students’ micro reviews at least 24 hours before the class, so budget some time to scan through them, in order to motivate individuals in the class during the discussion to expand or justify their responses. You have to manage the class, by keeping people on topic, encouraging everyone to speak, and making sure the discussion is not dominated by a few people. You can be creative in the formats such as group discussion, debate, etc. You should carefully prepare the discussion points, for example:

• How well are the HCI methods used in the paper?
• What are the strengths and weaknesses of the paper?
• What are potential extensions and future work?
• What critical reflections could we have?

The paper presentation is graded using the following marking scheme:

Each paper presentation should last about 40 minutes, roughly 20 minutes for the presentation and 20 minutes for the discussion. Remember that you do not need to cover every nitty gritty details about the paper, because the whole class also read the paper beforehand; aim for a profound thinking rather than an on-surface description of the paper.

Upload your presentation slides for the class seminar, named W[week#]_[team name]_seminar.[pdf/pptx], under the appropriate directory in the shared folder on MS Teams by EOD on the day of the class during that week.

Class Project

One (1) term-long research project on designing, implementing, running, and analyzing a controlled experiment. As a central part of the project, each team needs to conduct a controlled experiment on evaluating a new interaction technique or assessing some aspects of existing interaction techniques that were not studied before. The project topic must be approved by the instructor. Define research hypotheses that bring contributions to the state of the art. You are free to use any language and toolkit for your project. The controlled experiment needs to involve at least 10 participants. You can conduct the experiment with either students in the class or users out of the class; regardless, you need to follow the ethics and consent process using the materials on the Resources page.

The class project is split into different components as the following:

The writing must use the ACM Latex or Word template with the \documentclass[acmsmall,screen]{acmart} option. When working in pairs, please include an Acknowledgement section that briefly describes what each team member contributes to the overall assignment when appropriate (e.g., roles and task distributions; not necessarily everyone has to contribute to the writing).

Submit your writing, slides, or video, named [team name]_[proposal/presentation/final_paper/video].[pdf/pptx/mp4], to the corresponding LEARN Dropbox by the posted deadlines.

Project Proposal

Each team needs to write a brief proposal (less than 1000 words; around 2-3 pages, excluding references) that outlines the project idea. You should start thinking about your project early. You are encouraged to talk to the instructor before finalizing the topic. The proposal should contain the following parts:

• Title: The title should be catchy, short, and descriptive.
• Abstract: The abstract should be less than 150 words and summarize your paper, including contributions, approach, and results. You could use the following template:
  • 1) context and background (1 sentence),
  • 2) challenges and problems to solve (1 sentence),
  • 3) your proposed approach and main contributions (1-3 sentences), and
  • 4) expected results (1-2 sentences).
• Introduction: The goal of the introduction is to provide the context, deliver the motivation, summarize the paper, basically, why do you do it, and how? You could use this template:
  • 1) provide the background or context of the research and present the unaddressed research problems or questions (1 paragraph),
  • 2) summarize a few key relevant work and why they do not work for your problem (1 paragraph),
  • 3) describe your approach and how it addresses the problem (1-2 paragraphs), and
  • 4) list 1-3 key contributions of the paper (1 paragraph).
• Approach: This section describes the high level technical approach and/or design, languages/libraries/toolkits you plan to use, and how you will address the research questions or problems using controlled experiments (e.g., your initial hypotheses, experimental protocol, and data measurements). Try to be as detailed as possible and include sketches or figures if it makes explaining your approach easier.

The marking scheme is:

Project Presentation

Each team is required to present the work in class, answer questions, and get feedback from other students, in a style of academic conferences/workshops. Each presentation session is limited to a 10-minute talk and a 5-minute Q & A, in total, 15 minutes maximum. The presentation should include the motivation, research problem/questions, key related works, technical approach, and initial results of the project. When working in pairs, please include a contribution slide in the end to indicate the task distribution among team members.

The marking scheme is:

Final Paper and Video Figure

Each team needs to write an academic paper (less than 5000 words; around 8 to 12 pages, excluding references) that describes the entire class project in detail. The ideal final paper should be in a quality that is ready to submit to a conference workshop. You need to refine the project proposal and extend it by summarizing the work throughout the term. Besides a revised Introduction, you need to have the following parts in your final paper:

• Related Work: The literature review should include scholarly articles, industry and trade articles (as appropriate), books and magazines, and a review of existing products (where applicable). As a general rule of thumb, you should survey at least 15 related references. You need to summarize each related work concisely and emphasize the differences or novelty of your idea. You want your literature survey to:
  • 1) Be comprehensive in breadth and depth and related theories have been investigated;
  • 2) Point out aspects of the problem space that have not been solved or examined in detail; and
  • 3) Distinguish your work from the state of the art and emphasize the novelty.
• Technical Approach: You will extend the Approach of your proposal with more details. The proposed new technique or aspects of an existing technique should be described in a human-computer interaction perspective, e.g., what people see when they use it, what it can be used for, why it is interesting or useful, etc. You should also include sufficient details that someone could reproduce it themselves. You are encouraged to provide diagrams and figures to demonstrate your technical approach.
• Experiment and Results: This describes the participants, apparatus, design, tasks, and procedure in your controlled experiment. You should provide enough details so that someone else can replicate your experiment. Also, you need to analyze the data using appropriate statistical analysis and present the results using charts.
• Discussion: This reflects on what was done, implications for designers and researchers, some issues or problems encountered in the evaluation, and limitations of the approach. This section is your chance to reflect on any lessons learned.
• Future Work: This presents advice and directions for others doing work similar to your project, suggesting how things could be improved or better.

In addition, each team need to submit a video figure (less than 4 minutes) to demonstrate your technique and/or experiment tasks. Ensure your video is encoded with MPEG-4 using H.264 (check Handbrake for encoding/re-encoding your video). The video could be made public on YouTube or the class website.

The marking scheme is:

---