Experimental methods
for Human-Computer Interaction

This special topics graduate course examines experimental methods used in human-computer interaction (HCI) research. The most frequently used include observations, field studies, surveys, usability studies, interviews, focus groups, and controlled experiments. We will cover both the design an statistical analysis of such experiments. Weekly student-led research seminars and statistical analysis assignments using RStudio will prepare students with the theoretical and practical background for a final project.

The primary learning outcomes are:

• familiarity with HCI research experimental methods
• knowledge of related statistical methods
• experience writing statistical analysis code using RStudio

Course information below (marking scheme and graded components), see also:

• Course schedule page (readings, seminars, exercises, and project deadlines).
• Learn Site (submission dropboxes, grading).

Graded Components:

20%	Participation
10%	Micro-reviews
15%	Exercises
15%	Seminar Presentation
40%	Project

You should also read the course policies.

Participation is important: students are expected to attend all classes, read the papers, and contribute to class discussions. As a rule of thumb, you are expected to contribute to class discussion at least 1 or 2 times each class: ask a question, comment on a topic, clarify a point, etc.

Attendance alone is not enough for your participation mark. You must participate.

Worth 20% of grade: mark based on level of participation in class discussions.

One day before each class with seminar papers, students will submit a compact three-part review of each paper being presented. The micro-review must answer three specific questions:

1. What is a specific strength? (e.g. Is something very original and innovative? Is some aspect an elegant solution? Does it solve a very important problem?)
2. What is a specific weakness? (e.g. Is there an incorrect assumption? Is there a methodological problem? Is there a better solution to some part?)
3. If you did research to extend or build on this, what would you do? (e.g. This could be an idea for a course project.)

Each question must be answered in less than 400 characters. Some hints when writing your answers are below and in an expanded list:

• Be specific and provide details.
• Do not simply paraphrase the abstract: your goal is to demonstrate your read the paper and thought about it.
• Do not focus on commercial aspects like cost, size, or "fit and finish": research is not the same as a KickStarter campaign.
• Do not comment on the quality of writing or paper presentation: your goal is to focus on the research described in the paper.
• Use grammatically correct English writing: usually one or two sentences per question: this is not a tweet or text, do not use unusual abbreviations to fit the character limit.

You can optionally rate the paper on a scale of 1 to 10.

All micro-reviews will be made public to the class after the submission deadline (with the first name of each student).

Seminar papers are marked with an 'id' consisting of the first author's name and year (e.g. "smith95", beginning in week 2).

You do not need to do a micro-review for the paper you are presenting on, for any paper not marked with an 'id', or for any related papers.

Submit micro-reviews in the form linked from each paper id in the schedule before 10:00 AM on Thursday (24 hours before class).

There are small programming exercises each week to provide practical experience with statistical analysis. Each exercise will have one or more objectives that you must write code to solve. In addition to submitting final code, you will also submit short notes (less than 500 words) explaining what you did, what was hard, and the main resource(s) you used (blog posts, online tutorials, stackoverflow posts, papers, textbooks, etc.).

We will be using RStudio.

All exercises are done individually. Please review university policies on plagiarism.

Submit your work according to the instructions provided for each exercise in LEARN dropbox before the start of the next class.

Every student is required to lead a paper seminar. Each seminar consists in presenting a critique of the paper and lead a discussion.

Instructor Advance Review

You must submit draft slides of your presentation before 4:00 PM Wednesday for the instructor to review. Comments will be returned before 10:00 AM on Thursday.

Paper Presentation and Discussion
(20 minutes)

Use the paper content and examples as a way to introduce and address questions that go beyond the paper itself:

• Establish a context for the paper. Who are the authors (briefly)? What's the problem area? What is the relevant background? Why do we care? Why is the problem interesting, and why is it deep?
• What's the big idea and contribution of the paper? Why do you think the paper was accepted?
• How effective are the paper's results? How are they validated? What are the strengths and weaknesses?
• What are other related papers?
• What are potential extensions, future work?
• If possible, talk about how this paper might apply to your project.

As a rough guide, keep specific summary of information in the paper to less than 25% of your talk and the rest for answering the questions above. Spread the specific content of the paper around and use it to ground the discussions of the more important seminar questions. Remember, everyone in the class will have read the paper ahead of time to create their micro-reviews, so everyone already has a good grasp of what the paper is about. Whatever you do, do not simply give a summary of the paper contents (the grading rubric below makes it clear you will not do very well if you do).

If you find that your slides cover all sections of the paper in the same order in which they appear and you only show figures and tables from the paper, then you are creating a summary presentation. Don't do that.

Class Discussion

You are also responsible for leading discussion during (or after) your presentation. Be prepared to get the class started by seeding the discussion with open-ended questions and some controversial statements. You will have access to the class micro-reviews 24 hours before your seminar, so budget some time to read through them, summarize the responses on your slides. You are encouraged to call on individuals in the class during the discussion to expand or justify their responses (you will have their names associated with the micro-reviews). You will have to manage the class: this means keeping people on topic, encouraging everyone to speak, and making sure the discussion isn't dominated by a few people.

Submit your draft presentation slides to the corresponding LEARN dropbox before 4 PM on the Wednesday before your seminar.

Submit your final presentation and tutorial materials to the corresponding LEARN dropbox on the day of your seminar class. See the Learn dropbox for what exactly to submit.

You will design, implement, run and analyze a controlled experiment. The experiment can evaluate a new interaction technique, for example. It can also evaluate some aspects of existing interaction techniques that were not evaluated before. The experiment is the central part of your project. Define research hypothesis that bring a contribution on the state of the art. A requirement is that other students in the class run the experiment. Other types of experiments are also possible as long as that they can be run with other students in the class.

You are free to use any language and toolkit for your final project. However keep in mind the ability of running the experiment with other students in the class.

Projects can be done individually or as a pair. You will need to choose a problem that fits your skill set and that can be completed by the end of the term. There are four stages to the project: a proposal, a literature review, a design review with the instructor, and final deliverables including a paper, video, and presentation to the class.

The project grade is broken down as follows:

15%	Proposal
15%	Literature Review
70%	Final Deliverables (demo, paper, video)

You should start thinking about your project early in the course. There are two stages to the proposal to help you do this.

Initial Ideas

Come up with at least three possible project ideas. Submit these three ideas to the instructor by email for feedback. Each has very brief description helping to answer the following questions: What is the problem? What is the context? What kind of experiment do you want to run? What is its purpose? (what are your hypotheses?) What contributions would you expect compared to existing literature? (maybe early to answer this but if you have an idea) How would you implement your experiment?

Proposal

Write a 2 page proposal using the SIGCHI Conference Publications Format. You can write your proposal in present or past tense, as though you've already done the work (this is called “future writing” and it makes it easier to reuse what you wrote for the final paper). You can also write it as a more conventional proposal using future tense if you prefer. If you have not written a scientific article before, you should look at writing style guidelines.

The paper must have a title, abstract, three main sections (Introduction, Implementation and Experiment) as well as a list of References (all related work must be properly cited) and an Appendix for pair projects. More details below:

• The title should be short but descriptive, like a mini abstract. If possible, come up with a catchy name for your project and use it as part of the title.
• The abstract is less than 150 words and briefly states the motivation and a concise summary of what the project is, what it will do, how it will be built and evaluated. It ends with one or more (expected) contributions. Only mention previous work in the abstract if your project is very closely building on something or should be directly compared to it, typically there are no citations in abstracts.
• The introduction is typically four paragraphs: 1) states the problem context and the motivation for the project idea (why it’s useful, what problem it solves); 2) how your project idea is situated among the most relevant related work (similar interaction techniques, systems, or studies); 3) a description of the expected results of your project (what it is, what it will do, how it will be built, how it will be evaluated); 4) a clear statement what the main research contribution(s) are (i.e. the expected contributions when you're done). In addition, you must have a figure and caption to illustrate some key aspect of your project (this can be sketch-like, hand drawn is fine). Be sure to reference your figure in the introduction.
• The implementation section describes the high level technical approach and/or design, languages/libraries/toolkits you plan to use, and how you will accommodate specialized hardware requirements (if any). Be as detailed as possible and include sketch figures if it makes explaining your approach easier.
• Your research hypothesis and the experimental protocol to evaluate them.
• All references should be cited using the CHI citation format and references listed using the correct style in a References section. For those of you using Word, you are strongly encouraged to use a reference manager like Mendeley or Zotero.
• For projects done as a pair, provide an appendix describing how the tasks and responsibilities are expected to be divided between the two people.

Submit your proposal as a PDF to the corresponding LEARN dropbox before EOD October 23.

Revise the title, abstract, introduction, and implementation sections of your proposal if any changes have occurred, and add a “Related Work” section before the “Implementation” section with a full literature review.

Revise also your experiment, based on provided feedback and considerations after reading the related work. Describe your experiment in a way to make it fully replicable by someone reading your paper. Structuring your experiment section using clear subsections is recommended to ease reading. In the context of a controlled experiment, first describe "method and apparatus" detailing the hardware and API you plan to use, then describe the "Procedure, task and design" detailing the tasks users will perform, how they will complete them, what are the independent variables and how they will be administrated. Then provide information about participants (number of participants, profile...) and last your hypotheses and dependent variables.

The literature review should include scholarly articles, industry and trade articles (as appropriate), books and magazines, and a review of existing products (where applicable). As a general rule of thumb, if you are finding fewer than 15 related resources, you're not looking hard enough. You should model the style and structure of your literature review after the related work sections of seminar papers we have read in class. For citation and reference styles, see notes from the proposal guidelines.

Regarding structure, papers are discussed as related groups or categories, organized into subsections. Although the relevant parts of each previous paper are summarized concisely, you must synthesize several previous or alternative approaches, related technologies, related experiments, and/or related theory into high level summaries with conclusions.

Regarding style, when referring to a paper, you should generally give the authors last names if one or two authors (e.g. "Smith and Brown argue that ..."), the first author's last name plus "et. al." if three or more authors (e.g. "Smith et al. investigated how ..."), or just the project name (e.g. The CoolProject system introduced ...").

After reading your literature review, you the reader to think:

• the review is comprehensive in breadth and depth and related theories have been investigated
• (if reimplementing) reimplementing a previously published technique or commercial device, or replicating an experiment is valuable in terms of validation and replication
• (if your idea is novel) that there are still aspects of the problem space that have not been solved or examined in detail
• (if your idea is novel) that your work is somehow different and novel than what's been done before

A final literature review in a conference paper usually occupies one page not counting the reference list (about 1000 words give or take).

Submit your revised proposal and literature review as a PDF to the corresponding LEARN dropbox before EOD November 9.

You will discuss your project with the instructor for 10 - 15 minutes during the November 16 class time (appointments are available during and after class).

Sign up for a design review slot.

After completing the work for your project, you will revise and extend your literature review into a full paper write up. The final paper should be between 6 to 10 pages long (be concise, don't pad), and must include the following information:

• The title, abstract, introduction and related work from your literature review. These should be revised to match what you actually did and figures should be finished.
• A detailed description of your technique from a human-computer interaction perspective. This means describing what people see when they use it, what it can be used for, why it’s interesting or useful, etc. Figures like schematic flow diagrams, screen captures, video frame stills, etc. will be necessary to explain this clearly. Figures are important in this kind of paper, in many cases they convey more information than writing multiple paragraphs. Make sure you write descriptive captions for your figures.
• A detailed description of the implementation of the technique. This should be detailed enough that someone could reproduce it themselves (you provide algorithm and parameter details). Figures will be necessary to explain this clearly.
• A detailed description of your experiment. Describe tasks, procedure, design and participants. Provide enough details so that someone else can replicate your experiment. Analyze the results using appropriate statistical analysis.
• A discussion section reflecting on what was done, implications for designers and researchers, some light evaluation on technical or user performance, and limitations of the technique. You should be positive about what you did, but also frank and honest when discussing evaluation results and limitations.
• A future work section with advice and directions for others doing work similar to your project. This section is your chance to reflect on your lessons learned and to suggest how things could be improved or better. Include a brief description of what kind of formal user evaluation and/or technical evaluation could be done in the future.

Video

You will also produce a "video figure" demonstrating your technique and/or experiment tasks. The video must be less than 4 minutes (1 to 2 minutes is often enough). Upload your completed video to YouTube, Vimeo, or similar and provide the link in your paper.

Presentation

You will have a 10 minutes presentation to present your work in the final class. Prepare slides covering the different parts of your paper.

Submit your final paper as a PDF + video + Rmd file with data and html output to the corresponding LEARN dropbox before EOD December 11.

Academic Integrity

In order to maintain a culture of academic integrity, members of the University of Waterloo community are expected to promote honesty, trust, fairness, respect and responsibility. [Check www.uwaterloo.ca/academicintegrity/ for more information.]

Grievance

A student who believes that a decision affecting some aspect of his/her university life has been unfair or unreasonable may have grounds for initiating a grievance. Read Policy 70, Student Petitions and Grievances, Section 4, www.adm.uwaterloo.ca/infosec/Policies/policy70.htm. When in doubt please be certain to contact the department's administrative assistant who will provide further assistance.

Discipline

A student is expected to know what constitutes academic integrity [check www.uwaterloo.ca/academicintegrity/] to avoid committing an academic offence, and to take responsibility for his/her actions. A student who is unsure whether an action constitutes an offence, or who needs help in learning how to avoid offences (e.g., plagiarism, cheating) or about 'rules' for group work/collaboration should seek guidance from the course instructor, academic advisor, or the undergraduate Associate Dean. For information on categories of offences and types of penalties, students should refer to Policy 71, Student Discipline, www.adm.uwaterloo.ca/infosec/Policies/policy71.htm. For typical penalties check Guidelines for the Assessment of Penalties, www.adm.uwaterloo.ca/infosec/guidelines/penaltyguidelines.htm.

Appeals

A decision made or penalty imposed under Policy 70 (Student Petitions and Grievances) (other than a petition) or Policy 71 (Student Discipline) may be appealed if there is a ground. A student who believes he/she has a ground for an appeal should refer to Policy 72 (Student Appeals) www.adm.uwaterloo.ca/infosec/Policies/policy72.htm.

Students with Disabilities

The Office for persons with Disabilities (OPD), located in Needles Hall, Room 1132, collaborates with all academic departments to arrange appropriate accommodations for students with disabilities without compromising the academic integrity of the curriculum. If you require academic accommodations to lessen the impact of your disability, please register with the OPD at the beginning of each academic term.