CS889 Advanced Topics in HCI (Spring 2015)
Applied Computer Vision
for Human-Computer Interaction

Instructor

Daniel Vogel
dvogel@uwaterloo, office hours by appt. DC 3145

When and Where

Tuesdays 1:00 to 3:50 PM in DC 3313

This special topics graduate course examines human-computer interaction (HCI) research that applies computer vision. For the purpose of this course, computer vision is defined as algorithms to acquire, process, and analyse images or video to establish some level of understanding to control a computer or interpret information. HCI research topics focus on input and interaction techniques, but we will also cover other applications like interactive design tools and analysis of HCI experiments. Weekly student-led research seminars and computer vision programming assignments using OpenCV will prepare students with the theoretical and practical background for a final project.

The primary learning outcomes are:

• familiarity with HCI research using computer vision
• knowledge of fundamental computer vision algorithms and image processing techniques
• experience writing computer vision code using OpenCV

View course announcements and schedule (readings, seminars, exercises, and project deadlines).

The course marking scheme is:

20%	Participation
10%	Microreviews
15%	Exercises
15%	Seminar Presentation
40%	Project

You should also read the course policies.

Participation is important: students are expected to attend all classes, read the papers, and contribute to class discussions. As a rule of thumb, you are expected to contribute to class discussion at least 1 or 2 times each class: ask a question, comment on a topic, clarify a point, etc. Attendance alone is not enough for your participation mark.

Worth 20% of grade: mark based on level of participation in class discussions.

Before each class with seminar papers, all students will submit a three sentence review of each paper being presented. Seminar papers are marked with an 'id' consisting of the first author's name and year (e.g. "smith95", beginning in week 2).

Write an unadorned three sentence paragraph (without leading numbers, headings, new lines, etc.). In English writing, a sentence is always punctuated with period, question mark, or exclamation mark. The three sentences should answer the following questions, in this order:

1. What is the strongest aspect of the paper? (i.e. why do you think it was accepted?)
2. What is the weakest aspect of the paper? (i.e. what would you do differently?)
3. How could the topic be extended and refined in future work? Or, how could it influence your course project?

Your microreview with your name will be made public to the class after the submission deadline. You can optionally rate the paper on a scale of 1 to 5.

You do not need to do a microreview for the paper you are presenting on, for any paper not marked with an 'id', or for any related papers.

Submit microreviews using this form before 1:00 PM on Monday (the day before class).

There are small programming exercises each week in the first part of the course to provide practical experience implementing computer vision techniques. Each exercise will have 1 or more objectives that you must write code to solve. In addition to submitting final code with a screen capture showing your output, you will also submit a very short solution description (less than 500 words) explaining what you did, what was hard, and an annotated list of the main resource(s) you used (blog posts, online tutorials, stackoverflow posts, papers, textbooks, etc.). The solution description can be a simple text file.

We will primarily be using OpenCV with the Python bindings for in-class examples, but you are free to use another language like C++, C#, Java, or JavaScript, or another library like SimpleCV or EMGU for assignments. Matlab is a great tool for computer vision research, but arguably less useful for building interactive systems, so its use is discouraged. Note that you may not get much help from the instructor if you choose a language other than Python or a non-OpenCV library.

All exercises are done individually. Please review university policies on plagiarism.

The expectation is that all students have access to a machine with a webcam to complete these assignments, let the instructor know right away if this will be a problem. There are some USB webcams available to loan.

Submit your code, output capture, and solution description in a single zip archive to the corresponding LEARN dropbox before the start of the next class.

Every student is required to lead a paper seminar. Each seminar has three parts: present a critique of the paper (15 minutes), lead a discussion about the paper (10 - 15 minutes); give a mini-tutorial about one computer vision technique related to the paper (15 minutes). You may combine your critique presentation with the class discussion.

Instructor Advance Review

You must send a draft of your presentation and tutorial to the instructor by email before 9 AM Monday for review. Comments will be returned before 1 PM.

Paper Presentation
(15 minutes)

Use the paper content and examples as a way to introduce and address questions that go beyond the paper itself:

• Establish a context for the paper. What's the problem area? What is the relevant background? Why do we care? Why is the problem interesting, and why is it deep?
• What's the big idea and contribution of the paper? Why do you think the paper was accepted?
• How effective are the paper's results? What are the strengths and weaknesses?
• What are other related papers?
• What are potential extensions, future work?
• If possible, talk about how this paper might apply to your project.

As a rough guide, keep specific summary of information in the paper to less than 25% of your talk and the rest for answering the questions above. Spread the specific content of the paper around and use it to ground the discussions of the more important seminar questions. Remember, everyone in the class will have read the paper ahead of time to create their microreviews, so everyone already has a good grasp of what the paper did. Whatever you do, do not simply give a summary of the paper contents (the grading rubric below makes it clear you will not do very well if you do).

If you find that your slides cover all sections of the paper in the same order in which they appear and you only show figures and tables from the paper, then you are creating a summary presentation. Don't do that.

Class Discussion
(10 - 15 minutes)

You are also responsible for leading discussion during or after your presentation. Be prepared to get the class started by seeding the discussion with open-ended questions and some controversial statements. You will have access to the class microreviews 24 hours before your seminar, so budget some time to read through them, summarize the responses in a final slide. You are encouraged to call on individuals in the class during the discussion to expand or justify their responses (you will have their names associated with the microreviews). You will have to manage the class: this means keeping people on topic, encouraging everyone to speak, and making sure the discussion isn't dominated by a few people.

Technical Tutorial
(15 minutes)

You will give a short tutorial to introduce the theory and implementation of one computer vision technique related to the paper. Tutorial topics are indicated in the schedule beside each paper id.

Most papers do not provide enough technical background for the computer vision techniques, you will have to find secondary sources. Your objective is to provide enough theory to give intuition behind how the algorithm works and enough implementation details to get someone started using the technique. Most techniques can be implemented in OpenCV, but some (like Kinect Fusion) require specialized hardware and software.

Submit your final presentation and tutorial materials to the corresponding LEARN dropbox before your seminar begins.

You will design and implement a proof-of-concept demonstration using computer vision for human-computer interaction. This could be an input or interaction technique, augmented reality technique, design tool system, experiment analysis, toolkit, etc. Proof of concept means that enough of the system is implemented to 1) demonstrate technical feasibility and 2) test basic usability with a few real users.

Projects can be done individually or as a pair. You will need to choose a problem that fits your skill set and that can be completed by the end of the term. There are four stages to the project: a proposal, a literature review, a design review with the instructor, and final deliverables including a paper, video, and demo presentation to the class.

The project grade is broken down as follows:

15%	Proposal
15%	Literature Review
70%	Final Deliverables (demo, paper, video)

You should start thinking about your project early in the course. There are two stages to the proposal to help you do this.

Initial Ideas

Come up with at least three possible project ideas. Submit these three ideas (each with very brief description, maybe a small sketch if it makes it easier to understand) to the instructor for feedback.

Proposal

Write a 2 page proposal using the SIGCHI Conference Publications Format. You can write your proposal in present or past tense, as though you've already done the work (this is called “future writing” and it makes it easier to reuse what you wrote for the final paper). You can also write it as a more conventional proposal using future tense if you prefer. If you have not written a scientific article before, you should look at writing style guidelines.

The paper must have a title, abstract, two main sections (Introduction and Implementation) as well as a list of References (all related work must be properly cited) and an Appendix for pair projects, details below:

• The title should be short but descriptive, like a mini abstract. If possible, come up with a catchy name for your project and use it as part of the title.
• The abstract is less than 150 words and briefly states the motivation and a concise summary of what the project is, what it will do, how it will be built. It ends with one or more (expected) contributions. Only mention previous work in the abstract if your project is very closely building on something or should be directly compared to it, typically there are no citations in abstracts.
• The introduction is typically four paragraphs: 1) states the problem context and the motivation for the project idea (why it’s useful, what problem it solves); 2) how your project idea is situated among the most relevant related work (similar systems, algorithms, or techniques); 3) a description of the expected results of your project (what it is, what it will do, how it will be built); 4) a clear statement what the main research contribution(s) are (i.e. the expected contributions when you're done). In addition, you must have a figure and caption to illustrate some key aspect of your project (this can be sketch-like, hand drawn is fine). Be sure to reference your figure in the introduction.
• The implementation section describes the high level technical approach and/or design, languages/libraries/toolkits you plan to use, and how you will accommodate specialized hardware requirements (if any). Be as detailed as possible and include sketch figures if it makes explaining your approach easier.
• All references should be cited using the CHI citation format and references listed using the correct style in a References section. For those of you using Word, you are strongly encouraged to use a reference manager like Zotero with the SIGCHI CLS reference format.
• For projects done as a pair, provide an appendix describing how the tasks and responsibilities are expected to be divided between the two people.

Submit your proposal as a PDF to the corresponding LEARN dropbox before EOD Jun 9.

Revise the title, abstract, introduction, and implementation sections of your proposal if any changes have occurred, and add a “Related Work” section before the “Implementation” section with a full literature review.

The literature review should include scholarly articles, industry and trade articles (as appropriate), books and magazines, and a review of existing products (where applicable). As a general rule of thumb, if you are finding fewer than 15 related resources, you're not looking hard enough. You should model the style and structure of your literature review after the related work sections of seminar papers we have read in class. For citation and reference styles, see notes from the proposal guidelines.

Regarding structure, papers are discussed as related groups or categories, organized into subsections. Although the relevant parts of each previous paper are summarized concisely, you must synthesize several previous or alternative approaches, related technologies, and/or related theory into high level summaries with conclusions.

After reading your literature review, you want readers to think:

• the review is comprehensive in breadth and depth and related theories have been investigated
• that there are still aspects of the problem space that have not been solved or examined in detail
• that your work is somehow different and novel than what's been done before

A final literature review in a conference paper usually occupies one page not counting the reference list (about 1000 words give or take). At this stage of the class project, it is fine (perhaps even better) to write a bit more and refine it down to a suitable length when writing up your final paper (but keep it under 1500 words).

Submit your revised proposal and literature review as a PDF to the corresponding LEARN dropbox before EOD Jun 16.

You will discuss your project with the instructor for 10 - 15 minutes June 30 class (appointments are available during and after class).

Sign up for a design review slot.

After completing the work for your project, you will revise and extend your literature review into a full paper write up. The final paper should be between 7 and 10 pages long, and must include the following information:

• The title, abstract, introduction and related work from your literature review. These should be revised to match what you actually did and figures should be finished.
• A detailed description of your technique from a human-computer interaction perspective. This means describing what people see when they use it, what it can be used for, why it’s interesting or useful, etc. Figures like schematic flow diagrams, screen captures, video frame stills, etc. will be necessary to explain this clearly. Figures are important in this kind of paper, in many cases they convey more information than writing multiple paragraphs. Make sure you write descriptive captions for your figures.
• A detailed description of the implementation of the technique from a computer vision perspective. This should be detailed enough that someone could reproduce it themselves (you provide algorithm and parameter details). Figures will be necessary to explain this clearly.
• A discussion section reflecting on what was done, implications for designers and researchers, some light evaluation on technical or user performance, and limitations of the technique. You should be positive about what you did, but also frank and honest when discussing evaluation results and limitations.
• A future work section with advice and directions for others doing work similar to your project. This section is your chance to reflect on your lessons learned and to suggest how things could be improved or better. Include a brief description of a more formal user evaluation and/or technical evaluation that could be done.

NOTE: For the purpose of this class, the evaluation is expected to be informal and high-level, not a formal user study or technical quality assurance. User analysis might be qualitative observations from a few friends trying out your system (report what you saw and what they said). Technical analysis might be qualitative or quantitative results for using the system under different conditions (different lighting, inputs, difficulties).

I expect to see figures demonstrating the technique in some way (schematics, photos, video stills, screen captures, etc) and figures summarizing evaluation results.

Video

You will also produce a "video figure" demonstrating your technique and/or experiment tasks. The video must be less than 4 minutes (1 to 2 minutes is often enough). Upload your completed video to YouTube, Vimeo, or similar and provide the link in your paper.

Demo

You will have less than 10 minutes to demonstrate your technique in the final class. Showing an early cut of your video is also a good idea, especially if it's hard to demo live in the class. Note that this does not have to be a formal presentation, in fact you don't need to use a projector at all -- it may be easier/better to have the class gather around to see your demo and ideally try it out themselves. If you want to show a few slides to motivate your problem that's fine, but focus on a demo.

Submit your final paper as a PDF (with video link) to the corresponding LEARN dropbox before July 28.

Academic Integrity

In order to maintain a culture of academic integrity, members of the University of Waterloo community are expected to promote honesty, trust, fairness, respect and responsibility. [Check www.uwaterloo.ca/academicintegrity/ for more information.]

Grievance

A student who believes that a decision affecting some aspect of his/her university life has been unfair or unreasonable may have grounds for initiating a grievance. Read Policy 70, Student Petitions and Grievances, Section 4, www.adm.uwaterloo.ca/infosec/Policies/policy70.htm. When in doubt please be certain to contact the department's administrative assistant who will provide further assistance.

Discipline

A student is expected to know what constitutes academic integrity [check www.uwaterloo.ca/academicintegrity/] to avoid committing an academic offence, and to take responsibility for his/her actions. A student who is unsure whether an action constitutes an offence, or who needs help in learning how to avoid offences (e.g., plagiarism, cheating) or about 'rules' for group work/collaboration should seek guidance from the course instructor, academic advisor, or the undergraduate Associate Dean. For information on categories of offences and types of penalties, students should refer to Policy 71, Student Discipline, www.adm.uwaterloo.ca/infosec/Policies/policy71.htm. For typical penalties check Guidelines for the Assessment of Penalties, www.adm.uwaterloo.ca/infosec/guidelines/penaltyguidelines.htm.

Appeals

A decision made or penalty imposed under Policy 70 (Student Petitions and Grievances) (other than a petition) or Policy 71 (Student Discipline) may be appealed if there is a ground. A student who believes he/she has a ground for an appeal should refer to Policy 72 (Student Appeals) www.adm.uwaterloo.ca/infosec/Policies/policy72.htm.

Students with Disabilities

The Office for persons with Disabilities (OPD), located in Needles Hall, Room 1132, collaborates with all academic departments to arrange appropriate accommodations for students with disabilities without compromising the academic integrity of the curriculum. If you require academic accommodations to lessen the impact of your disability, please register with the OPD at the beginning of each academic term.