E6: Features

Use SIFT to track an object.

Objective 1

Write code to compute SIFT keypoints and descriptors for a distinctive object. I used a card from a child’s game with lots of image details.

Make sure you capture the image of object using the same camera you’ll use to track it in objective 2.
Crop the image to remove any background texture: you want SIFT to only use the actual object.
I converted my object to grayscale before computing SIFT features.
Display the original object image and the image with the keypoints (using drawKeypoints). You should see many keypoints!

objective 1 output

Objective 2

Track the object in a video feed by using one of the flann matching methods (like knnMatch) to test if the image you trained in objective 1 appears in the the video frame.

You don’t need to use background subtraction in this exercise.
I used Lowe’s “ratio test” as explained in the original paper. This tutorial explains how.
Draw all matched keypoints in the video frame to show when the object is being tracked. I used yellow circles. Note that the other card has no yellow circles, so it isn’t being tracked.

objective 2 output

Optional Experiments

Train SIFT on more than one object and identify each object. You could draw text on the image.
Compute a position for the object, perhaps using the mean keypoint location.
Time each frame to see how long SIFT takes to run (it’s quite slow).
Compare it with SURF or one of the other methods.

Submit

Follow the submission instructions on the course information page. Include your code and at least one frame capture from each objective that clearly shows everything was working. In your solution notes, write about any problems you ran into, and the main resource(s) you used (blog posts, online tutorials, stackoverflow posts, papers, textbooks, etc.). These resources should include brief descriptions of what the resource is and how it helped you. The writeup can be short, bullet points are recommended.

You must combine your code, screen captures, and notes together into a single PDF. The best way to do this is by doing everything in a single Jupyter notebook using markdown cells for your notes and inline images for your output. Then just export as PDF.