Optical Snow

(Joint work with Mike Langer, McGill University)

We introduce a new category of image motion called Optical Snow.

Optical snow is the type of motion an observer sees when watching a snow fall.  Flakes that are closer to the observer appear to move faster than flakes which are farther away.

Snow drawing

Optical snow also occurs when an observer moves relative to a densely cluttered scene.

Tree drawing

Note that in both cases, we have dense motion parallax.  Within each image region there can be an arbitrary number of speeds, corresponding to objects at different depths.

Despite the apparent complexity of such motion, we still have a rich perception of motion and depth structure.  Here are two simple image sequences showing dense cluttered motion.

Example 1: Camera moves horizontally
in front of a holly bush.
Example 2: Camera moves vertically
in a field of equal-sized spheres.
Holly bush (frame 0)
(click on image to see Mpeg video)
Falling spheres (frame 0)
(click on image to see Mpeg video)

Current approaches to motion estimation assume that image motion can be described a collection of small image patches, each with a unique image velocity.  The resulting motion field is referred to as optical flow.  Recently optical flow models have been extended to allow a small number of flow measurements at each point, eg., to allow for transparency effects and multiple motions near occlusion boundaries.  However, such methods are inappropriate for densely cluttered motion, since there can be an arbitrary number of motions within any image patch.

We argue that rather than a estimating a unique velocity as optical flow methods do, we should fit a linear familty of velocities in each image patch.  For the case of a camera translating laterally (ie., perpendicular to the viewing direction) this amounts to estimating the common direction of image motion.  An example of lateral motion is that seen be a camera pointing out the side of a moving vehicle.  For the general case of a forward moving and/or rotating camera, a linear range of velocities can be fit within small image patches.

We are currently using frequency-domain methods (ie., spatiotemporal filters) to estimate the direction of motion and the range of speeds.  In (Langer and Mann, ICCV'01; Mann and Langer, ICPR'02, Langer and Mann, IJCV'03) we characterize the structure of optical snow in the frequency domain and present algorithms for motion estimation.

NEW: That data and software for these experiments is available here.