A team of computer scientists has developed a new system that recognizes near-keyboard hand gestures to expand laptop interaction.
The new technology is an innovation in the field of human-computer interaction and allows users to give commands that would otherwise involve keyboard shortcuts or mouse round-trips.
Known as Typealike, the system was created by a team of researchers at the Cheriton School of Computer Science — Nalin Chhibber, a recent master’s graduate, Hemant Surale, a recent PhD graduate, Fabrice Matulic, a recent postdoctoral researcher, and Professor and Cheriton Faculty Fellow Daniel Vogel.
L to R: Nalin Chhibber, Hemant Surale, Fabrice Matulic and Daniel Vogel
“It started with a simple idea about new ways to use a webcam,” said Nalin. “The webcam is pointed at your face, but the most interaction happening on a computer is around your hands. So, we thought, what could we do if the webcam could pick up hand gestures?”
The initial insight led to the development of a small mechanical attachment that redirects the webcam downwards towards the hands. The team then created a software program capable of understanding distinct hand gestures in variable conditions and for different users. The team used machine learning techniques to train the Typealike program.
“It’s a neural network, so you need to show the algorithm examples of what you’re trying to detect,” said Fabrice. “Some people will make gestures a little bit differently, and hands vary in size, so you have to collect a lot of data from different people with different lighting conditions.”
The team recorded a database of hand gestures with dozens of research volunteers. They also had the volunteers do tests and surveys to help the team understand how to make the program as functional and versatile as possible.
Typealike
postures
are
postures
formed
using
the
left
or
right
hand,
an
open
or
closed
hand
form,
and
different
wrist
orientations,
all
further
distinguished
by
hand
position
when
on,
just
beside,
or
just
below
the
keyboard.
Different
postures
can
trigger
different
interactions,
for
example (a)
an
open
left
hand
at
90°
wrist
rotation
beside
the
keyboard
can
decrease
the
volume;
(b)
an
open
left
hand
at
0°
on
the
keyboard
can
open
a
screen
capture
snipping
tool;
(c)
a
closed
right
hand
0°
posture
below
the
keyboard
could
turn
the
car
in
a
racing
game;
(d)
an
open
right
hand
at
0°
on
the
keyboard
could
advance
a
document
page;
and
(e)
a
right
open
90°
hand
posture
beside
the
keyboard
could
increase
the
volume.
“We’re always setting out to make things people can easily use,” Professor Vogel said. “People look at something like Typealike, or other new tech in the field of human-computer interaction, and they say it just makes sense. That’s what we want. We want to make technology that’s intuitive and straightforward, but sometimes to do that takes a lot of complex research and sophisticated software.”
Learn more about Typealike on its project Github page and the feature article in Waterloo News.
Please also see Nalin Chhibber, Hemant Bhaskar Surale, Fabrice Matulic, and Daniel Vogel. 2021. Typealike: Near-Keyboard Hand Postures for Expanded Laptop Interaction. Proceedings of the ACM on Human-Computer Interaction 5, ISS, Article 486 (November 2021), DOI: https://doi.org/10.1145/3486952.
This research received a Best Paper Honorable Mention Award at ACM ISS 2021.