We often think of a smartphone as merely a communications device, but it’s a networked computer so small, powerful and indispensable in our lives that we carry one wherever we go. But this same portability means a smartphone can be easily misplaced or left behind, and because of its value as a pricey pocket computer and as a data access and storage device, smartphones are also targeted by opportunists, thieves and attackers.
Smartphone loss affects millions of people every year. They are the personal items most commonly left behind by passengers according to Uber, the ride-hailing taxi service. Kaspersky Lab, a cybersecurity and anti-virus provider, conducted a recent study that revealed that some 23,000 Android devices alone are lost or stolen every month. And recent surveys show that more than two-thirds of stolen smartphones are never recovered. Even if a smartphone isn’t swiped, many of us leave them unattended at work, making our devices an easy target for nosy coworkers who may want access to sensitive personal and business data.
Whether you use an Android or an iPhone, your smartphone comes with an app to help you recover it should it be misplaced or stolen. Apple’s Find My iPhone and Google’s Find My Device can be used to locate, recover or disable a lost or stolen smartphone. But often minutes or even hours pass between a device being misplaced or stolen and the owner’s realization, and for devices lost in public places that delay can be long enough for someone to take the device and put it into airplane mode, rendering the smartphone untrackable and unrecoverable.
Such apps can help users after the loss of a smartphone, but a device and data loss–prevention solution would be superior.
“I was in a restaurant and after I finished my meal I left without taking my phone,” recounts Jiayi Chen, a PhD candidate in the Cheriton School of Computer Science’s Cryptography, Security and Privacy (CrySP) group. “I was out the door and heading toward the bus stop when a waiter ran out and said, ‘Hey, you forgot your phone.’ I was lucky, but it got me thinking. What if a smartphone could detect whether it’s about to become unattended and then could alert the owner while the device was still within reach?”
That’s exactly the kind of system that Jiayi, his supervisor Cheriton School of Computer Science Professor Urs Hengartner and their colleagues Professor Hassan Khan from the University of Guelph and Professor Mohammad Mannan from Concordia University set out to create and evaluate.
Called Chaperone, the open-source app they developed for Android devices uses active acoustic sensing to detect a smartphone owner’s movements, and based on a complex analysis system locks the phone and alerts the owner when it detects a situation that could lead to loss.
Jiayi Chen is a PhD candidate in the Cryptography, Security, and Privacy (CrySP) group at the Cheriton School of Computer Science, working under the supervision of Professor Urs Hengartner. His research interests include mobile security, smartphone security, and social network privacy.
“Active acoustic sensing is essentially like sonar,” Jiayi explains. “When Chaperone is installed on an Android phone, it uses the device’s speakers to emit an inaudible high-frequency acoustic signal. It then detects the echo of that signal — its reflection from the phone’s owner as well as other people and nearby objects — using its microphone. Based on the changes in the reflected signals, Chaperone can distinguish nearby moving people from static objects. Then, Chaperone extracts the owner’s moving pattern and determines if the owner is about to leave the device unattended.”
Many loss-prevention solutions require additional hardware such as Bluetooth devices or radio frequency ID tags that a person wears so the smartphone can detect when it’s about to be unattended. Jiayi and his collaborators, however, wanted to design Chaperone as a stand-alone solution. All smartphones have a microphone and speaker, so they are capable of performing active acoustic sensing.
But to create a loss-prevention system with high accuracy, acceptable energy use, and the ability to operate in a variety of settings and environments, Chaperone uses four main modules that work together sequentially. The first module decides when to trigger acoustic sensing, the next module performs acoustic sensing, yet another tracks the smartphone user’s movement patterns, and, finally a decision module determines whether it’s likely a user is about to leave a phone unattended and, if so, to immediately lock the device and send an alert.
Chaperone consists of four main modules: trigger, acoustic sensing, user tracking, and decision making (view larger illustration).
Chaperone does not need to perform active acoustic sensing continuously. The trigger module is designed to invoke active acoustic sensing only when certain user-configurable and context-specific conditions are met, thereby reducing Chaperone’s power consumption. It does not need to be active when a user is holding the smartphone, when the device is on the user’s body perhaps tucked in a pocket or purse, or when the smartphone is at home or in a trusted location.
“Once activated, Chaperone’s active acoustic sensing module begins to sense a user’s movements,” Jiayi said. “Using the device’s speaker, Chaperone emits an inaudible high-frequency acoustic signal, then processes the received echo from the device’s microphone to calculate distance and speed of movement.”
This information is sent to Chaperone’s user-tracking module, which locates the smartphone’s user in the immediate environment by filtering echoes digitally from surrounding objects and background noise, and tracking the user’s movement relative to other people nearby.
“The user-tracking module obtains distance and speed information of the owner by analyzing the filtered signal,” Jiayi explains. “The module aims to exclude the effect of other nearby people and sends the user’s motion state information to the decision-making module.”
Many factors affect acoustic sensing.
While brewing coffee, the machine depicted on the left can emit similar high-frequency sounds as Chaperone does, making the signal echo more difficult to detect. The green area depicts the detection range of acoustic sensing. If the smartphone’s owner enters a detection blind area, caused by a wall in this example, acoustic sensing would fail to capture the owner. Therefore, Chaperone should be able to detect such potential device-loss situations and send alerts before it has occurred. Chaperone also has to process information from other people in the vicinity as they pass by (view larger illustration).
The decision-making module then determines if the user is about to leave the device, based on information from the acoustic sensing and user tracking modules. Many environmental factors can limit the detection capabilities in the real world, so dealing with obstacles in a room and other sources of sound requires a comprehensive analysis of processed echoes rather than relying solely on estimated distance alone.
“The decision-making module uses a number of criteria to decide whether a user is about to leave the phone — is the owner’s motion state classified as leaving, is the activity intensity fading, and is the user is no longer close to the device,” Jiayi said. “Only when the user’s movements satisfy these criteria will Chaperone make a positive detection. By using more than just distance information we help reduce the number of false positives.”
What Chaperone does next is configurable by the user. It can lock the phone immediately and trigger an alert — for example, a ringtone, vibration, notification sound, or flashing screen. “Because the alert is selected based on information collected by the trigger module, it’s tailored to the context,” Jiayi said. “If environmental noise is low as in a library, a gentle ringtone would be sufficient to get the user’s attention.”
To test Chaperone, Jiayi conducted several controlled tests in a lab, experimenting with the phone being placed at different angles relative to the user, varying user departure speeds, and the numbers of strangers in the vicinity. Chaperone’s performance was also evaluated using real-world conditions, in areas that have varying amounts of noise, people and diverse layouts and obstacles, and across eight locations: a library, office, restaurant, coffee shop, lounge, bus stop, in a vehicle, and at an academic venue. In all, Jiayi conducted more than 1,300 experiments across different conditions and real-world scenarios.
“The precision rate refers to the situation where there’s no smartphone loss but Chaperone sent the owner an alert,” Jiayi said. “In 93 percent of these cases, the user was actually leaving the device unattended. Or put another way, in 7 percent of the cases the user wasn’t leaving the smartphone but an alert was sent. This false alert — what’s called a false positive — happened in only 7 percent of the total number of alerts. Chaperone’s recall rate is also very high. If a person is leaving their smartphone behind, 96% of the time Chaperone send an alert.”
To assess user experience, Jiayi conducted a semi-structured interview with 17 participants to learn more about smartphone loss experiences, followed by hands-on experience with Chaperone, including areas where it could be improved.
Of the 17 participants, 13 were very satisfied or satisfied with Chaperone’s overall effectiveness. For detection ability, 16 out of 17 were very satisfied or satisfied, and for detection accuracy, 14 of out 17 were very satisfied or satisfied.
“Our current solution is designed based on a smartphone running Android 6.0 or newer,” Jiayi said. “The code is freely available so anyone can download it or improve it. Our experimental data and source code for our prototype are available on GitHub. This will help other researchers as well as let them contribute to the project.”
To learn more about this research, please see Jiayi Chen, Urs Hengartner, Hassan Khan, Mohammad Mannan. Chaperone: Real-time Locking and Loss Prevention for Smartphones. 29th USENIX Security Symposium. August 12–14, 2020.
