How a smartphone can overhear, spy, and track
An interesting fact recently appeared in the media: when communicating with a Wired reporter, a Facebook employee asked to turn off the smartphone so that it was more difficult for social networks to record the fact of their communication. To do this, Facebook could use satellite navigation or a microphone, although to prevent surveillance it would be enough to turn them off. There is probably something else on the phone that is worth being afraid of. A modern smartphone is packed with sensors: 2-3 cameras, a light sensor, an accelerometer, a gyroscope, GPS and GLONASS, a magnetometer and others. As NASA researchers using sensors for remote spacecraft repairs for other purposes, attackers can learn a lot about the owner of a smartphone by gaining access to one or several sensors. This is not about cases of physical hacking gadgets.by installing chips or adding wire , and about solving sensors such tasks as spying on users, wiretapping or getting a pincode.
Illustration for the PlaceRaider application that creates a 3D model of a room from frames made without the user's knowledge
The accelerometer is able to track the movement of the device in three axes. In particular, he helps the phone to flip the image, to put it vertically or horizontally depending on the position of the device. In 2006, the acceleration sensor first appeared in the phones, then it was the Nokia 5500 model, in which the accelerometer helped to realize the "sports" functions - the pedometer. The gyroscope was first added to the Apple smartphone - to the iPhone 4. The sensor allows you to control the car in racing games without pressing the arrows and ensures that the person is on the displayed map. For these amenities you have to pay security. Malefactors, having got access to data from one or several sensors, are capable to pull out from them a lot of useful. Such access is easy to get: go to this page , for example.from your smartphone, and you will see that JavaScript easily retrieves data from the gyroscope. The same is possible with HTML5.
Smartphones on iOS and Android are blocked using a digital pin code, graphic or fingerprint. In the first two cases, the phone can be hacked by finding out how it changes position when unlocked by the user. In the case of a digital pincode, researchers from the University of Newcastle UK learned how to guess from the first time in 74% of cases using several sensors - an accelerometer, a gyroscope and a magnetometer. On the third attempt, they cracked the code in 94% of cases.
Popular browsers Safari, Chrome, Firefox, Opera, and Dolphin initially have access to sensors, so attackers simply add an appropriate exploit to the site, rather than asking the owner for permission that is required when installing the app from stores.
A graphical pincode requires a quick password entry of four or more points on a 3x3 field. The field has 389,112 possible combinations, but researchers at the University of Pennsylvania are confident that in reality, users use an order of magnitude less circuitry. Some combinations are inconvenient for permanent use. The application running in the background, at the right time starts the accelerometer, then disables it and transmits the data to fraudsters. Researchers needed only one sensor for hacking.
A similar method in 2015 was used by scientists from the IT-University of Copenhagen, only in this case, the smartwatch watchednot only for entering the code on the smartphone, but also for entering a pin from the card at an ATM or store. Data from the gyro sensor in the clock was transmitted to the smartphone, from where it was sent to the server and unloaded into the CSV.
Millions of people work on laptops and desktop computers every day. Scammers can find out what a person is typing on the keyboard if the smartphone is close to her. Scientists from the Georgia Institute of Technology in 2011 programmed mobile devices to monitor the text typed on the keyboard: gadgets measured the vibrations of the table surface. According to scientists, the procedure was not easy, but the accuracy of determination at that time was up to 80%.
The iPhone 3GS smartphone didn’t work for this kind of work, but the iPhone 4, the first smartphone with a built-in gyro sensor, performed well. A group of researchers attempted to use a microphone, a more sensitive sensor, for surveillance. As a result, the accelerometer turned out to be the preferred method, since it is traditionally less protected by the system.
The technique developed by scientists looked for successive pairs of keystrokes. The application finds out in which place of the keyboard the keys were pressed - on the left-top and on the right-below, on the right-below and on the right-on-and also determines the distance for each pair of keys. It then compares the results with the preloaded dictionary. The method worked with words of three or more letters.
It is possible not only to steal data from the accelerometer, but also to control the device with its help, forcing the smartphone to perform the necessary actions for fraudsters. Speaker for $ 5 helped to crack 20 accelerometers from 5 manufacturers using sound waves. A group of researchers from the University of Michigan and the University of South Carolina used a "music virus", as they called their technique in an interview with The New-York Times, to make the Fitbit app believe that the user had completed thousands of steps and control the toy car using the phone. The goal of the researchers was to create software solutions to counter such attacks.
Since the gyroscope picks up sound vibrations, it can be used as a hard disk of a computer for hidden listening. Scientists from Stanford University and specialists from the Israeli defense company Rafael have found a way to turn the gyroscope of an Android smartphone into a constantly on microphone. They developed the Gyrophone app: sensors on many Android devices pick up vibrations from a sound frequency of 80 to 250 hertz.
The voice of an adult male has a frequency of 85 to 155 Hz, women - from 165 to 255 Hz. Consequently, the gyro sensor is able to listen to human speech. The iPhone gyroscope uses a frequency below 100 Hz, so it is not suitable for the same purpose, but, nevertheless, it can help you to recognize the speaker's gender in individual words. The accuracy of the instrument in 2014 was not very high - up to 64%.
Well-coordinated work of several sensors in the smartphone and machine learning will help to track the movements of the owner of the device with the satellite navigation turned off. The illustration below shows how exactly the route is determined by the method proposed by a group of researchers from the Institute of Electrical and Electronics Engineers (IEEE). Green indicates the path that the user has traveled by transport, orange indicates the distance traveled, and black indicates the data from the GPS.
The PinMe app maps sensor information to open data. First, the exploit receives information about the latest IP address of the smartphone and connecting to Wi-Fi to determine the starting point of the route. Then - in the direction, speed and frequency of stopping recognizes the difference between walking, driving a car and public transport, flights on an airplane. PinMe compares the obtained data with information from open sources: navigation data takes from OpenStreetMaps, elevation map in Google Maps, route data from airline schedules and railway lines. To clarify the route, the application used the Weather Channel weather service: accurate information on temperature and air pressure helps to neutralize the influence of weather conditions on the information collected by the sensors.
In 2010, a similar technology was used by the Japanese telecommunications corporation KDDI: the accelerometer in the smartphone was used to spy on employees. The data from the sensor made it possible to understand whether a person is walking up or down a flat surface, shaking out rubbish from their bins or washing floors. In 2015, specialists from Nanjing University in China used data from the accelerometer to monitor the movement of people in the subway.
To determine the location of the owner of the smartphone can the application that receives data on the battery status. This information can be obtained by any application, since it does not require additional permissions. Scientists from Stanford and specialists from the defense company Rafael, who have already been mentioned above, have developedPower Spy technology.
The location of the user is determined with 90% accuracy due to the analysis of the discharge rate of the battery: this is how the scientists determined the remoteness of the gadget from the repeaters. But such accuracy is possible only if the user is not the first time along this route.
In 2012, the American military research center in Indiana and scientists from Indiana University developed the PlaceRaider application for Android 2.3 smartphones, which could reconstruct the user's environment in 3D.
The user had to download the application with the ability to take photos and give him permission to use the camera and send them. PlaceRaider, working in the background, turned off the shutter sound, so as not to disturb the user. Then the program randomly took photos, saving information about the time, place and orientation of the smartphone. After filtering the photo and removing bad frames made, for example, in the user's pocket, the application sent them to the server, where the 3D model of the room was created.
To test the effectiveness of this idea, scientists gave the "infected" phone numbers to twenty volunteers who did not know about the application, and sent them to the office with various simple tasks. At the next stage, two groups of people viewed the results: one - separate photos, the second - 3D models. Both groups were looking for QR codes, checks, documents, as well as calendars that attackers could use to determine when the victim would not be in a certain place.
The application for the "end user", that is, in the worst case - the criminal, and in ours - the scientists, allowed to bring certain parts of the frame in the best traditions of Hollywood films. In this case, the person who opened the 3D-model could click on a certain point, after which the application searched for better quality photos from the base, taken closer to the desired place. The image below shows the number of the check lying on the table.
The greater the power, the greater the responsibility: it must be remembered by the developers of smartphones and applications to them, which today offer endless possibilities for hacking users' wallets, tracking movement and identifying interests for more accurate advertising targeting. In real life, of course, most of these researchers are interesting at best to the screenwriters of the Black Mirror.
Real hackers periodically develop cool ways to withdraw money from the public, but they do not implement them very well. For example, in February 2018 they were able to downloada miner at the government sites of the UK, USA and Canada, forcing him to earn cryptocurrency for himself within four hours. Instead of getting a huge amount of information from these sites and selling it, they hooked up a miner and earned $ 24. However, after clarifying the circumstances and the money, the mining service did not pay them.
Illustration for the PlaceRaider application that creates a 3D model of a room from frames made without the user's knowledge
The accelerometer is able to track the movement of the device in three axes. In particular, he helps the phone to flip the image, to put it vertically or horizontally depending on the position of the device. In 2006, the acceleration sensor first appeared in the phones, then it was the Nokia 5500 model, in which the accelerometer helped to realize the "sports" functions - the pedometer. The gyroscope was first added to the Apple smartphone - to the iPhone 4. The sensor allows you to control the car in racing games without pressing the arrows and ensures that the person is on the displayed map. For these amenities you have to pay security. Malefactors, having got access to data from one or several sensors, are capable to pull out from them a lot of useful. Such access is easy to get: go to this page , for example.from your smartphone, and you will see that JavaScript easily retrieves data from the gyroscope. The same is possible with HTML5.
Smartphones on iOS and Android are blocked using a digital pin code, graphic or fingerprint. In the first two cases, the phone can be hacked by finding out how it changes position when unlocked by the user. In the case of a digital pincode, researchers from the University of Newcastle UK learned how to guess from the first time in 74% of cases using several sensors - an accelerometer, a gyroscope and a magnetometer. On the third attempt, they cracked the code in 94% of cases.
Popular browsers Safari, Chrome, Firefox, Opera, and Dolphin initially have access to sensors, so attackers simply add an appropriate exploit to the site, rather than asking the owner for permission that is required when installing the app from stores.
A graphical pincode requires a quick password entry of four or more points on a 3x3 field. The field has 389,112 possible combinations, but researchers at the University of Pennsylvania are confident that in reality, users use an order of magnitude less circuitry. Some combinations are inconvenient for permanent use. The application running in the background, at the right time starts the accelerometer, then disables it and transmits the data to fraudsters. Researchers needed only one sensor for hacking.
A similar method in 2015 was used by scientists from the IT-University of Copenhagen, only in this case, the smartwatch watchednot only for entering the code on the smartphone, but also for entering a pin from the card at an ATM or store. Data from the gyro sensor in the clock was transmitted to the smartphone, from where it was sent to the server and unloaded into the CSV.
Millions of people work on laptops and desktop computers every day. Scammers can find out what a person is typing on the keyboard if the smartphone is close to her. Scientists from the Georgia Institute of Technology in 2011 programmed mobile devices to monitor the text typed on the keyboard: gadgets measured the vibrations of the table surface. According to scientists, the procedure was not easy, but the accuracy of determination at that time was up to 80%.
The iPhone 3GS smartphone didn’t work for this kind of work, but the iPhone 4, the first smartphone with a built-in gyro sensor, performed well. A group of researchers attempted to use a microphone, a more sensitive sensor, for surveillance. As a result, the accelerometer turned out to be the preferred method, since it is traditionally less protected by the system.
The technique developed by scientists looked for successive pairs of keystrokes. The application finds out in which place of the keyboard the keys were pressed - on the left-top and on the right-below, on the right-below and on the right-on-and also determines the distance for each pair of keys. It then compares the results with the preloaded dictionary. The method worked with words of three or more letters.
It is possible not only to steal data from the accelerometer, but also to control the device with its help, forcing the smartphone to perform the necessary actions for fraudsters. Speaker for $ 5 helped to crack 20 accelerometers from 5 manufacturers using sound waves. A group of researchers from the University of Michigan and the University of South Carolina used a "music virus", as they called their technique in an interview with The New-York Times, to make the Fitbit app believe that the user had completed thousands of steps and control the toy car using the phone. The goal of the researchers was to create software solutions to counter such attacks.
Since the gyroscope picks up sound vibrations, it can be used as a hard disk of a computer for hidden listening. Scientists from Stanford University and specialists from the Israeli defense company Rafael have found a way to turn the gyroscope of an Android smartphone into a constantly on microphone. They developed the Gyrophone app: sensors on many Android devices pick up vibrations from a sound frequency of 80 to 250 hertz.
The voice of an adult male has a frequency of 85 to 155 Hz, women - from 165 to 255 Hz. Consequently, the gyro sensor is able to listen to human speech. The iPhone gyroscope uses a frequency below 100 Hz, so it is not suitable for the same purpose, but, nevertheless, it can help you to recognize the speaker's gender in individual words. The accuracy of the instrument in 2014 was not very high - up to 64%.
Well-coordinated work of several sensors in the smartphone and machine learning will help to track the movements of the owner of the device with the satellite navigation turned off. The illustration below shows how exactly the route is determined by the method proposed by a group of researchers from the Institute of Electrical and Electronics Engineers (IEEE). Green indicates the path that the user has traveled by transport, orange indicates the distance traveled, and black indicates the data from the GPS.
The PinMe app maps sensor information to open data. First, the exploit receives information about the latest IP address of the smartphone and connecting to Wi-Fi to determine the starting point of the route. Then - in the direction, speed and frequency of stopping recognizes the difference between walking, driving a car and public transport, flights on an airplane. PinMe compares the obtained data with information from open sources: navigation data takes from OpenStreetMaps, elevation map in Google Maps, route data from airline schedules and railway lines. To clarify the route, the application used the Weather Channel weather service: accurate information on temperature and air pressure helps to neutralize the influence of weather conditions on the information collected by the sensors.
In 2010, a similar technology was used by the Japanese telecommunications corporation KDDI: the accelerometer in the smartphone was used to spy on employees. The data from the sensor made it possible to understand whether a person is walking up or down a flat surface, shaking out rubbish from their bins or washing floors. In 2015, specialists from Nanjing University in China used data from the accelerometer to monitor the movement of people in the subway.
To determine the location of the owner of the smartphone can the application that receives data on the battery status. This information can be obtained by any application, since it does not require additional permissions. Scientists from Stanford and specialists from the defense company Rafael, who have already been mentioned above, have developedPower Spy technology.
The location of the user is determined with 90% accuracy due to the analysis of the discharge rate of the battery: this is how the scientists determined the remoteness of the gadget from the repeaters. But such accuracy is possible only if the user is not the first time along this route.
In 2012, the American military research center in Indiana and scientists from Indiana University developed the PlaceRaider application for Android 2.3 smartphones, which could reconstruct the user's environment in 3D.
The user had to download the application with the ability to take photos and give him permission to use the camera and send them. PlaceRaider, working in the background, turned off the shutter sound, so as not to disturb the user. Then the program randomly took photos, saving information about the time, place and orientation of the smartphone. After filtering the photo and removing bad frames made, for example, in the user's pocket, the application sent them to the server, where the 3D model of the room was created.
To test the effectiveness of this idea, scientists gave the "infected" phone numbers to twenty volunteers who did not know about the application, and sent them to the office with various simple tasks. At the next stage, two groups of people viewed the results: one - separate photos, the second - 3D models. Both groups were looking for QR codes, checks, documents, as well as calendars that attackers could use to determine when the victim would not be in a certain place.
The application for the "end user", that is, in the worst case - the criminal, and in ours - the scientists, allowed to bring certain parts of the frame in the best traditions of Hollywood films. In this case, the person who opened the 3D-model could click on a certain point, after which the application searched for better quality photos from the base, taken closer to the desired place. The image below shows the number of the check lying on the table.
The greater the power, the greater the responsibility: it must be remembered by the developers of smartphones and applications to them, which today offer endless possibilities for hacking users' wallets, tracking movement and identifying interests for more accurate advertising targeting. In real life, of course, most of these researchers are interesting at best to the screenwriters of the Black Mirror.
Real hackers periodically develop cool ways to withdraw money from the public, but they do not implement them very well. For example, in February 2018 they were able to downloada miner at the government sites of the UK, USA and Canada, forcing him to earn cryptocurrency for himself within four hours. Instead of getting a huge amount of information from these sites and selling it, they hooked up a miner and earned $ 24. However, after clarifying the circumstances and the money, the mining service did not pay them.