Privacy Protection for Audio Sensing Against Multi-Microphone Adversaries

Abstract

Audio-based sensing enables fine-grained human activity detection, such as sensing hand gestures and contact-free estimation of the breathing rate. A passive adversary, equipped with microphones, can leverage the ongoing sensing to infer private information about individuals. Further, with multiple microphones, a beamforming-capable adversary can defeat the previously-proposed privacy protection obfuscation techniques. Such an adversary can isolate the obfuscation signal and cancel it, even when situated behind a wall. AudioSentry is the first to address the privacy problem in audio sensing by protecting the users against a multi-microphone adversary. It utilizes the commodity and audio-capable devices, already available in the user’s environment, to form a distributed obfuscator array. AudioSentry packs a novel technique to carefully generate obfuscation beams in different directions, preventing the multi-microphone adversary from canceling the obfuscation signal. AudioSentry follows by a dynamic channel estimation scheme to preserve authorized sensing under obfuscation. AudioSentry offers the advantages of being practical to deploy and effective against an adversary with a large number of microphones. Our extensive evaluations with commodity devices show that protects the user’s privacy against a 16-microphone adversary with only four commodity obfuscators, regardless of the adversary’s position. AudioSentry provides its privacy-preserving features with little overhead on the authorized sensor.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • [1] [n. d.]. Amazon Alexa Premium Far-Field Voice Development Kit. https://developer.amazon.com/alexa-voice-service/devkits/amazon-premium-voice. ([n. d.]).

  • [2] [n. d.]. Apple Ipad Tech Specs. https://www.apple.com/ipad-9.7/specs/. ([n. d.]).

  • [3] [n. d.]. HomePod reinvents music in the home. https://www.apple.com/newsroom/2017/06/homepod-reinventsmusic-in-the-home/. ([n. d.]).

  • [4] [n. d.]. Pixel 2 Tech Specs. https://store.google.com/us/product/pixel_2_specs?hl=en-US. ([n. d.]).

  • [5] 2017. The Nielsen Total Audience Report: Q2 2017. (November 2017). Retrieved May 8, 2018 from http://www.nielsen.com/us/en/insights/reports/2017/thenielsen-total-audience-q2-2017.html

  • [6] 2018. Elliptic Labs Makes Smart Speakers More Intelligent with New INNER REFLECTION™ Ultrasound Virtual Sensor. (February 2018). Retrieved May 8, 2018 from http://www.ellipticlabs.com/2018/02/26/elliptic-labstouch-free-ultrasound-gesture-technology-leveraging\-thequalcomm-snapdragon-neural-processing-engine-2/

  • [7] 2018. The Smart Audio Report, Fall/Winter 2017 from NPR and Edison Research. (January 2018). Retrieved May 8, 2018 from https://www.nationalpublicmedia.com/smart-audioreport/

  • [8] Md Tanvir Islam Aumi, Sidhant Gupta, Mayank Goel, Eric Larson, and Shwetak Patel. 2013. DopLink: using the doppler effect for multi-device interaction. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 583–586.

  • [9] Daniel S. Berger, Francesco Gringoli, Nicolò Facchi, Ivan Martinovic, and Jens Schmitt. 2014. Gaining Insight on Friendly Jamming in a Real-world IEEE 802.11 Network. In Proceedings of the 2014 ACM Conference on Security and Privacy in Wireless & Mobile Networks (WiSec ’14). 105–116. https://doi.org/10.1145/2627393.2627403

  • [10] Andreas Braun, Stefan Krepp, and Arjan Kuijper. 2015. Acoustic tracking of hand activities on surfaces. In Proceedings of the 2nd international Workshop on Sensor-based Activity Recognition and Interaction. ACM, 9.

  • [11] James Brown, Ibrahim Ethem Bagci, Alex King, and Utz Roedig. 2013. Defend Your Home#: Jamming Unsolicited Messages in the Smart Home. In Proceedings of the 2Nd ACM Workshop on Hot Topics on Wireless Network Security and Privacy (HotWiSec ’13). 1–6. https://doi.org/10.1145/2463183.2463185

  • [12] Ke-Yu Chen, Daniel Ashbrook, Mayank Goel, Sung-Hyuck Lee, and Shwetak Patel. 2014. AirLink: sharing files between multiple devices using in-air gestures. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 565–569.

  • [13] Qi Alfred Chen, Zhiyun Qian, and Zhuoqing Morley Mao. 2014. Peeking into Your App without Actually Seeing It: UI State Inference and Novel Android Attacks.. In USENIX Security Symposium. 1037–1052.

  • [14] Seungeun Chung and Injong Rhee. 2016. vTrack: virtual trackpad interface using mm-level sound source localization for mobile interaction. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. ACM, 41–44.

  • [15] Amit Das, Ivan Tashev, and Shoaib Mohammed. 2017. Ultrasound based gesture recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on. IEEE, 406–410.

  • [16] Lucas Davi, Alexandra Dmitrienko, Ahmad-Reza Sadeghi, and Marcel Winandy. 2010. Privilege escalation attacks on android. In international conference on Information security. Springer, 346–360.

  • [17] William Enck, Peter Gilbert, Seungyeop Han, Vasant Tendulkar, Byung-Gon Chun, Landon P Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N Sheth. 2014. TaintDroid: an information-flow tracking system for realtime privacy monitoring on smartphones. ACM Transactions on Computer Systems (TOCS) 32, 2 (2014), 5.

  • [18] Viktor Erdélyi, Trung-Kien Le, Bobby Bhattacharjee, Peter Druschel, and Nobutaka Ono. 2018. Sonoloc: Scalable positioning of commodity mobile devices. (2018).

  • [19] Angelo Farina. 2007. Advancements in impulse response measurements by sine sweeps. In Audio Engineering Society Convention 122. Audio Engineering Society.

  • [20] Kassem Fawaz, Kyu-Han Kim, and Kang G Shin. 2016. Protecting Privacy of BLE Device Users.. In USENIX Security Symposium. 1205–1221.

  • [21] Biying Fu, Dinesh Vaithyalingam Gangatharan, Arjan Kuijper, Florian Kirchbuchner, and Andreas Braun. 2017. Exercise Monitoring On Consumer Smart Phones Using Ultrasonic Sensing. In Proceedings of the 4th international Workshop on Sensor-based Activity Recognition and Interaction. ACM, 9.

  • [22] Biying Fu, Jakob Karolus, Tobias Grosse-Puppendahl, Jonathan Hermann, and Arjan Kuijper. 2015. Opportunities for activity recognition using ultrasound doppler sensing on unmodified mobile phones. In Proceedings of the 2nd international Workshop on Sensor-based Activity Recognition and Interaction. ACM, 8.

  • [23] Shyamnath Gollakota, Haitham Hassanieh, Benjamin Ransford, Dina Katabi, and Kevin Fu. 2011. They can hear your heartbeats: non-invasive security for implantable medical devices. In ACM SIGCOMM Computer Communication Review, Vol. 41. ACM, 2–13.

  • [24] Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Soundwave: using the doppler effect to sense gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1911–1914.

  • [25] Chris Harrison, Julia Schwarz, and Scott E Hudson. 2011. TapSense: enhancing finger interaction on touch surfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 627–636.

  • [26] Wenchao Huang, Yan Xiong, Xiang-Yang Li, Hao Lin, Xufei Mao, Panlong Yang, and Yunhao Liu. 2014. Shake and walk: Acoustic direction finding and fine-grained indoor localization using smartphones. In INFOCOM, 2014 Proceedings IEEE. IEEE, 370–378.

  • [27] Yu Seung Kim and Patrick Tague. 2014. Proximity-based Wireless Access Control Through Considerate Jamming. In Proceedings of the ACM MobiCom Workshop on Security and Privacy in Mobile Environments (SPME ’14). 19–24. https://doi.org/10.1145/2646584.2646588

  • [28] Tao Li, Yimin Chen, Jingchao Sun, Xiaocong Jin, and Yanchao Zhang. 2016. iLock: Immediate and Automatic Locking of Mobile Devices Against Data Theft. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS ’16). ACM, New York, NY, USA, 933–944. https://doi.org/10.1145/2976749.2978294

  • [29] Jian Liu, Yan Wang, Gorkem Kar, Yingying Chen, Jie Yang, and Marco Gruteser. 2015. Snooping keystrokes with mmlevel audio ranging on a single phone. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 142–154.

  • [30] Wenguang Mao, Jian He, and Lili Qiu. 2016. CAT: high-precision acoustic motion tracking. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 69–81.

  • [31] Ivan Martinovic, Paul Pichota, and Jens B. Schmitt. 2009. Jamming for Good: A Fresh Approach to Authentic Communication in WSNs. In Proceedings of the Second ACM Conference on Wireless Network Security (WiSec ’09). 161–168. https://doi.org/10.1145/1514274.1514298

  • [32] Raghuraman Mudumbai, D Richard Brown Iii, Upamanyu Madhow, and H Vincent Poor. 2009. Distributed transmit beamforming: challenges and recent progress. IEEE Communications Magazine 47, 2 (2009), 102–110.

  • [33] Rajalakshmi Nandakumar, Shyamnath Gollakota, and Nathaniel Watson. 2015. Contactless sleep apnea detection on smartphones. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 45–57.

  • [34] Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. Fingerio: Using active sonar for fine-grained finger tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1515–1525.

  • [35] Rajalakshmi Nandakumar, Alex Takakuwa, Tadayoshi Kohno, and Shyamnath Gollakota. 2017. CovertBand: Activity Information Leakage Using Music. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 87 (Sept. 2017), 24 pages. https://doi.org/10.1145/3131897

  • [36] Chunyi Peng, Guobin Shen, Yongguang Zhang, Yanlin Li, and Kun Tan. 2007. Beepbeep: a high accuracy acoustic ranging system using cots mobile devices. In Proceedings of the 5th international conference on Embedded networked sensor systems. ACM, 1–14.

  • [37] Kun Qian, Chenshu Wu, Fu Xiao, Yue Zheng, Yi Zhang, Zheng Yang, and Yunhao Liu. 2018. Acousticcardiogram: Monitoring Heartbeats using Acoustic Signals on Smart Devices. In Proceedings of the IEEE International Conference on Computer Communications (Infocom ’18).

  • [38] Yue Qiao, Ouyang Zhang, Wenjie Zhou, Kannan Srinivasan, and Anish Arora. 2016. PhyCloak: Obfuscating Sensing from Communication Signals.. In USENIX Annual Technical Conference.

  • [39] Hanif Rahbari and Marwan Krunz. 2014. Friendly CryptoJam: A Mechanism for Securing Physical-layer Attributes. In Proceedings of the 2014 ACM Conference on Security and Privacy in Wireless & Mobile Networks (WiSec ’14). 129–140. https://doi.org/10.1145/2627393.2627415

  • [40] Nirupam Roy, Haitham Hassanieh, and Romit Roy Choudhury. 2017. Backdoor: Making microphones hear inaudible sounds. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 2–14.

  • [41] Nirupam Roy, Sheng Shen, Haitham Hassanieh, and Romit Roy Choudhury. 2018. Inaudible Voice Commands: The Long-Range Attack and Defense. In 15th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 18). 547–560.

  • [42] Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: enabling fine-grained hand gesture detection by decoding echo signal. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 474–485.

  • [43] Roman Schlegel, Kehuan Zhang, Xiao-yong Zhou, Mehool Intwala, Apu Kapadia, and XiaoFeng Wang. 2011. Sound-comber: A Stealthy and Context-Aware Sound Trojan for Smartphones.. In NDSS, Vol. 11. 17–33.

  • [44] Matthias Schulz, Francesco Gringoli, Daniel Steinmetzer, Michael Koch, and Matthias Hollick. 2017. Massive Reactive Smartphone-based Jamming Using Arbitrary Waveforms and Adaptive Power Control. In Proceedings of the 10th ACM Conference on Security and Privacy in Wireless and Mobile Networks (WiSec ’17). 111–121. https://doi.org/10.1145/3098243.3098253

  • [45] W. Shen, P. Ning, X. He, and H. Dai. 2013. Ally Friendly Jamming: How to Jam Your Enemy and Maintain Your Own Wireless Connectivity at the Same Time. In 2013 IEEE Symposium on Security and Privacy. 174–188. https://doi.org/10.1109/SP.2013.22

  • [46] Adam Smith, Hari Balakrishnan, Michel Goraczko, and Nissanka Priyantha. 2004. Tracking moving devices with the cricket location system. In Proceedings of the 2nd international conference on Mobile systems, applications, and services. ACM, 190–202.

  • [47] Zheng Sun, Aveek Purohit, Raja Bose, and Pei Zhang. 2013. Spartacus: spatially-aware interaction for mobile devices through energy-efficient audio sensing. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services. ACM, 263–276.

  • [48] Sanjib Sur, Teng Wei, and Xinyu Zhang. 2014. Autodirective audio capturing through a synchronized smartphone array. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. ACM, 28–41.

  • [49] Nils Ole Tippenhauer, Luka Malisa, Aanjhan Ranganathan, and Srdjan Capkun. 2013. On limitations of friendly jamming for confidentiality. In Security and Privacy (SP), 2013 IEEE Symposium on. IEEE, 160–173.

  • [50] Barry D Van Veen and Kevin M Buckley. 1988. Beamforming: A versatile approach to spatial filtering. IEEE assp magazine 5, 2 (1988), 4–24.

  • [51] Junyi Wang, Zhou Lan, Chin-Sean Sum, Chang-Woo Pyo, Jing Gao, Tuncer Baykas, Azizur Rahman, Ryuhei Funada, Fumihide Kojima, Ismail Lakkis, et al. 2009. Beamforming codebook design and performance evaluation for 60GHz wideband WPANs. In Vehicular Technology Conference Fall (VTC 2009-Fall), 2009 IEEE 70th. IEEE, 1–6.

  • [52] Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RFIDraw: virtual touch screen in the air using RF signals. In ACM SIGCOMM Computer Communication Review, Vol. 44. ACM, 235–246.

  • [53] Tianben Wang, Daqing Zhang, Yuanqing Zheng, Tao Gu, Xingshe Zhou, and Bernadette Dorizzi. 2018. C-FMCW Based Contactless Respiration Detection Using Acoustic Signal. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article 170 (Jan. 2018), 20 pages. https://doi.org/10.1145/3161188

  • [54] Wei Wang, Alex X Liu, and Ke Sun. 2016. Device-free gesture tracking using acoustic signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 82–94.

  • [55] Xuyu Wang, Runze Huang, and Shiwen Mao. 2017. Sonar-Beat: Sonar phase for breathing beat monitoring with smart-phones. In Computer Communication and Networks (ICCCN), 2017 26th International Conference on. IEEE, 1–8.

  • [56] Hiroki Watanabe, Tsutomu Terada, and Masahiko Tsukamoto. 2013. Ultrasound-based movement sensing, gesture-, and context-recognition. In Proceedings of the 2013 International Symposium on Wearable Computers. ACM, 57–64.

  • [57] Ryan West. 2008. The psychology of security. Commun. ACM 51, 4 (2008), 34–40.

  • [58] Matthias Wilhelm, Ivan Martinovic, Jens B. Schmitt, and Vincent Lenders. 2011. WiFire: A Firewall for Wireless Networks. In Proceedings of the ACM SIGCOMM 2011 Conference (SIGCOMM ’11). 456–457. https://doi.org/10.1145/2018436.2018518

  • [59] Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 15–29.

  • [60] Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-Grained Acoustic-based Device-Free Tracking. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys ’17). 15–28. https://doi.org/10.1145/3081333.3081356

  • [61] B. Zhang, Q. Zhan, S. Chen, M. Li, K. Ren, C. Wang, and D. Ma. 2014. ssrP riW hisper : Enabling Keyless Secure Acoustic Communication for Smartphones. IEEE Internet of Things Journal 1, 1 (Feb 2014), 33–45. https://doi.org/10.1109/JIOT.2014.2297998

  • [62] Guoming Zhang, Chen Yan, Xiaoyu Ji, Tianchen Zhang, Taimin Zhang, and Wenyuan Xu. 2017. DolphinAttack: Inaudible voice commands. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 103–117.

OPEN ACCESS

Journal + Issues

Search