Listen Only When Spoken To: Interpersonal Communication Cues as Smart Speaker Privacy Controls

Abraham Mhaidli 1 , Manikandan Kandadai Venkatesh 2 , Yixin Zou 3  and Florian Schaub 4
  • 1 University of Michigan School of Information,
  • 2 University of Michigan School of Information,
  • 3 University of Michigan School of Information,
  • 4 University of Michigan School of Information,

Abstract

Internet of Things and smart home technologies pose challenges for providing effective privacy controls to users, as smart devices lack both traditional screens and input interfaces. We investigate the potential for leveraging interpersonal communication cues as privacy controls in the IoT context, in particular for smart speakers. We propose privacy controls based on two kinds of interpersonal communication cues – gaze direction and voice volume level – that only selectively activate a smart speaker’s microphone or voice recognition when the device is being addressed, in order to avoid constant listening and speech recognition by the smart speaker microphones and reduce false device activation. We implement these privacy controls in a smart speaker prototype and assess their feasibility, usability and user perception in two lab studies. We find that privacy controls based on interpersonal communication cues are practical, do not impair the smart speaker’s functionality, and can be easily used by users to selectively mute the microphone. Based on our findings, we discuss insights regarding the use of interpersonal cues as privacy controls for smart speakers and other IoT devices.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • [1] Noura Abdi, Kopo M. Ramokapane, and Jose M. Such. More than smart speakers: Security and privacy perceptions of smart home personal assistants. In Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019), Santa Clara, CA, August 2019. USENIX Association.

  • [2] Mark Altosaar, Roel Vertegaal, Changuk Sohn, and Daniel Cheng. Auraorb: Using social awareness cues in the design of progressive notification appliances. In Proceedings of the 18th Australia Conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments, OZCHI ’06, pages 159–166, New York, NY, USA, 2006. ACM.

  • [3] Tawfiq Ammari, Jofish Kaye, Janice Y. Tsai, and Frank Bentley. Music, search, and iot: How people (really) use voice assistants. ACM Trans. Comput.-Hum. Interact., 26(3):17:1–17:28, April 2019.

  • [4] Newgenn Apps. 13 IoT Statistics Defining the Future of Internet of Things. Newgenn Apps, https://www.newgenapps.com/blog/iot-statistics-internet-of-things-future-research-data, 2018. Online; accessed 09/01/2018.

  • [5] Richard Baguely and Colin McDonald. “Appliance Science: Alexa, how does Alexa work? The science of the Amazon Echo”. CNET, https://www.cnet.com/news/appliance-science-alexa-how-does-alexa-work-the-science-of-amazons-echo/, 2016. Online; accessed 02/26/2019.

  • [6] Aaron Bangor, Philip Kortum, and James Miller. Determining what individual sus scores mean: Adding an adjective rating scale. J. Usability Studies, 4(3):114–123, May 2009.

  • [7] Brian Barret. What Amazon Echo and Google Home do with your voice data. Wired, https://www.wired.com/story/amazon-echo-and-google-home-voice-data-delete/, November 2017. Online: accessed 05/08/2018.

  • [8] John Brooke. SUS – a quick and dirty usability scale. In Usability evaluation in industry. Taylor & Francis, London, 1996.

  • [9] Joseph Bugeja, Andreas Jacobsson, and Paul Davidsson. On privacy and security challenges in smart connected homes. In Intelligence and Security Informatics Conference (EISIC), pages 172–175. IEEE, 2016.

  • [10] Daniel J Butler, Justin Huang, Franziska Roesner, and Maya Cakmak. The privacy-utility tradeoff for remotely teleoperated robots. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, pages 27–34. ACM, 2015.

  • [11] Varun Chandrasekaran, Kassem Fawaz, Bilge Mutlu, and Suman Banerjee. Characterizing privacy perceptions of voice assistants: A technology probe study. arXiv preprint, arXiv:1812.00263, 2018.

  • [12] Goodwin Charles. Conversational organization: Interaction between speakers and hearers. New York, Academic Press, 1981.

  • [13] Peng Cheng, Ibrahim Ethem Bagci, Jeff Yan, and Utz Roedig. Smart speaker privacy control-acoustic tagging for personal voice assistants. In IEEE Workshop on the Internet of Safe Things (SafeThings 2019), 2019.

  • [14] Eun Kyoung Choe, Sunny Consolvo, Jaeyeon Jung, Beverly Harrison, and Julie A Kientz. Living in a glass house: a survey of private moments in the home. In Proceedings of the 13th international conference on Ubiquitous computing, pages 41–44. ACM, 2011.

  • [15] Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint, arXiv:1805.10190, 2018.

  • [16] Ry Crist and Andrew Gebhart. Everything you need to know about the Amazon Echo. CNET, https://www.cnet.com/how-to/amazon-echo-alexa-everything-you-need-to-know/, 2017. Online; accessed 02/26/2019.

  • [17] Mark L Daly and Knapp John Augustine. Handbook of interpersonal communication. Sage, 2002.

  • [18] Tamara Denning, Cynthia Matuszek, Karl Koscher, Joshua R Smith, and Tadayoshi Kohno. A spotlight on security and privacy risks with future household robots: attacks and lessons. In Proceedings of the 11th international conference on Ubiquitous computing, pages 105–114. ACM, 2009.

  • [19] Joseph A DeVito. Interpersonal communication. New York: Longman Inc, 2007.

  • [20] Nora A Draper and Joseph Turow. The corporate cultivation of digital resignation. New Media & Society, 21(8):1824–1839, 2019.

  • [21] Jide S. Edu, Jose M. Such, and Guillermo Suarez-Tangil. Smart home personal assistants: A security and privacy review. arXiv preprint, arXiv:1903.05593, 2019.

  • [22] Serge Egelman, Raghudeep Kannavara, and Richard Chow. Is this thing on?: Crowdsourcing privacy indicators for ubiquitous sensing platforms. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1669–1678. ACM, 2015.

  • [23] Stefano Fasciani. Voice Controlled interface for Digital Musical Instrument. PhD thesis, 2014. Online; accessed 05/09/2018.

  • [24] Brian Feldman. “like everyone else, amazon’s alexa is laughing at you”. New York Magazine, http://nymag.com/intelligencer/2018/03/amazon-alexa-is-laughing-atyou.html, 2018. Online; accessed 02/26/2019.

  • [25] Huan Feng, Kassem Fawaz, and Kang G Shin. Continuous authentication for voice assistants. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking, pages 343–355. ACM, 2017.

  • [26] Catherine S Fichten, Vicki Tagalakis, Darlene Judd, John Wright, and Rhonda Amsel. Verbal and nonverbal communication cues in daily conversations and dating. The Journal of Social Psychology, 132(6):751–769, 1992.

  • [27] Nathaniel Fruchter and Ilaria Liccardi. Consumer attitudes towards privacy and security in home assistants. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 2018.

  • [28] Chuhan Gao, Varun Chandrasekaran, Kassem Fawaz, and Suman Banerjee. Traversing the quagmire that is privacy in your smart home. In Proceedings of the 2018 Workshop on IoT Security and Privacy, pages 22–28. ACM, 2018.

  • [29] Tom Gross. Ambient interfaces in a web-based theatre of work. In Proceedings of the 10th Euromicro Workshop on Parallel, Distributed and Network-based Processing, pages 55–62. IEEE, 2002.

  • [30] Susumu Harada, Jacob O Wobbrock, and James A Landay. Voice games: investigation into the use of non-speech voice input for making computer games more accessible. In IFIP Conference on Human-Computer Interaction, pages 11–29. Springer, 2011.

  • [31] Richard Jones. Communication in the real world: An introduction to communication studies. The Saylor Foundation, 2013.

  • [32] Bjorn Karmann. Project alias. https://bjoernkarmann.dk/project_alias, 2018. Online; accessed 08/22/2019.

  • [33] Jacob Kastrenakes. Amazon now lets you tell Alexa to delete your voice recordings. The Verge, https://www.theverge.com/2019/5/29/18644027/amazon-alexa-delete-voice-recordings-command-privacy-hub, 2019. Online; accessed 08/28/2019.

  • [34] Bastian Könings, Florian Schaub, and Michael Weber. Privacy and trust in ambient intelligent environments. In Next Generation Intelligent Environments, pages 133–164. Springer, 2016.

  • [35] Saadi Lahlou, Marc Langheinrich, and Carsten Röcker. Privacy and trust issues with invisible computers. Commun. ACM, 48(3):59–60, March 2005.

  • [36] Marc Langheinrich and Florian Schaub. Privacy in mobile and pervasive computing. Synthesis Lectures on Mobile and Pervasive Computing, 10(1):1–139, 2018.

  • [37] Josephine Lau, Benjamin Zimmerman, and Florian Schaub. Alexa, are you listening?: Privacy perceptions, concerns and privacy-seeking behaviors with smart speakers. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):102, 2018.

  • [38] Hosub Lee and Alfred Kobsa. Privacy preference modeling and prediction in a simulated campuswide iot environment. In 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 276–285. IEEE, 2017.

  • [39] James R. Lewis. The system usability scale: Past, present, and future. International Journal of Human–Computer Interaction, 34(7):577–590, 2018.

  • [40] Yuting Liao, Jessica Vitak, Priya Kumar, Michael Zimmer, and Katherine Kritikos. Understanding the role of privacy and trust in intelligent personal assistant adoption. In International Conference on Information, pages 102–113. Springer, 2019.

  • [41] Nathan Malkin, Joe Deatrick, Allen Tong, Primal Wijesekera, Serge Egelman, and David Wagner. Privacy attitudes of smart speaker users. Proceedings on Privacy Enhancing Technologies, 2019(4):250–271, 2019.

  • [42] Aadil Mamuji, Roel Vertegaal, J Shell, Thanh Pham, and Changuk Sohn. Auralamp: Contextual speech recognition in an eye contact sensing light appliance. In Extended abstracts of the ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), 2003.

  • [43] Zied Mani and Ines Chouk. Drivers of consumers’ resistance to smart products. Journal of Marketing Management, 33(1-2):76–97, 2017.

  • [44] Martin, Taylor. 12 ways to use Alexa in the kitchen. CNET, https://www.cnet.com/how-to/amazon-echo-ways-to-usealexa-in-the-kitchen/, November 2017. Online; accessed 05/05/2018.

  • [45] Evelyn McClave, Helen Kim, Rita Tamer, and Milo Mileff. Head movements in the context of speech in arabic, bulgarian, korean, and african-american vernacular english. Gesture, 7(3):343–390, 2007.

  • [46] Emily McReynolds, Sarah Hubbard, Timothy Lau, Aditya Saraf, Maya Cakmak, and Franziska Roesner. Toys that listen: A study of parents, children, and internet-connected toys. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, pages 5197–5207, New York, NY, USA, 2017. ACM.

  • [47] Pardis Emami Naeini, Sruti Bhagavatula, Hana Habib, Martin Degeling, Lujo Bauer, Lorrie Cranor, and Norman Sadeh. Privacy expectations and preferences in an iot world. In Symposium on Usable Privacy and Security (SOUPS), 2017.

  • [48] Kayako Nakagawa, Masahiro Shiomi, Kazuhiko Shinozawa, Reo Matsumura, Hiroshi Ishiguro, and Norihiro Hagita. Effect of robot’s whispering behavior on people’s motivation. International Journal of Social Robotics, 5(1):5–16, Jan 2013.

  • [49] Michael Natale. Convergence of mean vocal intensity in dyadic communication as a function of social desirability. Journal of Personality and Social Psychology, 32(5):790, 1975.

  • [50] David H Nguyen and Elizabeth D Mynatt. Privacy mirrors: understanding and shaping socio-technical ubiquitous computing systems. Technical report, Georgia Institute of Technology, 2002.

  • [51] Carol AE Nickerson. A note on “a concordance correlation coefficient to evaluate reproducibility”. Biometrics, pages 1503–1507, 1997.

  • [52] Katarzyna Olejnik, Italo Dacosta, Joana Soares Machado, Kévin Huguenin, Mohammad Emtiyaz Khan, and Jean-Pierre Hubaux. Smarper: Context-aware and automatic runtime-permissions for mobile devices. In IEEE Symposium on Security and Privacy, pages 1058–1076. IEEE, 2017.

  • [53] Antti Oulasvirta, Aurora Pihlajamaa, Jukka Perkiö, Debarshi Ray, Taneli Vähäkangas, Tero Hasu, Niklas Vainio, and Petri Myllymäki. Long-term effects of ubiquitous surveillance in the home. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 41–50. ACM, 2012.

  • [54] Richard A. Page and Joseph L. Balloun. The effect of voice volume on the perception of personality. The Journal of Social Psychology, 105(1):65–72, 1978.

  • [55] Rebecca S Portnoff, Linda N Lee, Serge Egelman, Pratyush Mishra, Derek Leung, and David Wagner. Somebody’s watching me?: Assessing the effectiveness of webcam indicator lights. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1649–1658. ACM, 2015.

  • [56] Amanda Purington, Jessie G. Taft, Shruti Sannon, Natalya N. Bazarova, and Samuel Hardman Taylor. “alexa is my new bff”: Social roles, user satisfaction, and person-ification of the amazon echo. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 2853–2859, New York, NY, USA, 2017. ACM.

  • [57] Nirupam Roy, Haitham Hassanieh, and Romit Roy Choudhury. Backdoor: Sounds that a microphone can record, but that humans can’t hear. GetMobile: Mobile Computing and Communications, 21(4):25–29, 2018.

  • [58] Nirupam Roy, Sheng Shen, Haitham Hassanieh, and Romit Roy Choudhury. Inaudible voice commands: The long-range attack and defense. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’18), pages 547–560, 2018.

  • [59] Florian Schaub. Context-adaptive privacy mechanisms. In Aris Gkoulalas-Divanis and Claudio Bettini, editors, Handbook of Mobile Data Privacy, pages 337–372. Springer International Publishing, Cham, 2018.

  • [60] Florian Schaub, Rebecca Balebako, Adam L Durity, and Lorrie Faith Cranor. A design space for effective privacy notices. In Eleventh Symposium On Usable Privacy and Security (SOUPS 2015), pages 1–17, 2015.

  • [61] Florian Schaub, Bastian Könings, and Michael Weber. Context-adaptive privacy: Leveraging context awareness to support privacy decision making. IEEE Pervasive Computing, 14(1):34–43, Jan 2015.

  • [62] Florian Schaub, Bastian Könings, Peter Lang, Björn Wiedersheim, Christian Winkler, and Michael Weber. Prical: context-adaptive privacy in ambient calendar displays. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 499–510. ACM, 2014.

  • [63] Bill Schilit, Norman Adams, and Roy Want. Context-aware computing applications. In First Workshop on Mobile Computing Systems and Applications, pages 85–90. IEEE, 1994.

  • [64] Albrecht Schmidt. Implicit human computer interaction through context. Personal Technologies, 4(2):191–199, Jun 2000.

  • [65] Singleton, Micah. Alexa can now set reminders for you. The Verge, https://www.theverge.com/circuitbreaker/2017/6/1/15724474/alexa-echo-amazon-reminders-named-timers, June 2017. Online; accessed 05/05/2018.

  • [66] Adam J. Sporka, Sri H. Kurniawan, and Pavel Slavík. Acoustic control of mouse pointer. Universal Access in the Information Society, 4(3):237–245, Mar 2006.

  • [67] Blase Ur, Jaeyeon Jung, and Stuart Schechter. Intruders versus intrusiveness: teens’ and parents’ perspectives on home-entryway surveillance. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 129–139. ACM, 2014.

  • [68] Roel Vertegaal, Aadil Mamuji, Changuk Sohn, and Daniel Cheng. Media eyepliances: Using eye tracking for remote control focus selection of appliances. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems, pages 1861–1864, New York, NY, USA, 2005. ACM.

  • [69] Roel Vertegaal, Robert Slagter, Gerrit van der Veer, and Anton Nijholt. Eye gaze patterns in conversations: There is more to conversational agents than meets the eyes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 301–308, New York, NY, USA, 2001. ACM.

  • [70] Mark Weiser. Some computer science issues in ubiquitous computing. Commun. ACM, 36(7):75–84, July 1993.

  • [71] Joan Welkowitz, Stanley Feldstein, Mark Finklestein, and Lawrence Aylesworth. Changes in vocal intensity as a function of interspeaker influence. Perceptual and Motor Skills, 35(3):715–718, 1972.

  • [72] Primal Wijesekera, Arjun Baokar, Lynn Tsai, Joel Reardon, Serge Egelman, David Wagner, and Konstantin Beznosov. The feasibility of dynamically granted permissions: Aligning mobile privacy with user preferences. In IEEE Symposium on Security and Privacy, pages 1077–1093. IEEE, 2017.

  • [73] Danny Wyatt, Tanzeem Choudhury, and Henry Kautz. Capturing spontaneous conversation and social dynamics: A privacy-sensitive data collection effort. In IEEE International Conference on Acoustics, Speech and Signal Processing, volume 4, pages IV–213. IEEE, 2007.

  • [74] Eric Zeng, Shrirang Mare, and Franziska Roesner. End user security & privacy concerns with smart homes. In Symposium on Usable Privacy and Security (SOUPS), 2017.

  • [75] Serena Zheng, Marshini Chetty, and Nick Feamster. User perceptions of privacy in smart homes. arXiv preprint, arXiv:1802.08182, 2018.

OPEN ACCESS

Journal + Issues

Search