Smart devices are listening to more than our words
May 17, 2021

Smart devices are listening to more than our words

Voice recognition technology could be used to learn things about us, like if we're sick or depressed.

Picture this: You’re not feeling so hot and you say to your smart speaker, “Robot, I’m hungry,” and you cough. And the device says, “Would you like a recipe for chicken soup?” And then, “By the way, would you like to order cough drops with one-hour delivery?”

This is the scenario laid out in one of Amazon’s patents. And it shows how voice recognition technology could be used to learn things about us, beyond the words we say to our devices, like whether we’re sick or depressed.

I spoke with Joseph Turow, a communications professor at the University of Pennsylvania, who writes about all of this in his new book “The Voice Catchers: How Marketers Listen In to Exploit Your Feelings, Your Privacy, and Your Wallet.” I asked him if the technology is there yet. The following is an edited transcript of our conversation.

A headshot of Joe Turow, a professor at the University of Pennsylvania and the author of the new book “The Voice Catchers: How Marketers Listen In to Exploit Your Feelings, Your Privacy, and Your Wallet.”
Joe Turow (Photo courtesy Kyle Cassidy)

Joseph Turow: Well, companies believe it’s very sophisticated. They think that by using technologies that look at the sound that you have, and [your] syntax, and sometimes in combination with the actual words you use — they think it helps them sell and make people feel better. How do they know? Where do they get the data from? Amazon’s pretty close lipped about it.

Amy Scott: You argue in the book that voice profiling is a gateway to a whole new era of biometric profiling by marketers, even more than facial recognition. Why is that?

Turow: Yeah, well, facial is, of course, important. The thing about facial, though, is people are worried about it. And it’s not as integrated into the home as voice is becoming. I mean, some people have three, four or five of these devices in their home. They talk to their phones every day. I spoke to new home builders who are integrating them directly into the walls of the home. So these are going to become part of our everyday experiences — “Turn on the light. Turn down the shades.” And as a result, the notion of giving data to companies and then having them evaluate them is going to become more plausible.

Scott: You say we are doing this all the time. But I think a lot of folks are pretty resistant to this technology. What about you? Do you have any of these devices in your house?

Turow: Yeah, well, to be honest, I did it in the beginning because I felt I had to learn about it. You can’t write about this stuff without having it. I have a Google device in my study here and I have a couple of Amazon devices around the house. I’m not too happy with the idea, however. And if I got the sense that they were beginning to go more into profiling me, I would probably shut them off.

A diagram from an Amazon voice recognition patent, showing a conversation between a sick user and a smart speaker. The speaker is able to determine the user is sick from their cough.
An Amazon patent shows how voice recognition technology can learn things about us from the way we talk to a smart device.

Scott: For people who are really creeped out about marketers using our supposed mood to sell us things, are there any positive societal benefits to this technology? For example, someone living alone and a device kind of paying attention to how they’re sounding that day.

A cover image of Joe Turow's new book, "The Voice Catchers: How Marketers Listen In to Exploit Your Feelings, Your Privacy, and Your Wallet."

Turow: There are companies that are looking into trying to see whether voice can be a predictor of someone’s health. Looking, for example, at Alzheimer’s disease — can a person’s voice predict whether a person is becoming more and more likely to have Alzheimer’s? And these things are really important. My suggestion in the book: This stuff should be banned from marketing. To me, that’s a red line. Think of the dangers of a society where immigrants, for example, are listened to not just by what they say, but for their sounds of supposedly where they come from. There are a lot of really scary things that one can spin off in all of this, and we have to be scared of that.

Scott: What do you want both consumers and, I should say, regulators to take away from your book?

Turow: For consumers, I want us to be wary, to realize that this is a kind of seductive surveillance. There are so many historical precedents to the idea that we can get used to almost anything. I mean, we can get used to things because they seem like they’re terrific. And down the line, we see negative aspects. So I say we have to be wary. When it comes to regulators, my point is that regulators have not been so great at actually regulating a lot of our marketing technologies. And I think when it comes to something like voice and other biometrics, sometimes it’s OK to say, “No, we’re not going to allow it.”

Scott: Can consumers opt out of this? Is there a way to have the convenience of these devices without the voice analysis?

Turow: Reading the privacy policies of a lot of these companies, it’s extremely difficult to figure out exactly what they’re doing.

Related links: More insight from Amy Scott

Devices aren’t just listening to us in our homes. Ford just announced it will soon offer a three-year complimentary subscription to Amazon’s Alexa in hundreds of thousands of its cars and trucks. After those three years, drivers have to pay for it. Alexa was already available in many models via smartphone app, but now it will be embedded in the vehicle operating system. It’s not just Amazon; Google and Apple have also been forging partnerships with carmakers.

Our producer Stephanie Hughes tried to see if Alexa would actually try to sell her cough drops, like Amazon’s patent imagined. She asked a series of questions, including one that was kind of leading: “I could use some cough drops.” Alexa’s first response was to recite the Wikipedia definition of a throat lozenge. Stephanie had to specifically ask for cough drops before Alexa served up some links on Amazon. So, you know, work in progress.

Finally, Joe Turow talked about how people are worried about facial recognition. Well, The Washington Post had a good piece on one reason to worry. Upload a photo to the face-search site PimEyes, and it’ll scan through more than 900 million images in less than a second to find matches. What makes PimEyes stand out isn’t necessarily its technology, but that anyone can use it — not just law enforcement. Including stalkers. You can block photos from showing up in a search, but it’ll cost you $80 a month.

The future of this podcast starts with you.

Every day, the “Marketplace Tech” team demystifies the digital economy with stories that explore more than just Big Tech. We’re committed to covering topics that matter to you and the world around us, diving deep into how technology intersects with climate change, inequity, and disinformation.

As part of a nonprofit newsroom, we’re counting on listeners like you to keep this public service paywall-free and available to all.

Support “Marketplace Tech” in any amount today and become a partner in our mission.

The team

Molly Wood Host
Michael Lipkin Senior Producer
Stephanie Hughes Producer
Daniel Shin Producer
Jesús Alvarado Associate Producer