Power Marketplace’s public service journalism 💙 Give Now
Your smart speaker may capture your voice-activated fails
Apr 12, 2019

Your smart speaker may capture your voice-activated fails

Are you ever really alone when talking to your smart speaker? It turns out there might just be someone on the other side listening to you as part of the platforms' quality assurance measures.

Are you ever really alone when talking to your smart speaker? It turns out there might just be someone on the other side listening to you, especially when your voice commands go haywire. That’s according to Anthonio Pettit, a writer for Microsoft who’s worked on voice platforms like Microsoft’s Cortana, Samsung’s Bixby and Amazon’s Echo. He was even part of a team responsible for listening in on the times when you might have teased your Echo. (“Alexa, get me a beer,” is something he’s heard a lot.)

Pettit was part of a recent episode of KUOW’s “Prime(d)” podcast on smart speaker technology, including the “quality assurance” bin where accidental and pear-shaped voice commands wind up. “Prime(d)” co-host Joshua McNichols talked with Marketplace Tech host Molly Wood about his reporting on the range of audio Amazon collects. And if you were wondering, it’s pretty broad. The following is an edited transcript of their conversation.

Joshua McNichols: Those sort of accidental moments where maybe a cat knocked over a vase and it made a sound that sort of sounded like “Alexa” when the vase fell — those recordings go into a bucket called quality assurance, where somebody is listening to them and classifying “Why did this not turn out as we had planned?” We’ve written all these pairs of questions and answers like, “If humans say this, then Alexa should return this answer,” but some of those just aren’t classifiable, and they have to judge what went wrong.

Molly Wood: How did they, before we dig into the implications of that, how do they do that? I mean, they really just apply human logic to it? [They] listen to it and think “Oh, OK, this doesn’t match the script”? Or is there also some engineering that goes into it?

McNichols: Exactly. Well, I’m sure there’s algorithms that help them with some of them, but a lot of them go to a human, and a human listens to them because humans are still better at interpreting human speech in many ways than artificial intelligence.

Wood: How hard would it be for one of these engineers, one of these people in the quality assurance department, to figure out who the user is?

McNichols: Well, the information that comes to them is anonymized. But who knows what might be recorded in that time period? It could be some information that could be identifiable. And if somebody managed to attack one of these companies and get these recordings or something, then that’s another thing. But we don’t have any evidence that that has happened yet.

Wood: Well, because even anonymized, there have been plenty of incidents throughout technology’s history of anonymized data being pretty easily tracked back to an IP address.

McNichols: Yeah, that’s right. And we know that data that may not be easily ascribed to you at this time at some point in the future may be more easily correlated with you by combining different data sets that creates associations that aren’t in one set of data alone.

Related links: more insight from Molly Wood

Elsewhere in Amazon news: CEO Jeff Bezos put out a letter to shareholders this week in which he talked about the company’s failures and said that the development of the Echo smart speaker started around the same time as the company’s biggest financial failure — the Amazon Fire phone. Yeah, remember that?

It made me think, because at the time it seemed really obvious that Amazon might need a phone to complete its ecosystem since most of our shopping, browsing and ad delivery was becoming mobile. But instead, smart speakers are among the fastest growing categories in consumer electronics.

On Thursday, Credit Suisse assigned a neutral rating to Apple and said it expected the company’s smartphone sales to decline by double digits because, well, phones have peaked.

Bezos’ letter said the company will have to keep experimenting with potentially multibillion-dollar failures in order to keep innovating.

One other note on Amazon.

The company has been criticized for how it treats its workers at fulfillment centers and warehouses: low pay, long hours and really not fun working conditions — including people peeing in bottles because they’re afraid to leave their desks.

Well, on Thursday, pilots at Amazon Air protested over low pay and what they said are terrible working conditions. They also feel the pressure to keep supply chain costs low for Amazon, creating bad conditions for cargo pilots across that industry.

That’s maybe not the ideal way to finance the multibillion-dollar experiments.

The future of this podcast starts with you.

Every day, the “Marketplace Tech” team demystifies the digital economy with stories that explore more than just Big Tech. We’re committed to covering topics that matter to you and the world around us, diving deep into how technology intersects with climate change, inequity, and disinformation.

As part of a nonprofit newsroom, we’re counting on listeners like you to keep this public service paywall-free and available to all.

Support “Marketplace Tech” in any amount today and become a partner in our mission.

The team