Why PRISM is powered by private-sector technology

This undated photo provided by the National Security Agency (NSA) shows its headquarters in Fort Meade, Md.

We continue to learn more about the scope of the U.S. government’s data collection efforts. According to The Wall Street Journal, the NSA, National Security Agency, has relied on technology developed in the private sector to sift through information it’s collected. 

The government’s been using something called Apache Hadoop, The Journal reports, an open-source software runs circles around what the government’s got in house. The people who developed Hadoop call it software for “scalable, distributed computing.”

Garth Gibson, a computer scientist at Carnegie Melon, says it’s used “to process a huge amount of data in a relatively short period of time using a lot of computing resources.”

Hadoop takes the data and breaks it into smaller pieces, so thousands of computers can split up the workload. It’s part of Yahoo!’s search engine, it’s behind Facebook’s social network and now the government can use it for surveillance and to find patterns.

Amy Apon chairs the division of computer science at Clemson University. She says to think of Hadoop like a gas station. “They have rows and rows of gas pumps, and lots of cars can pull in and get gas at the same time.” More computers means more efficiency.

The government relies on Hadoop and systems like it because they work well, they’ve been improved over time, and they’re not that expensive.

“The government wants to use the most cost-effective technology it can to accomplish its goals,” says tech analyst Carl Howe, with the Yankee Group. “I mean, it’s no different than any other business.”

But the government is not leading the way here. According to Ken Birman, the N. Rama Rao Professor of Computer Science at Cornell University, companies that have developed “distributed computing” programs like Hadoop have an edge. They have used open-source to collaborate, and they have outspent the government on innovation.

“All that investment has created a very powerful technology base,” he says. One the government just can’t match.

About the author

David Gura is a reporter for Marketplace, based in the Washington, D.C. bureau.
Log in to post2 Comments

Like frmiller, I found this story very odd and disjoint. The nsa isn't just using open source systems like Hadoop, they're releasing extremely interesting software in the space (One example: http://accumulo.apache.org/) , so I say they're one of the leading innovators in the big data space (open source or non-open source)

I'm surprised you guys didn't talk further with cloudera or one of the other hadoop companies who would be happy to provide some education on the matter.

I don't understand the point of this article. It is critical of the government for not having created something as good as Hadoop. So when private companies use Hadoop, it is smart and cool, but when the government uses it, it shows how inadequate they are? If the government did spend the money developing its own system, I'm sure it would be criticized for wasting money and not using what's available in the private sector. Again, what is the point?

With Generous Support From...