The NSA’s Big Data Problem

by Bryan Walsh, Time Magazine

It’s still unclear exactly how the National Security Agency (NSA) is carrying out digital surveillance on us. But we know one thing for sure: the government is collecting a whole lot of data. And privacy concerns aside, the real challenge for the NSA is not so much collecting that information but figuring out how to use it to help keep the country safe–and doing it against the clock while minimizing mistakes. “This is a big-data challenge,” says Viktor Mayer-Schönberger, the Oxford Internet Institute’s professor of Internet governance and regulation. “You have lots and lots of noise with a potential signal buried inside, but it’s hard to differentiate the two.”

With the national-security establishment still a black box–albeit a leaking one–it’s worth looking at how private companies are grappling with those big-data challenges. From logistics firms trying to keep millions of packages moving on schedule to airlines trying to predict delays, businesses are working to make sense of their own vast data sets. And they’re turning to specialized data-consulting firms that have the expertise to pull the signal from the noise through something called complex event processing. “There’s so much data flowing around now and a huge need to analyze it,” says Matt Quinn, the CTO of Silicon Valley enterprise-software firm the Information Bus Co. (TIBCO). “That’s led to companies like us developing the technology to take advantage of it.”

How does it work? Say you’re a financial firm looking to detect fraud. You may have as many as 300,000 transactions per second, each of which can be considered an event. And each of those events has countless data points that go along with it–the size of the transaction, its type, its location. Out of that overflowing stream of data, complex event processing tries to pull out recognizable patterns that can alert you to aberrations. And it has to happen in near real time–a fraud alert that goes off days after the theft will do little to prevent loss. “What you try to do is correlate those events into something larger,” says Quinn.

……. What’s different is the sheer scale of the data that’s being collected in a connected world–and the computing power available to mine that information.  ……..   Article

DCL: Of course anybody with a pea for a brain suspected this was going on. Certainly the terrorists did. In fact, the only people who seemed surprised were the press. I don’t mind if NSA spies on my Internet transactions. What does worry me is how secure is the data they collect and deposit somewhere.  If the Russian mafia can get their hands on that …. then I’m really worried. But our press doesn’t seem to mention that problem with government spying.