Archives 25 April 2016

“The general idea is to find datasets so enormous that they can reveal patterns invisible to conventional inquiry… But there’s a problem: It’s tempting to think that with such an incredible volume of data behind them, studies relying on big data couldn’t be wrong. But the bigness of the data can imbue the results with a false sense of certainty. Many of them are probably bogus — and the reasons why should give us pause about any research that blindly trusts big data.”

For example, Google’s database of scanned books represents 4% of all books ever published, but in this data set, “The Lord of the Rings gets no more influence than, say, Witchcraft Persecutions in Bavaria.” And the name Lanny appears to be one of the most common in early-20th century fiction — solely because Upton Sinclair published 11 different novels about a character named Lanny Budd.

The problem seems to be skewed data and misinterpretation. (The article points to the failure of Google Flu Trends, which it turns out “was largely predicting winter”.) The article’s conclusion? “Rather than succumb to ‘big data hubris,’ the rest of us would do well to keep our sceptic hats on — even when someone points to billions of words.”

“Alphabet’s [Google] executive chairman, Eric Schmidt, recently joined a Department of Defense advisory panel. Facebook recently hired a former director at the U.S. military’s research lab, Darpa. Uber employs Barack Obama’s former campaign manager David Plouffe and Amazon.com tapped his former spokesman Jay Carney. Google, Facebook, Uber and Apple collectively employ a couple of dozen former analysts for America’s spy agencies, who openly list their resumes on LinkedIn.

These connections are neither new nor secret. But the fact they are so accepted illustrates how tech’s leaders — even amid current fights over encryption and surveillance — are still seen as mostly U.S. firms that back up American values. Christopher Soghoian, a technologist with the American Civil Liberties Union, said low-level employees’ government connections matter less than leading executives’ ties to government. For instance, at least a dozen Google engineers have worked at the NSA, according to publicly available records on LinkedIn. And, this being Silicon Valley, not everyone who worked for a spy agency advertises that on LinkedIn. Soghoian, a vocal critic of mass surveillance, said Google hiring an ex-hacker for the NSA to work on security doesn’t really bother him. “But Eric Schmidt having a close relationship with the White House does…”