August 4, 2013

Nate Silver and Small Data

Colby Cosh writes on prognosticator Nate Silver's move from the New York Times to ESPN
It is amusing to me, and I bet it is pretty amusing to him, that Nate Silver has become so venerated even as the popular understanding of what he does remains so impoverished. ... Silver went to the trouble of writing a whole book about what he does, but it is in the nature of such books to be bought more than they are read and read more than they are understood. The proof is that Silver’s name has become inextricably linked with one of the oppressive buzzwords of 2013: “big data”. 
People know “big data” has something to do with statistics, and, hey, who’s the most famous statistics guy on the planet? I’ll let you in on a bluffer’s secret: what “big data” denotes are massive realtime streams of ever-changing information, such as web traffic or the Twitter “firehose”, that can potentially be bent to commercial purposes using powerful and bleeding-edge computational techniques. Silver has always worked exclusively, at least in public, with what might be called “small data”: sets of a few hundred political polls or ballplayers’ statistical lines. He is living proof that there is money to be made applying 50- or 100-year old statistical nostrums and exploratory techniques to such small data. 

As for why Silver moved from a newspaper to a TV network, why wouldn't an ambitious young media figure want to be with a powerful TV network? What makes you money is getting your name and face recognized, and TV is a lot better for the latter.

5 comments:

  1. The scary thing is look who is on TV now.

    ReplyDelete
  2. I can never understand Silver's system. It seems like he justifies his own views by oversampling the data, and hopes that he's right so the left can make fun of the right for being "anti-math" and stupid. But then again I sound like those in 2012 who simply thought minorities were overpolled and that the Obama excitement wasn't as great. I forgot that Coleman Young was elected for 20 years, and the DNC is more and more as inflammatory as the ANC.

    ReplyDelete
  3. I blushed slightly at what Cosh wrote, because I've had Silver's book on my bedside table for about three weeks without opening it.

    ReplyDelete
  4. Same here Monroe.

    I mean simple common sense was telling me that Silver was full of shit.

    I remember him being trumpeted for predicting Obama winning in 2008, as though it was some off the wall prediction.

    But just looking at the methodology of the polls he heavily weighted I thought "no way this guy is on to something. He's got to be full of shit."

    Lo and behold, he ended up being right in 2012, much to my chagrin.

    ReplyDelete

Comments are moderated, at whim.