"The Weather Man Is Not a Moron"

I harp on one key issue in philosophy of science a lot because I get a lot of backtalk along the lines of: Everybody knows that the social sciences are a fraud. They can't predict whether the stock market will go up or down tomorrow, so how you can say that social science data suggests that letting in a bunch of unskilled illegal immigrants today will lead, all else being equal, to lower school test scores later when their kids get into school? If smart rich guys can't predict the stock market tomorrow, how can an evil nobody like you predict school test scores in a decade? Nobody can predict anything!

Nate Silver writes an article, The Weather Man Is Not a Moron, about how weather forecasting has improved dramatically, which it has. The forecast on the evening news is much more accurate than when I was a boy.

In 2008, Chris Anderson, the editor of Wired magazine, wrote optimistically of the era of Big Data. So voluminous were our databases and so powerful were our computers, he claimed, that there was no longer much need for theory, or even the scientific method. At the time, it was hard to disagree. 

But if prediction is the truest way to put our information to the test, we have not scored well. In November 2007, economists in the Survey of Professional Forecasters — examining some 45,000 economic-data series — foresaw less than a 1-in-500 chance of an economic meltdown as severe as the one that would begin one month later. ...

The one area in which our predictions are making extraordinary progress, however, is perhaps the most unlikely field [weather].

But this dichotomy between market forecasting and weather forecasting shouldn't be all that surprising if you keep in mind that there is, theoretically, a fundamental difference between forecasting events that respond to forecasts (e.g., the stock market) v. forecasting events that don't respond to forecasts (e.g., the weather). Theoretically, the former is resistant to improvement while the latter is not. Improving the weather forecasts is hard in an absolute sense, but the project lacks the special kind of futility that attempts to permanently beat other people's forecasts have.

Hurricanes don't respond to better forecasts by sitting down together and hashing out more sophisticated ways to fool weathermen.

In contrast, say you come up with a better way to predict whether the stock market will go up or down tomorrow. After awhile, your competitors in the stock market forecasting game will notice you are now riding around in a G6 and they will start trying to reverse engineer your method, or hire away one of your employees, or rifle through your trash. Eventually, your method will be widely enough known that the stock market won't go down tomorrow when your method says it will, because it will go down today because everybody who is anybody is already anticipating the decline that your system predicts. So, after awhile, your system will be so widely used it will be useless.

Let's simplify this a little by thinking for a moment not about the stock market as a whole, but just about one company. Consider Apple. In the absolute sense, it's obvious that Apple stock is worth a lot of money because it his highly likely to make a lot of money in the future.  ("Making money" is just an approximation of what stock analysts predict, but it's close enough for my purposes). But everybody knows that.

Whether or not you want to buy Apple stock depends instead on the relative question of whether it will turn out to be worth more money than the stock price. Will Apple make more money than the market's consensus of forecasts? That's obviously a more difficult, second-order question than whether Apple will make a lot of money. (But perhaps you have an insight that lets you predict the future better than the market. For example, maybe you realize that all Apple has to do to make even more money is stop having an all white male set of top executives.)

It may seem rather daunting to try to out-predict the experts on Apple's future. The thing is, however, that you can do pretty well just by flipping a coin: heads Apple will go up, tails apple will go down.

Financial economists call this the Efficient-Market Hypothesis. This does not mean that markets are more efficient than government at achieving various goals. It means that unless you have inside information, it's really hard for an investor to beat the stock market in the long run because others will adopt your forecasting tools.

The name Efficient-Market is most unfortunate because it's referring to the speed at which information is incorporated into forecasts, but is woozy on the accuracy of interpretation of the forecast. A phrase like Agile-Market Hypothesis might have been better.

For example, if the headline in the Wall Street Journal tomorrow morning is "iPhone Causes Brain Tumors," you won't beat the market by sauntering in and selling your Apple stock around noonish. Markets tend to be pretty agile (i.e., efficient) at acting upon new information.

On the other hand, the markets' interpretation of information is often wrong. For example, in the mid-2000s, the news that illegal immigrants were pouring into the exurbs of California, Arizona, Nevada, and Florida to build expensive new houses for subprime borrowers trying desperately to get their children out of school districts overrun by the children of illegal aliens was greeted almost universally as Positive Economic News. What could possibly go wrong?

Heck, a half decade later, this interpretation of What Went Wrong is largely verboten. If you read Michael Lewis's The Big Short carefully, yeah, you can kind of pick it up if you have an evil mind. But can you imagine a speaker at either party's convention saying what I just said?

Is the Efficient-Markets hypothesis true? One obvious problem with it is that the Forbes 400 is full of zillionaires who beat the market long enough to make the Forbes 400. Were they just lucky? Or is the Efficient Markets Hypothesis wrong? Perhaps you can make so much money in the short run from identifying a major inefficiency, such as the recent subprime unpleasantness, that you can wind up very rich if you have the humility to then retire from placing such big bets?

Or, could it be that the Efficient-Market Hypothesis is right, and a lot of the market beaters beat the market the old fashioned way: by insider trading?

About a half decade ago, there was a lot of publicity about what enormous ROIs the endowment managers at Yale and Harvard were generating. When I looked into it, there was a correlation between endowment ROI and how hard it was to get into that college. For example, Cornell had the worst ROI in the Ivy League. I hypothesized that maybe this pattern could be explained by asking: "If you had some inside information that you couldn't act upon yourself for fear of jail but could conceivably share with somebody you don't do business with in return for a huge favor, what would you risk to get your kid into Harvard v. what would you risk to ket your kid into Cornell?"

This theory was extremely unpopular, so forget I ever mentioned it. The SEC has never, as far as I know, prosecuted anybody for bartering inside information for college admission. In fact, as far as I know, nobody has ever even been investigated for this, so, obviously, it must never ever have happened, and we'll just have to look for the explanation of why the desirable colleges' endowments outperform the less desirable colleges' endowments elsewhere. Clearly, Harvard and Yale beat the market by investing in, uh, timber. Yeah, timber, that's the ticket!

In any case, the Efficient-Market Hypothesis embodies the crucial conceptual difference between trying to forecast the behavior of systems that respond to forecasts and those that don't. There's no Efficient-Weather Hypothesis. That's because if you get better at forecasting the weather, you stay better at forecasting the weather.

Potentially, forecasting the performance of the children of new immigrants ought to be hard because the U.S. government should be using feedback from past performance to adjust policy to get the optimal mix. If illegal immigrant drywallers from Guatemala aren't working out so well in the long run, okay, let fewer of them in. But of course, thinking about this subject is crimethink and even the numbers are hatestats.