September 7, 2012

"The Weather Man Is Not a Moron"

I harp on one key issue in philosophy of science a lot because I get a lot of backtalk along the lines of: Everybody knows that the social sciences are a fraud. They can't predict whether the stock market will go up or down tomorrow, so how you can say that social science data suggests that letting in a bunch of unskilled illegal immigrants today will lead, all else being equal, to lower school test scores later when their kids get into school? If smart rich guys can't predict the stock market tomorrow, how can an evil nobody like you predict school test scores in a decade? Nobody can predict anything!

Nate Silver writes an article, The Weather Man Is Not a Moron, about how weather forecasting has improved dramatically, which it has. The forecast on the evening news is much more accurate than when I was a boy.
In 2008, Chris Anderson, the editor of Wired magazine, wrote optimistically of the era of Big Data. So voluminous were our databases and so powerful were our computers, he claimed, that there was no longer much need for theory, or even the scientific method. At the time, it was hard to disagree. 
But if prediction is the truest way to put our information to the test, we have not scored well. In November 2007, economists in the Survey of Professional Forecasters — examining some 45,000 economic-data series — foresaw less than a 1-in-500 chance of an economic meltdown as severe as the one that would begin one month later. ...
The one area in which our predictions are making extraordinary progress, however, is perhaps the most unlikely field [weather].

But this dichotomy between market forecasting and weather forecasting shouldn't be all that surprising if you keep in mind that there is, theoretically, a fundamental difference between forecasting events that respond to forecasts (e.g., the stock market) v. forecasting events that don't respond to forecasts (e.g., the weather). Theoretically, the former is resistant to improvement while the latter is not. Improving the weather forecasts is hard in an absolute sense, but the project lacks the special kind of futility that attempts to permanently beat other people's forecasts have.

Hurricanes don't respond to better forecasts by sitting down together and hashing out more sophisticated ways to fool weathermen.

In contrast, say you come up with a better way to predict whether the stock market will go up or down tomorrow. After awhile, your competitors in the stock market forecasting game will notice you are now riding around in a G6 and they will start trying to reverse engineer your method, or hire away one of your employees, or rifle through your trash. Eventually, your method will be widely enough known that the stock market won't go down tomorrow when your method says it will, because it will go down today because everybody who is anybody is already anticipating the decline that your system predicts. So, after awhile, your system will be so widely used it will be useless.

Let's simplify this a little by thinking for a moment not about the stock market as a whole, but just about one company. Consider Apple. In the absolute sense, it's obvious that Apple stock is worth a lot of money because it his highly likely to make a lot of money in the future.  ("Making money" is just an approximation of what stock analysts predict, but it's close enough for my purposes). But everybody knows that.

Whether or not you want to buy Apple stock depends instead on the relative question of whether it will turn out to be worth more money than the stock price. Will Apple make more money than the market's consensus of forecasts? That's obviously a more difficult, second-order question than whether Apple will make a lot of money. (But perhaps you have an insight that lets you predict the future better than the market. For example, maybe you realize that all Apple has to do to make even more money is stop having an all white male set of top executives.)

It may seem rather daunting to try to out-predict the experts on Apple's future. The thing is, however, that you can do pretty well just by flipping a coin: heads Apple will go up, tails apple will go down.

Financial economists call this the Efficient-Market Hypothesis. This does not mean that markets are more efficient than government at achieving various goals. It means that unless you have inside information, it's really hard for an investor to beat the stock market in the long run because others will adopt your forecasting tools.

The name Efficient-Market is most unfortunate because it's referring to the speed at which information is incorporated into forecasts, but is woozy on the accuracy of interpretation of the forecast. A phrase like Agile-Market Hypothesis might have been better.

For example, if the headline in the Wall Street Journal tomorrow morning is "iPhone Causes Brain Tumors," you won't beat the market by sauntering in and selling your Apple stock around noonish. Markets tend to be pretty agile (i.e., efficient) at acting upon new information.

On the other hand, the markets' interpretation of information is often wrong. For example, in the mid-2000s, the news that illegal immigrants were pouring into the exurbs of California, Arizona, Nevada, and Florida to build expensive new houses for subprime borrowers trying desperately to get their children out of school districts overrun by the children of illegal aliens was greeted almost universally as Positive Economic News. What could possibly go wrong?

Heck, a half decade later, this interpretation of What Went Wrong is largely verboten. If you read Michael Lewis's The Big Short carefully, yeah, you can kind of pick it up if you have an evil mind. But can you imagine a speaker at either party's convention saying what I just said?

Is the Efficient-Markets hypothesis true? One obvious problem with it is that the Forbes 400 is full of zillionaires who beat the market long enough to make the Forbes 400. Were they just lucky? Or is the Efficient Markets Hypothesis wrong? Perhaps you can make so much money in the short run from identifying a major inefficiency, such as the recent subprime unpleasantness, that you can wind up very rich if you have the humility to then retire from placing such big bets?

Or, could it be that the Efficient-Market Hypothesis is right, and a lot of the market beaters beat the market the old fashioned way: by insider trading?

About a half decade ago, there was a lot of publicity about what enormous ROIs the endowment managers at Yale and Harvard were generating. When I looked into it, there was a correlation between endowment ROI and how hard it was to get into that college. For example, Cornell had the worst ROI in the Ivy League. I hypothesized that maybe this pattern could be explained by asking: "If you had some inside information that you couldn't act upon yourself for fear of jail but could conceivably share with somebody you don't do business with in return for a huge favor, what would you risk to get your kid into Harvard v. what would you risk to ket your kid into Cornell?"

This theory was extremely unpopular, so forget I ever mentioned it. The SEC has never, as far as I know, prosecuted anybody for bartering inside information for college admission. In fact, as far as I know, nobody has ever even been investigated for this, so, obviously, it must never ever have happened, and we'll just have to look for the explanation of why the desirable colleges' endowments outperform the less desirable colleges' endowments elsewhere. Clearly, Harvard and Yale beat the market by investing in, uh, timber. Yeah, timber, that's the ticket!

In any case, the Efficient-Market Hypothesis embodies the crucial conceptual difference between trying to forecast the behavior of systems that respond to forecasts and those that don't. There's no Efficient-Weather Hypothesis. That's because if you get better at forecasting the weather, you stay better at forecasting the weather.

Potentially, forecasting the performance of the children of new immigrants ought to be hard because the U.S. government should be using feedback from past performance to adjust policy to get the optimal mix. If illegal immigrant drywallers from Guatemala aren't working out so well in the long run, okay, let fewer of them in. But of course, thinking about this subject is crimethink and even the numbers are hatestats.

107 comments:

JeremiahJohnbalaya said...

The post covers the topic of US climate modelers being so obsessed with long term modelling that they lag the better short-term models by about a day.

US weather prediction falling behind

There's a point in there that the difference between a reliable seven day forecast, and a reliable eight day forecast (or whatever the exact numbers, that's close) might be on the order of BILLIONS of dollars lost to the economy.

US weather prediction falling behind

Clutch cargo cult said...

Actually efficient markets mean that even if you know the news you cannot profit from it. For example, today's employment report might very well have been devastating (it sure had Obama depressed last night). The employment picture painted by the report is grim. Yet somehow the market shrugged it off. What if you knew the result last night? Would you have sold ES futures? It wouldn't have paid. The truth is its all cooked in. The weather market is not efficient, no one put any money up, no one will lose their job or savings if they are wrong.

Carol said...

I thought the candlestick method worked remarkably well. Maybe it just reflects predictions already baked in.

Anonymous said...

If smart rich guys can't predict the stock market tomorrow ..


Not to belabor the obvious here, but the reason those "smart rich kids" are rich is precisely because they can predict the stock market tomorrow. Of course it does not benefit them in any way to share that knowledge with the world free of charge, so they don't do that.

Anonymous said...

I have come to suspect that the reason the Efficient Market Hypothesis is touted so much is to stop the majority of students from even trying to identify superior stocks, or methods in determining trends. Who pays for business school buildings? Who stands to benefit?

It's the Wall Street equivalent of the African tribes with profitable trade routes telling all sorts of hokey horror stories to that white hunter type who was set on going on safari there, that was recently mentioned here. I can't find it, perhaps someone can point that article out to me.

Or fishermen telling newcomers that "You'll never find any fish there!" when the newcomer asks about their favorite patch.

Captain Tripps said...

Well, Steve, you are essentially describing the difference between a human system (the market) and a non-human system (the weather). I remember learning about Fama’s Efficient Market Hypothesis in Finance Grad School, but I thought at the time it was far too simplistic. When information becomes available to market participants, they don’t all react to the information in the same way at the same time. Some participants will quickly (as you point out, the agility factor) and correctly interpret the current and future effects of the information on an asset’s price and effectively buy/sell the asset for profit accordingly. Others will take more time to digest the information, but also correctly interpret the future outcome, but will miss the arbitrage opportunity and simply file the information away for future reference. Still others will quickly process the information but come to incorrect or inefficient conclusions, and realize a loss with their buy/sell decision. And so on. So not all participants “efficiently” process the new information, only a handful do and capitalize on the arbitrage opportunity. The kicker is, most of that handful who make the correct buy/sell decision, do so that one time. The next time new information comes along, the handful who made the arbitrage profit the previous time most likely find themselves incorrectly interpreting the new information and missing the opportunity, or worse yet, incurring a loss because of an incorrect interpretation. And so it goes. Very rare is the market participant who consistently correctly interprets new market information for arbitrage profit opportunities. My guess would be those that do consistently beat the market have an additional inside edge; not necessarily “insider information” as it is commonly thought of (an unfair advantage), but more like an ability to connect to key information sources and gauge market opportunities through indirect clues. I guess my ultimate point is that, market prediction is like predicting human behavior, since markets are essentially human things. So, when you get right down to it, pretty much unpredictable.
Now, predicting the weather and climate change, that’s a whole different kettle of fish…

Anonymous said...

I'm the anonymous who just commented about the African tribes.

You can get an indication of just how much bullshit the EMH is, by the riches of various people on Wall Street who have been outperforming the market for a long, long time. And also, the fanaticism with which "quants" (IOW quantitative math geniuses) are recruited at Wall Street, rather than say, chimps or fortune tellers.

Of course, the non-conspiratorial explanation for the popularity of the EMH is that quantitative people like math, they like systems, and they prefer to approximate systems imperfectly rather than leave their nice office to figure out how the real world works and interact with people. Look at economists, with their fictional homo economicus who places no utility on ethnic or family concerns. Why? Because modelling the reality is hard. But running numbers is fun.

Anonymous said...

The ONLY way you can make money [consistently, even over relatively short periods of time] in the markets is IF YOU KNOW SOMETHING THAT FEW [IF ANY] OTHERS KNOW.

It's just tautological.

The probability that you could win as few as, say, four random short bets in a row is

(0.5)^4 = 0.0625 = 6%.

You simply CANNOT win at this stuff unless you know something that almost everyone else does not know.

PS: Short Contracts are pure, unadulterated evil.

They're the legalization of what Faulkner wrote about in Barn Burning.

In a just society, being a party to a short contract would be a capital offense.

Steve Sailer said...

"Well, Steve, you are essentially describing the difference between a human system (the market) and a non-human system (the weather)."

No, I think it's important to distinguish between systems that respond to forecasts and those who don't. For example, if I announce that I forecast that Usain Bolt will win the 100m dash, does that have much influence. Maybe a little. Maybe Usain thinks, "Sailer says I've got it in the bag, so I'll just stay out partying a little longer." But, on the whole, there isn't much impact of forecasts on this human system.

On the other hand, if I announce at noon everyday my forecast of whether the stock market will go up or down tomorrow and I'm right, say, 100 days in a row, eventually there will come a day when the stock market doesn't go up tomorrow like I said it would, it goes up right now this afternoon in anticipation. And then somebody figures out my trick and starts anticipating what I'll announce, so the stock market goes up at 11:59 am. And so forth and so on.

Luke Lea said...

Also human behavior is infinitely more complex than the weather, or just about anything else you can think of.

Eric Falkenstein said...

"agile markets" vs. "efficient markets". I think that's brilliant. There are so many stupid arguments instigated by the implications of 'efficient', which are really stupid because in this case it is simply a very specific theory, and 'efficient' is just a word for that specific definition.

I really appreciate your writings.

DaveinHackensack said...

There's sort of a corollary to the Efficient Market Hypothesis that often goes unstated: inefficiencies can persist if there isn't big money to be made in exploiting them. Here's a real world example, the tiny OTC stock WIRX. It has working capital of $7.7 million and no debt, but it has a market cap of less than $3.7 million, i.e., it's trading for less than half its liquid assets. So, in theory, an investor could buy enough shares to take control of the company, and them have those assets distributed as cash, and more than double his money.

In practice, though, the potential money to be made here is too small to move the needle for a big hedgie or activist investor. Maybe a penny ante activist will eventually notice this one, and figure out a way to take control without the spread on this illiquid stock cutting to far into his profits, but until then, the inefficiency will persist because it's too small to interest a big fish.

DirtyTricks said...

Steve,

Could you change the comments section so we can reply to individual comments directly?

Anonymous said...

"Not to belabor the obvious here, but the reason those "smart rich kids" are rich is precisely because they can predict the stock market tomorrow. Of course it does not benefit them in any way to share that knowledge with the world free of charge, so they don't do that." - Even if there is no incentive for them to tell everyone now(and it wouldn't be free of charge, book deals to tend to produce revenue), eventually the techniques or their use will be obsolete, And they should write tell-all books about how they beat the market. About what ingenious tricks they pulled off.

Unless what they did amounted to front running and insider trading of course.

Anonymous said...

DirtyTricks: "Could you change the comments section so we can reply to individual comments directly?"

Because there is not read/unread markup in the comments, doing this makes finding new posts in comments that exceed 50 entries ridiculously difficult. Not a good idea. Every blog that employs it is a PITA to sift through comments.


Martin (I Ain't No Nazi!) High-Digger said...

You've lost me. What the hell is this post about? The efficiency of markets? Predicting the deleterious effects of immigration? Cheating to get into Harvard?

The commenters here seem to have opted for the first choice, ignoring the other stuff.

And the title doesn't help.

Sometimes, it would seem, evilness of thought and clarity don't go together…

TGGP said...

One guy who famously bet big against subprime was John Paulson, as detailed in "The Big Short". He really looked like a genius then. His fund's been losing money hand over fist since then, betting that the dollar would severely devalue from what I recall. Now he looks more lucky than smart.

A short is a bet that a price will go down. Provided you have the means to cover your losses if it goes up (and most complaints I hear about it don't involve that actually happening), it sounds reasonable. If you're already exposed to the risk of an asset declining in price, you could hedge against it with a short contract. Lots of companies hate being shorted, since it indicates a lack of confidence in them, but that's just shooting the messenger.

Mr. Anon said...

"Anonymous said...

PS: Short Contracts are pure, unadulterated evil."

Are you referring to short selling? If so, I entirely agree. It ought to be illegal

Auntie Analogue said...

Life is a crapshoot. Those who own and run the house lose sometimes; but because they own the house they survive hits. The ordinary shooters, however, get skunked every time. No highfaluting theories are needed - after all, this comes under the heading of "You don't need a weatherman to know which way the wind blows."

beowulf said...

The Germans may not have a word for this but George Soros does, reflexivity.

"Soros' writings focus heavily on the concept of reflexivity, where the biases of individuals enter into market transactions, potentially changing the perception of fundamentals of the economy.
He has stated that his own financial success has been attributable to the edge accorded by his understanding of the action of the reflexive effect."
http://en.wikipedia.org/wiki/George_Soros#Reflexivity.2C_financial_markets.2C_and_economic_theory

Likewise, as Paul Davidson has noted, where American Keynesians went off the rails (beginning with Larry Summers's uncle, Paul Samuelson, in the 1940s) was ignoring Lord Keynes's actual rejection of the view that the economic future is predetermined by market fundamentals ("Soros’s concept of reflexivity, therefore, is the equivalent of Keynes’s rejection of the ergodic axiom").
http://rwer.wordpress.com/2012/03/28/the-ergodic-axiom-davidson-versus-stiglitz-and-lucas/

Matt said...

Side note: Krugman and Gingrich both have written about being influenced by Asimov's Foundation series, in which the protagonist comes up with a near-perfect predictive model of human behavior on large scales.

One of the main rules of the model is that its predictions can't be made public, or it won't work. This ought to have been more instructive to two guys who're in the business of public prognostication, but c'est la vie.

Anonymous said...

And then there's the Russian way of forecasting, which is to offer certain incentives:
http://articles.cnn.com/2005-03-02/world/moscow.weather_1_mayor-yuri-luzhkov-forecasts-chief-weatherman?_s=PM:WORLD

socks said...

"In that past, at least in some departments,
you’ve heard that there is no such thing as buying an undervalued stock, ... that the market is efficient, ...Therefore, there’s no use thinking. And, of course, from my standpoint I’d like to have everyone believe that, because it’s a terrific advantage to be in a game where your opponent has been taught not to think. ... It’s the old story that if there’s a $20 bill on the floor there’s no
sense picking it up because it can’t be there."

Warren Buffet, http://www.tilsonfunds.com/BuffettNotreDame.pdf

Ali said...

It's easy enough to make money in the stock market even if you're far, far away from any sources of inside information.

Stock market swings are based on a fear and greed.

The market has a recurrent habit of pushing prices to the stratosphere for stocks who it deems the future is rosy. Conversely it pounds into the ground stocks that have unexpected mishaps.

Plenty of people have done very well by avoiding the former and buying the latter like Cliff Asness and David Dreman.

I'm sitting on a 70% profit from buying British home construction stocks. The only connection I had was my broadband subscription.

Steve Johnson said...

"Potentially, forecasting the performance of the children of new immigrants ought to be hard because the U.S. government should be using feedback from past performance to adjust policy to get the optimal mix. Illegal immigrant drywallers from Guatemala aren't working out so well in the long run, okay, let fewer of them in."

While you're next leveling the dumb supporters of the current policy (pretty much everyone who writes for the public) they're next leveling you.

What makes you think that illegal immigrant drywallers from Guatemala aren't working out well? Because California is a worse place for you to live? So what. Plenty more people in prisons, on welfare, receiving Medicare, lots of child protective services visits going on, loads of cheap drywall getting put up, and lots of new Democrat voters.

Sounds like a successful policy to me!

Funcrusher said...

"Could you change the comments section so we can reply to individual comments directly?"

Yes. This.

DaveinHackensack said...

Let's see if this reply works (using mobile version of blog).

DaveinHackensack said...

Short sellers can perform a useful function in the market, calling attention to bad management, fraud, etc. David Einhorn, for example, raised red flags about Lehman well before its collapse.

Anonymous said...

The weather is determined by *known* scientific laws and effects and thus, in theory, was predictable in the sense that entities that follow mathematical relations are predictable.
The problem was, until recently, was the the sheer enormous number of varaibles - and the infinity of relations between the varaibles, in even the most local weather system that made mathematically based prediction, except crude and inaccurate gross over simplications, impossible.
Super computers, capable of insane numbers of calaculations per second, changed all that.

Now, my beef with economics is this - basically it is not founded on certain mathematical relations between a myriad of measureable, predictable entities, and thus amenable to exhaustive analysis.
It's all politics and political dogma, rabid lefties and righties taking pot shots at each other non stop.
Please bear that in mind.

Aaron in Israel said...

Could you please keep the comments the way they are, without enabling direct replies? Thank you.

The current set-up gives comments that are less bad and less prolific. With direct replying, you get endless threads of "I didn't say that" "Yes you did" "No I didn't" "Well you said that three months ago" etc.

Oh, I have a comment on the Efficient Market Hypothesis, while I'm here. Every model is false. The question is not whether the model is accurate, but whether the predictions and explanations based on a given model are useful, where usefulness is defined in a specific context (usually including accuracy).

Anonymous said...

It's not that simple Steve.
You forget that people have vested, selfish interests - an basically only care about those interests and couldn't give a damn about anything else.
Weather forecasters only care about weather forecasting - the more accurate they are the better they feel about themselves and the more kudos they earn.
Now take immigration driven 'booms'. Huge vested interests have stakes in boosting immigration - they will lie and lie to boost their pet interest. The might be ideologically driven lefties, they might be the genetic cohorts of immigrants themselves, they might br property developers out to make a quick buck. The point is that their own sectional interest is more omportant to them than any 'national' interest, so they keep plugging away - they lie, they con, they propagandize, they demonise - they get their way. No one notices or cares what's going on, so they get their way, The 90% of the population who are sheep believe te lies.
The real point is, that getting as amny of their 'own' people into the USA as possible, to live the good life, is their overwhelming interest so they fight hard for it.

Anonymous said...

It's an old, old story.
The classic examples are the Dutch tulip 'bubble' of the 17th century and the British 'South Sea Company' bubble of the 18th century.
The point is that markets are driven by the herd mentality. As long as there is 'quick easy' money to be made, everyone wants a piece of the pie - the great motivating factor is to get in while the going is good, and not be left on the sidelines like a chump. So everyone does pile in - the 'wise' the 'clever' the 'cautious' and all, they believe their own propaganda and do what the herd's doing.
The name of the game of capitalism is to make money. If you don't make money on the uptick, why, by definition you are a loser and a chump, so you get caught up in the rush.
It's an inherent flaw of capitalism, no matter how much people say they won't do it, or how much they say it's dumb, irrational etc they will keep doing it from now till doomsday - as long as 'money' is to be made money will follow it.

Silver said...

The name Efficient-Market is most unfortunate because it's referring to the speed at which information is incorporated into forecasts, but is woozy on the accuracy of interpretation of the forecast. A phrase like Agile-Market Hypothesis might have been better.

The term "efficient" here arose from economists' desperate desire to project an image of being able to describe economic events with physics-like precision. "Efficient" -- a person saying using that term almost can't help but sound as though he knows what he's talking about. It's the same with economists' fascination with "equilibrium." Equilibrium theories didn't actually arise from careful observations of economic behavior; it just seemed plausible enough and boy was it ever "elegant" and that's all they needed.

You can get an indication of just how much bullshit the EMH is, by the riches of various people on Wall Street who have been outperforming the market for a long, long time. And also, the fanaticism with which "quants" (IOW quantitative math geniuses) are recruited at Wall Street, rather than say, chimps or fortune tellers.

Thing is you can be lucky (and not know it) for a very long time. Consider a coin-tossing competition involving one million participants. They pair off and toss a coin against each other, the losing dropping out and the winner continuing onto the next round. The winner will have predicted the coin toss correctly 30 times in a row. That hardly makes him a genius.

Personally, I don't doubt that skill also plays a role. It's just that it's not always clear how much is skill and how much is luck.

Anyway, winning big in financial speculation doesn't require always being right. You can be wrong a lot. The trick is to make a lot more when you're right than you lose when you're wrong.

AL said...

"On the other hand, the markets' interpretation of information is often wrong."

That's a major catch. One of the assumptions of the Efficient Market Hypothesis is that market agents hold rational expectations, i.e. they are not systematically wrong (though individual agents may be wrong). If we agree that market participants were, in aggregate, systematically wrong in their expectations of IQ, future earning power, etc. of illegal immigrants to whom mortgages were being issued, we can no longer hold the EMH accountable.

When rational expectations are also politically prohibited hatethoughts, the market is unlikely to hold those rational expectations. Hence I propose the formation of the Sailer Fund to exploit such market inefficiencies.

On a lesser point, your intimations of insider trading by university endowments seem to neglect the fact that not every investment is subject to the EMH. Many people who allocate capital for a living deal in investments that are not openly traded. When a tech whiz wants to raise capital to fund his new idea, he doesn't stand on the floor of an exchange and auction off the right to invest in his business to the highest bidder. Instead, he sits down across a desk from someone who makes these investments for a living. This person conducts research into the tech whiz and his idea, all the while compiling non-public information which helps him decide whether to invest. Among this person's clients are likely to be entities such as CalPERS or an Ivy League endowment.

Aaron B. said...

A big difference between predicting stock market behavior and predicting immigrant test scores is that one is based on human behavior and the other is based on human potential.

If you buy and sell stocks, you could double your money tomorrow or lose it all. You could make 10 straight good calls or 10 straight bad ones. Luck plays into it, as do your biases and the actions of others.

Test scores, on the other hand, have an upper limit. You can choose to score below your potential on a test -- be drunk, tired, or indifferent -- but you can't choose to test higher. I can't just decide to score 150 on an IQ test one day if I'm really 125. And no one's come up with a way to make their kids score that much higher either.

So it's not like some immigrant group scores anywhere between 50 and 150, and we're trying to guess what they'll score this time. We already know how they scored back home, and how much of a boost groups tend to get from coming here. There's an upper limit, and a pretty narrow range of likely movement beneath that.

If people really believed that "anything can happen," someone could "bet on" the children of immigrants doing better than anyone else expects, by going to their parents and offering scholarships in exchange for considerations in hiring their children when they're grown. No one tries anything like that because everyone pretty much does know what will happen, whether we admit it or not. So any "betting" on their potential has to be done with other people's money -- or in cases like Bill Gates, money that's so trivial you can afford to spend it on compliments.

Anonymous said...

In any case, the Efficient-Market Hypothesis embodies the crucial conceptual difference between trying to forecast the behavior of systems that respond to forecasts and those that don't. There's no Efficient-Weather Hypothesis. That's because if you get better at forecasting the weather, you stay better at forecasting the weather.

Another way of explaining this would be: imagine we had machines that let us manipulate the weather. Then people would fire up their weather control machines in response to forecasts of bad weather, changing the result to something different than what had been predicted. That's basically what goes on with the stock market. But since we don't have weather control machines yet, forecasting the weather is fundamentally different from forecasting the stock market.

Conatus said...

It's like the Heisenberg Principal of Uncertainty, if your stock forecasting is right on the money, those subatomic retail investors will do what you do and poof! there goes your edge.
Maybe the guys who are too right in stock investing are keeping their ideas to themselves, making billions for themselves and storing it all away in Dynasty Trusts in Delaware, Alaska, Florida, New Jersey and Nevada where the trust will last forever, thus insuring their continuing control of the Whole Process.

Anonymous said...

Thing is you can be lucky (and not know it) for a very long time. Consider a coin-tossing competition involving one million participants. They pair off and toss a coin against each other, the losing dropping out and the winner continuing onto the next round. The winner will have predicted the coin toss correctly 30 times in a row. That hardly makes him a genius.

It doesn't. The stock market is harder than your game. It's more akin to picking the 1/10 or 1/100 stocks that do the best, each time. Even with a billion participants who need to pick the top 10% of stocks each time but with no method, only random chance, after 9 turns there is no one left playing.

I realize that you can pick some average stocks and have nearly the performance, but consider the number of investment decisions required.

When a person buys stock, even the same stock, they generally do it at a number of points in time, as they get the money to allocate. Each decision is separate - they can invest in the same stock, or other stocks.

Someone who believes in diversification is going to buy a different stock each time, or multiple stocks. He ends up with a hodgepodge of average, good and bad companies. He will never have exceptional results. He will never have terrible results.

Someone else who is prepared to back his own judgment will pick less stocks, and be prepared to live or die by his skill. He will still put in money as it comes to him. Each decision is a separate bet, even if it is in the same company each time, or a very few.

Over the course of an investing career, someone who is prepared to back himself to the hilt and invest properly can easily make 10-30 investment decisions in a year. It's hard to say that someone is even exceptional for sure without a longer term picture. Investing for 10 years may give someone at least 100-300 decisions.

Couple that number of decisions with the need to do significantly better than average, and luck alone just doesn't cut it. There are not enough people in the world for someone to have that sort of luck, if there is no skill involved. Consider that there are only only 10^80 atoms in the observable universe, and getting top 10% results on average over 100 investment decisions should have a probability on the order of 10^-80. Certainly if every decision must be 10% or better.

Anonymous said...

The commenters here seem to have opted for the first choice, ignoring the other stuff.

All contrarians think that the EMH is so much bullshit. It's the sort of thing that primarily professors and naive students believe in, so it's a popular thing to comment on.

Steve is probably right about the insider trading thing. It seems logical. We know that the investment performance of politicians is superlative, and can surmise the real reason for that. Seems logical that this sort of thing would extend to getting one's kid into Harvard.

I always enjoy Steve talking about obvious conspiracies that occur, making a valid theory about it, and poking fun at the current fashionable opinion that it is impossible to figure out a conspiracy or that they don't ever occur, and that only nutcases theorize about them. However, I don't always wish to comment on it.

And everyone knows how stupid or mendacious the elite are on immigration. Their pathetic attitude towards immigration has got nothing to do with overlooking the poor performance of the Hispanic Horde flooding the border. (And really, do you want to be put out of your job by importing a new elite of East Asians? Didn't think so.)

Anonymous said...

I recommended against the mobile view change to the blog and you didn't listen to me then (it really really sucks by the way) so I don't have much hope of being listened to now as enamored with "progress" as you seem to be but changing the commenting system would suck. Look at Taki's for proof of that. Some moby will arrive and say, "you're racist!" And a thousand commenters will respond, etc. The current method was better and allowing for mobile views still sucks terribly.


Anonymous said...

Steve, thanks for this post.

First of all, the market in large liquid S&P stocks is relatively efficient. Out of every hundred really smart hard working people that try to beat the index over a ten year period, only one succeeds. Go to the vanguard web site. Indexing beats active management 99% of the time

When you are talking about micro cap value stocks, the market is not efficient. it is quite possible for a smart determined and hard working person to analyze micro cap value stocks and get a legally obtained edge.

To Dave in Hackensack, I want to let you know that I have a number of close friends that run micro cap proxy battles. The key to doubling the stock price is you have to call up enough shareholders, build a coalition that contains more than 30% of the outstanding shares. With that coalition in hand you can get a lawyer to run a proxy battle to put in a new board of directors that will double your money.

Again, Dave you can't do any of this until and unless you (Dave) actually do the work to get in touch with the shareholders and build the coalition.

Anonymous said...

Steve, thanks for this post.

First of all, the market in large liquid S&P stocks is relatively efficient. Out of every hundred really smart hard working people that try to beat the index over a ten year period, only one succeeds. Go to the vanguard web site. Indexing beats active management 99% of the time

When you are talking about micro cap value stocks, the market is not efficient. it is quite possible for a smart determined and hard working person to analyze micro cap value stocks and get a legally obtained edge.

To Dave in Hackensack, I want to let you know that I have a number of close friends that run micro cap proxy battles. The key to doubling the stock price is you have to call up enough shareholders, build a coalition that contains more than 30% of the outstanding shares. With that coalition in hand you can get a lawyer to run a proxy battle to put in a new board of directors that will double your money.

Again, Dave you can't do any of this until and unless you (Dave) actually do the work to get in touch with the shareholders and build the coalition.

Anonymous said...

In light of all the damage they did to the living standards of the blamelessly innocent worldwide with their flawed, cockeyed theories, I propose that 'quants' should be referred to by the Anglo-Saxon vulgarism which is, in fact, the etymological couplet of the Chaucerian English word 'quant'.
(He in fact used the word 'quant' with its full sexual connotations in his work).

Anonymous said...

This is an excellent, excellent article, the best I've read of yours.

Cennbeorc

Anonymous said...

OT:

"Lance Armstrong’s Secret Is Out

The news leaks about The Secret Race have vastly undersold its importance. Tyler Hamilton’s book is a historic, definitive indictment of cycling’s culture of doping during the Armstrong era."

http://www.outsideonline.com/outdoor-adventure/media/books/Keyes-hamilton-the-secret-race.html

"Here’s the reality: The Secret Race isn’t just a game changer for the Lance Armstrong myth. It’s the game ender. No one can read this book with an open mind and still credibly believe that Armstrong didn’t dope. It’s impossible. That doesn’t change the fact that he survived cancer and helped millions of people through Livestrong, but the myth of the clean-racing hero who came back from the dead is, well, dead."

...

"Beginning with his first doping experiences as a member of the U.S Postal Service team in 1997, Hamilton reveals not only what he and other riders were doing and taking (EPO, steroids, testosterone, Actovegin, blood transfusions, and on and on), but also how they were taking it (in the case of EPO, intravenously—and Hamilton has the scar to prove it). He tells us how most riders evaded detection (one trick: French laws bar testers from showing up between 10 p.m. and 6 a.m., so cyclists “microdosed” EPO at ten and the drug was gone by morning) and how the game was rigged in a way that made testing nearly irrelevant (“If you were careful and paid attention,” writes Hamilton, “you could dope and be 99 percent certain that you would not get caught”). Supporters still clinging to the claim that Armstrong passed more than 500 drug controls will be shocked to learn how insignificant those tests really were.

Not that all this doping and evading was a cinch. Hamilton describes the exhausting deceptions and logistics required to obtain the drugs, hide the drugs, store the blood bags, schedule the dosing—the hundreds of details necessary to maintain the high-40s hematocrit level that keeps a racer competitive on the course and safe in the control room. At times the evasive measures sound like techniques from a cheap spy novel. There are disguises, prepaid cell phones, clandestine meet-ups in random hotel rooms, and lots and lots of code names, including “red eggs” (testosterone pills), “Edgar” (EPO), and “oil” (testosterone drops). At one point, Hamilton got a text from his doctor on his prepaid phone during a Tour de France rest day: “The restaurant is 167 miles away.” Translation: Meet me in room 167 for your blood transfusion.

The drugs are everywhere, and as Hamilton explains, Armstrong was not just another cyclist caught in the middle of an established drug culture—he was a pioneer pushing into uncharted territory. In this sense, the book destroys another myth: that everyone was doing it, so Armstrong was, in a weird way, just competing on a level playing field. There was no level playing field. With his connections to Michele Ferrari, the best dishonest doctor in the business, Armstrong was always “two years ahead of what everybody else was doing,” Hamilton writes. Even on the Postal squad there was a pecking order. Armstrong got the superior treatments."

Anonymous said...

Not to belabor the obvious here, but the reason those "smart rich kids" are rich is precisely because they can predict the stock market tomorrow.

A variant of the EMH says that you might be able to beat the market by applying more analysis--but that the increased cost of the analysis offsets your increased return.

I suspect the EMH is broadly true. In fact most of the big managed mutual funds don't beat the market, nor do the big pension funds. The EMH tends to fall down in low information markets, and in bubbles/manias like the housing bubble and the tech stock bubble.

I haven't seen good data on hedge fund or private equity performance. That's a little different in that they aren't necessarily just buying and selling financial instruments, but might also be buying and running the business. In that case management skill is also entering into the question.

Note that the indexers are basically free-riding off of the analysis work of everyone else. The assumption is that big stocks like Apple have been analyzed to a fare-thee-well by clever green eyeshade types who have arbitraged away all the easy money. The stock picker using a dart board is really making use of all that analysis, he's just not paying for it.

Anonymous said...

"Also human behavior is infinitely more complex than the weather, or just about anything else you can think of."

Mmmmm, I don't know about that. You're pretty assured that people who are promised something and have a track record of getting it --like food, shelter, free health care, you know, free stuff--will reward the one who promised it. So, other people look at the fulfilled promise of free stuff, think, "Hey, yeah, if that person can get it so can I," and they too go after it.

That's a pretty predictable human behavior.

a very knowing American said...

Karl Popper made more or less the same point about the impossibility of predicting future human history in "The Poverty of Historicism." He was mainly arguing against Marxism.

But it might be useful to distinguish between (a) cases where I can change my strategy in response to your strategy if I have the incentive, and (b) cases where it's just too hard for me to change. For example, (a) most people find it hard to lose (or gain) weight. But if there's money at stake, if you bet someone a million dollars they can't lose or gain a lot of weight (or, say, impersonate the opposite sex on the internet), they just might take your money. On the other hand, (b) no amount of money can get a person to impersonate someone with a much higher IQ. A half hour's IQ test will smoke them out.

This sad fact may explain why a lot of Sailerian predictions based on group IQ differences are almost certain to come true, no matter what unpredictable things the educational establishment comes up with in the future.

Anonymous said...

Also, weather data is what it is seen through satellites and other instruments.
Much of financial data is controlled by institutions and those are controlled by many who are crooks.
Same with media coverage of crime. If you go by the media, you'd think crimes are committed by 'youths' regardless of race or color. They just be 'youths' and 'teens'.

Btw, does this mean global warming is real?

Anonymous said...

Though human activity certainly contribute to climate changes, most of what happens in weather is beyond our control. Regardless of what we wish or think, weather is gonna be weather. Whether Microsoft wins or Google wins, weather is gonna be weather. Whether Democrats win or Republicans win, weather will be weather.
But the economy is driven by what people do, and so predicting the economy is like predicting human behavior, and that is never easy.
Also, the natural laws that govern the weather stay the same.
But the laws of economy changes as science and technology change.

I think one thing that can overturn all our prediction of future demographics is bio-engineering. If we figure out a way to 'perfect' human genes through bio-genetics and decide on a way to produce mass clones of 'perfect humans', all the rules will be changed.

Anonymous said...

Steve,

To say that the Forbes 400 list disproves EMT is odd. Most people on that list made their money through creating or running a specific business. There's a difference between a businessman and an investor - though, granted, those lines do blur occasionally.

A person building a business from the ground up doesn't need to predict the stock market. He needs to create a compelling product and manage a firm to produce that product. That says nothing about EMT.

Yes, there are some investors on the list - Soros and Buffett - but they're the exception.

I personally believe that timing the market or picking stocks that will outperform is likely possible, i.e. there are people who beat the market through skill rather than luck. However, I also believe that they are unbelievable rare and that neither I nor you - or even the people doing it themselves - can distinguish who is beating the market through luck and who is beating the market through skill. (There’s also the question of whether a person who’s able to beat the market through skill in one economic environment can continue to do so in different environments, i.e. one guy may be very good in down times while another intuitively understands inflationary times.)

Therefore, that means that trying to find someone to beat the market is simply a bet, an unimaginably bad bet. Your odds of finding someone who will beat the market over the long run through skill are probably one in a million. Moreover, the amount that you gain by winning that bet likely isn't that much, while the amount that you lose - the almost certain outcome - is quite high.

It's a suckers bet, but one so tantalizing - especially for smart people who just can't accept that their high IQ doesn't help them in something that seems so influenced by brains - that the vast majority of people keep trying.

Anonymous said...

You don't need a weatherman to know which way Obama bows.

Pat Boyle said...

First let me mention that accurate weather forecasting reaches an impassible barrier around a week in the future. Better computers are not going to fix that.

To understand this listen to Jeff Goldblum's little speech in Jurassic Park. He lectures on chaos. It really has nothing to do with the plot and the dinosaurs but it has a lot to do with weather forecasting. He brings up the phenomenon of sensitivity to intial conditions that leads to unpredictable results. Lorenz noticed decades ago that tiny differences in intial conditions led to huge and upredictable results.

This is sometimes called the "Butterfly Effect". That's a colorful name like Schodinger's Cat that's easy to hold onto mentally, but it's very real.

You can see another aspect of chaos when you fill your bathtub. Run the water slowly and you get laminar flow. In theory you could track every water molecule and predict its next position in the stream. But turn the tap just a little bit more and you now have chaotic flow. No amount of computing power on the planet could now predict its outcome except statisically. Chaotic flow is not random - it is completely determined and predictable (in theory) but the actual behavior is so complex it must be treated as if it were random. There are very simple equations that after a couple iterations on your PC yield unexpected results. They are predictable in the sense if you run them again with exactly the same starting point you will always get the same answer. But you cannot predict the next step without knowledge of the equation. It could go up, it could go down. This is the position of the weather caster. At some point knowledge of conditions doesn't matter.

This is the hard limit which weather prediction comes up against. You can predict perfectly for a couple days but after that you might as well use a Ouija board.

My second observation is that even knowing certain types of "inside information" is useless. Back when I was teaching data communications and doing technical editing for Novell Books, I knew a bit about routing. I knew for example that Cisco was about to be overtaken in the market because it used less efficient algorithms. There were two new emergent router companies that used better procedures. Not very many people were aware of this at the time so I decided to cash in on my special knowledge. I invested. What I hadn't counted on was that Cisco certainly understood this very well - they bought both of those little companies. Once again I had failed to become rich.

Albertosaurus

Mr. Anon said...

"DaveinHackensack said...

Short sellers can perform a useful function in the market, calling attention to bad management, fraud, etc."

I suppose that burglars perform a useful function in the market too: by liberating assets that were not in circulation, because - you know - OTHER people owned them, and then injecting them into the economy.

Short selling consists of somebody else and YOUR broker conspiring to hose YOU with YOUR own stock. It should be banned.

Mr. Anon said...

"Anonymous Aaron in Israel said...

Could you please keep the comments the way they are, without enabling direct replies? Thank you."

I second Aaron's request. I like your comments section fine the way it is now. Thanks.

Anonymous said...

Elliot wave predicts the market, or at least gives you a good road map

People who say stuff like that are just ignorant...They aren't smart enough to know that they know nothing.

DaveinHackensack said...

I think reflexivity refers to a different phenomenon, the effect that investors and traders can have on the real economy. E.g., if investors think Spain will have trouble financing it's debt, they'll dump Spanish debt, bidding up the yields on it, thus making it more expensive for Spain to finance its debt.

What Sailer is talking about here isn't the impact of market participants on real economy fundamentals, but the impact of market participants on market predictions, particularly predictions by individuals or indicators with track records of accuracy.

DaveinHackensack said...

Warren Buffett addressed the coin-flipping argument years ago in his Super Investors of Graham-and-Doddville speech in the '80s. I think one of the other commenters here may have linked to it.

DaveinHackensack said...

It's about the difference between making predictions about complex systems where the predictions themselves have no effect on the system (e.g., weather), and where they do (e.g., the stock market).

DaveinHackensack said...

Doable, but easier said than done. For every beaten-down bargain stock there are so-called "value traps" that may look cheap on a backward-looking valuation basis but never recover.

Anonymous said...

AGAIN: YOU CANNOT KEEP WINNING SHORT BETS UNLESS YOU KNOW SOMETHING THAT MOST [IF NOT ALL] OTHER PEOPLE DO NOT KNOW.

If you know that Mark Cuban is about to sell Yahoo! $5.9 billion in absolutely worthless "internet television" vaporware, then you SHORT Yahoo!.

If you know that Herbert and Marion Sandler are about to sell Wachovia a portfolio of $24 billion in absolutely worthless sand state liar loans, then you SHORT Wachovia.

If you know that Goldman Sachs is about to subvert the McCain campaign, and that they have Stanley Ann Dunham's mentor's son installed at the NY Fed, to secretly launder them several billion dollars in hush money a few months later, then you go LONG on Goldman-Sachs.

But, of course, almost no one knows about these things.

That's why the people who do know about these things become billionaires, while the rest of us eat hot dogs and baked beans on Saturday night.

Average Joe said...

Nobody can predict anything!

This is the sort of thinking that led to a significant minority of white people voting for an unqualified black man in the 2008 presidential election. After all, there was absolutely no way of anticipating that Obama would be a bad president!

Silver said...

No, I think it's important to distinguish between systems that respond to forecasts and those who don't. For example, if I announce that I forecast that Usain Bolt will win the 100m dash, does that have much influence. Maybe a little. Maybe Usain thinks, "Sailer says I've got it in the bag, so I'll just stay out partying a little longer." But, on the whole, there isn't much impact of forecasts on this human system.

I've wondered about this a lot when it comes to team sports. Hot favorites occasionally lose and long shots occasionally win, as everyone knows. Would this occur more/less often if "no one knew" they were favorites/underdogs? I've found being a contrarian can sometimes pay off handsomely (and earn you big time bragging rights). The trick is to limit your losses when you hit losing streaks and to bet big when the going's good. (How do you know when? You don't. But it doesn't change the fact that you'll lose less when betting small and wrong and win more when betting big and right.)

It may seem rather daunting to try to out-predict the experts on Apple's future. The thing is, however, that you can do pretty well just by flipping a coin: heads Apple will go up, tails apple will go down.

Sailer, you're a tyro. Getting the direction right is only one factor and not even a particularly important one. I'd rate the magnitude of the move as vastly more important. I'd be infinitely happier getting direction wrong on a big move against me -- say I go long a stock at $100 that eventually drops to $50 with me getting out at $90 -- than getting direction right on a small move I call correctly -- eg going long at $100 and getting out at $105 on a move that only hits $110. Control your losses and you'll eventually hit upon a big winner -- if you have the mental fortitude to stay with it (rather than booking a quick win for the satisfaction of being able to tell people you were "right").

Anonymous said...

I honestly don´t see what the big deal is with insider trading. I wish I could get some of that good inside info....

The Anti-Gnostic said...

EMH has weak and strong variants. It strikes me that if competition for profits shrinks margins then competition for positive ROI should do the same thing. Asymmetry of information does not last long. Didn't Peter Lynch leave the game when large numbers of folks just started following him around the market?

Re: the weather man, it's kind of fascinating in a collapsing-crane-video kind of way how proletarian local TV news is getting. How much do those poor sods make now, minimum wage? Room and board?

Mexico takes weather prediction somewhat less seriously. Fox News definitely takes a more relaxed approach to current events as well.

jody said...

you guys are way behind. thinking about actual men making trades.

with modern computers and good algorithms making all your trades in 1 millisecond increments, it's now routine to come out ahead of the market year over year.

you don't need any inside information, or any information at all, beyond the basic data parameters required for your algorithm. your limiting factor becomes sheer speed of data transmission.

extraneous information is irrelevant. is the CEO gonna dump shares? irrelevant. was there bad weather this summer decreasing the supply of an agricultural commodity? irrelevant. is there about to be a war increasing the price of an energy commodity? irrelevant. is the money supply going to be increased next week, causing the price of gold and silver to go up? irrelevant.

it's totally, completely, wildly irrelevant whether you're literally inside ben bernanke's head. even if you are ben bernanke yourself, you can't make as much money knowing ahead of time what you're going to say about QE3, then you would if you simply run a supercomputer making 1 ms trades based on algorithms and a very fast connection directy to NYSE.

DaveinHackensack said...

OK, N.B. re replying to comments: the mobile version of Blogger, as viewed on an iPhone, includes a "reply" button, which I used a few times above, in lieu of copying and pasting the comment I was replying to above my own comment. Now that I'm looking at this comment thread on my desktop (using Firefox), it appears that that reply button doesn't actually work.

I suggest Steve add Disqus as a commenting system. It would make longer threads easier to follow.

Silver said...

with modern computers and good algorithms making all your trades in 1 millisecond increments, it's now routine to come out ahead of the market year over year.

On a risk-adjusted basis? Bullshit it's routine. Anyway, the operative phrase here is "good algorithms." What is a "good algorithm" when the market behaves the way your algorithm expects can quickly become a "bad algorithm" when market behavior changes. Of course, it's always possible to make a ton of money while the going's good, not lose too much when the going's bad, and use your track record to sell seminars.


DaveinHackensack said...

"I suppose that burglars perform a useful function in the market too: by liberating assets that were not in circulation, because - you know - OTHER people owned them, and then injecting them into the economy."

You are parading your ignorance if you think that's an apt analogy.

"Short selling consists of somebody else and YOUR broker conspiring to hose YOU with YOUR own stock. It should be banned."

Short selling is just selling a stock you borrowed, in the hopes of buying it back in the future for a lower price and profiting from the difference. That's not a "conspiracy" any more than buying a stock now and hoping to sell it in the future for a higher price is.

It's true that short sellers will often "talk their book", telling people why they think a stock sucks, but that can be a useful corrective considering that long investors ta;l their book in the opposite direction day in and day out, and there are plenty more long investors than there are short investors. David Einhorn probably saved some long investors in Lehman money by encouraging them to sell their stock before the company went up in flames.

Short sellers tend to be unpopular, because most investors are long only. But, as in the case with Einhorn and Lehman, the truth can be unpopular. Short sellers are sort of the Steve Sailers of the market.

map said...

A more interesting question is why the EMH is so heavily promoted. The EMH is the standard theory of every finance and financial economics program. It is even pushed onto the lowliest of retail investors. The question is...why?

To answer this question, look at how the financial industry (from exchanges to money managers) actually makes money.

Fund managers get a percentage of a pool of funds in total and/or a percentage of the profits. This means that, at minimum, a fund manager can make his earnings increase by simply increasing the size of his pool of funds, regardless of how well his stock picks actually do. Conversely, decreases in his pool of funds can lead to losses even if his stock picks are good.

In other words, managing other people's money is not just about picking winners and avoiding losers. It's also about managing redemptions by reducing the incidence of investors wanting their money back by encouraging a buy-and-hold strategy among the public. The EMH has been very successful in conditioning the general public into not bolting from their investments in every downturn, even in the event of 30% losses.

This has effectively lined the pockets of Wall Street for decades. Not only does the EMH stabilize management fees by stabilizing fund sizes, the money constantly flowing into Wall Street provides both liquidity and insurance (hedging) for proprietary trading operations that actually make real money by managing volatility. The rest of the market is merely there to provide the "float" upon which these huge Wall Street fortunes are made.

Nobody makes money on the EMH. The long-term data and other benchmarks are complete garbage and exist only for legal and marketing purposes.




map said...

As I said, EMH long-term data is garbage. Consider the micro- or small-cap effect. People often tout this as a proof that the EMH is incorrect. The small-cap effect actually exists because indexes are weighted in favor of the biggest companies with the largest number of shares. The S&P 500 index fund is not truly diversified equally among all 500 companies. Rather, the biggest firms are the ones that move the market the most, even if their volatility moves less than the smaller firms.

Exchanges set up these weighted indexes to please their biggest clients, guaranteeing that the index will track the performance of these biggest clients.

Heck, I would not be surprised if 40% of the Nasdaq's movement is Apple, Microsoft, and Google. How diversified, then, are any of these funds?

map said...

Incidentally, there is always money to be made in the market because people leave money on the table all of the time.

The purpose of the quants is not to predict markets. The purpose is to manage the volatility in indexes and stocks in a way consistent with the risk profile of a trading operation. You do not try to make more money by betting more than you can safely handle. Consequently, you leave those profits on the table for a more able investor to scarf up.

Funds that don't match their investing style to their personal risk profiles blow out very quickly.

Silver said...

In other words, managing other people's money is not just about picking winners and avoiding losers. It's also about managing redemptions by reducing the incidence of investors wanting their money back by encouraging a buy-and-hold strategy among the public. The EMH has been very successful in conditioning the general public into not bolting from their investments in every downturn, even in the event of 30% losses.

It's hard to believe mutual funds would benefit from pushing EMH. What's their angle? "Listen: No one can beat the market over the long-term -- and neither can we!" Lol. You'd think their interest would be in downplaying EMH in order to present themselves as experts. "Few people can beat the market, but we can, so trust us." Anyway, it's not as if buy-and-hold requires EMH to be true in order to suggest it as the best strategy for passive investors.

Anonymous said...

Dave in Hackensack,
do you have micro caps in your portfolio where you are sure that a new board of directors can sell off assets and double the stock price?

If so, have you tried organizing a proxy battle yourself or tried bringing the stock to a micro cap proxy battle firm?

Anonymous said...

"If smart rich guys can't predict the stock market tomorrow"

The financial markets are rigged.

The smart rich guys who do the rigging can predict what is going to happen* - but obviously aren't going to tell anyone.

(*Except occasionally when they get too greedy and there's a big meltdown.)

The rest i.e. the people who aren't part of the system, can't predict what's going to happen - because the system is rigged.

Since the advent of pension funds the stock market has gradually been re-engineered to rob pension funds and that is all they do.

JSM said...

"Short sellers are sort of the Steve Sailers of the market."

Hooey. Naked shortselling, especially in the microcap market, is ubiquitous. Naked shortselling is where, rather than borrow shares that exist, you simply make a sell transaction and take the money, all the while "promising" to deliver the shares later, hoping that the downdraft you caused will bring a glut of shares for sale to the market and enable you to get hold of enough shares to cover your nakedness.

It's counterfeiting, pure and simple, and it's computers that make the scam possible.

And even "legit" shortselling is far too often an instance where YOUR broker loans to a shortseller, not his own shares, but YOURS that he's holding for you, so that the shortseller can profit and the broker can profit (from the commission on the shortselling trade), but YOU, Mr. Long, are being royally screwed over. Nobody asked YOUR permission to sell short YOUR stocks. That's why the whole thing stinks to high heaven.

DaveinHackensack said...

Silver,

Indexing is a huge business which benefits from the promotion of EMH. A lot of financial planners benefit from it too: if none of your clients expect you to find active managers that can beat the market, then you can just out them in passive investments; that frees up time for you to find more clients. It also frees up room for you to charge more in fees, since passive/index funds tend to have lower fees.

Anon,

The one I own now that fits that bill is WIRX. I haven't contacted any activist investors about it, but I have one or two in mind I may try. If you know of any nano cap activists, feel free to mention one here.


JSM,

I wasn't describing or defending naked short selling, which I believe should be more actively combatted.

David Davenport said...

For example, in the mid-2000s, the news that illegal immigrants were pouring into the exurbs of California, Arizona, Nevada, and Florida to build expensive new houses for subprime borrowers trying desperately to get their children out of school districts overrun by the children of illegal aliens was greeted almost universally as Positive Economic News.

Would Evelyn Waugh or John Updike write a sentence that way?


You simply CANNOT win at this stuff unless you know something that almost everyone else does not know.

Jesus help us all, including bitter men. Every time I see ads for Scottrade, E*Trade, etc. on CNBC, I think , "There must be a lot of poor devils out there, laid off or forced into early retirement, trying to be ace short term stock traders... And thereby burning their cash in a hurry."

Me? I'm no better than so-so at stock picking. Guys like me are better off holding Vanguard ETF's.

Anonymous said...

Steve, I wish every MBA learned his lessons as well as you have, and your final comment (about how an optimal immigration policy would' make certain predictions difficult) is excellent. (I'm a retired finance prof.)

Mr. Anon said...

"DaveinHackensack said...

You are parading your ignorance if you think that's an apt analogy."

It was not meant as an analogy, merely as an example of another kind of thievery that one could, I suppose, find an upside to.

Short selling is just selling a stock you borrowed,...."

Borrowed from whom? Who loans it to them? Does Mr. Smith, the short-seller, call up Mr. Jones and say: "Hey, old man, care to loan me some of that valuable stock you own, so that I can sell it and depress it's value. I'll give it back to you when I'm done."? Why would anyone consent to that?

No. He borrows it from a broker - who loans out stock owned by his clients, i.e., who loans out an asset belonging to somebody else, for the purpose of lowering it's value. The broker profits. The short-seller profits. The customer gets screwed.

That should at least be considered a breach of the broker's fiduciary responsibility. Ideally, it would be considered malfeasance. Short of that, perpaps people could begin to demand that whatever agreement they sign with their broker stipulate that he NOT loan out their stock, and then sue him if he violates it. Perhaps they should sue the broker anyway.

"Short sellers tend to be unpopular, because most investors are long only."

Drug dealers are unpopular too. So are pimps. Perhaps short-sellers are unpopular because they are greedy dicks.

Anonymous said...

A more interesting question is why the EMH is so heavily promoted.

The financial industry would much prefer investors to put their money into managed funds. The management markup on index funds is miniscule. Vanguard S&P 500 mutual fund has an expense ratio of 0.17% and the ETF has an expense ratio of 0.05%, while similar actively managed funds have an expense ratio in the 1-2% range. Hedgies famously charge 2% of funds invested plus 20% of the profits.

Right out of the gate the actively managed funds need to beat the market by about 1% year just to break even compared to the indexers. About 80% of the actively managed funds don't even equal the market in any given year.

Index funds leave much less money for the financial industry to buy hookers and blow.

TGGP said...

People who buy an asset hoping that they will make money when it goes up in price are not greedy. Not at all.

map said...

Silver @ 9/8/12 4:38 PM

It's hard to believe mutual funds would benefit from pushing EMH. What's their angle? "Listen: No one can beat the market over the long-term -- and neither can we!" Lol. You'd think their interest would be in downplaying EMH in order to present themselves as experts. "Few people can beat the market, but we can, so trust us." Anyway, it's not as if buy-and-hold requires EMH to be true in order to suggest it as the best strategy for passive investors.

The problem with presenting oneself as an expert is that if your predictions are wrong, if investors actually went through a 30% loss through your fund in the 1960’s, then your fund would quickly shut down as every investors demands their remaining money back. Remember, mutual funds are pushing the EMH to reduce the incidence of redemptions. That’s what kills a fund. Trading off a little bit of stock-picking appearance to remove the threat of redemptions is a small price to pay. That is their angle: “No, Mr. Smith, in the long run the market returns 10%. Don’t worry about the short-run volatility. That is just random noise. Hold on to your shares.”

You don’t see a benefit in this?
Was buy and hold the best strategy for passive investors in 2008 when they were about to retire?

EMH style investing is best for a certain type of really low-risk, low maintenance investor, someone who could put his money into a money-market account or cd.

The real issue is why EMH investing is so heavily promoted as the unbeatable strategy?

map said...

Anonymous @9/8/12 9:19 PM

"The financial industry would much prefer investors to put their money into managed funds. The management markup on index funds is miniscule. Vanguard S&P 500 mutual fund has an expense ratio of 0.17% and the ETF has an expense ratio of 0.05%, while similar actively managed funds have an expense ratio in the 1-2% range. Hedgies famously charge 2% of funds invested plus 20% of the profits."

Vanguard and other index funds have a lot more money under management then hedge funds. Vanguard's S&P 500 fund probably has more that $100 billion under management. At 0.17 percent, that is $170 million in fees for doing nothing on minimal staff and rental space.

And don't forget, Vanguard has the option of juicing its returns by investing some portion of that $100 billion dollars to smaller hedge funds, whose profits probably accrue to other Vanguard entities their investors don't know about.


David Davenport said...

(Steve, please omit this post if it is a duplicate post. )

Meteorologist Ed Lorenz is often credited with the discovery of chaotic mathematics in 1963. Henri Poincare, studying the three [mutually attracting] body problem in the 1890's may have been the pioneer who foreshadowed chaos theory.

In regard to weather forecasting, chaos theory says that precisely
predicting weather beyond a week or ten days from the present may be mathematically impossible, no matter how much more powerful computers become.

As far as I know, the late Dr. Lorenz never stated whether or not long range climate modelling is also subject to chaotic effects.

Some people also suspect that stock market behavior is chaotic.

David Davenport said...

Edward Norton Lorenz

From Wikipedia, the free encyclopedia

Chaos theory
Lorenz attractor
Butterfly effect

Edward Norton Lorenz (May 23, 1917 – April 16, 2008)[1] was an American mathematician and meteorologist, and a pioneer of chaos theory.[2] He discovered the strange attractor notion and coined the term butterfly effect.

... After his return from the war, he decided to study meteorology.[2] Lorenz earned two degrees in the area from the Massachusetts Institute of Technology where he later was a professor for many years. He was a Professor Emeritus at MIT from 1969 until his death.[2]

During the 1950s, Lorenz became skeptical of the appropriateness of the linear statistical models in meteorology, as most atmospheric phenomena involved in weather forecasting are non-linear.[2] His work on the topic culminated in the publication of his 1963 paper Deterministic Nonperiodic Flow in Journal of the Atmospheric Sciences, and with it, the foundation of chaos theory.[2][4] His description of the butterfly effect followed in 1969,[2][5][6]. ...

Lorenz built a mathematical model of the way air moves around in the atmosphere. As Lorenz studied weather patterns he began to realize that they did not always change as predicted. Minute variations in the initial values of variables in his twelve-variable computer weather model (c. 1960) would result in grossly divergent weather patterns.[2] This sensitive dependence on initial conditions came to be known as the butterfly effect (it also meant that weather predictions from more than about a week out are generally fairly inaccurate).[10]
Lorenz went on to explore the underlying mathematics and published his conclusions in a seminal work titled Deterministic Nonperiodic Flow, in which he described a relatively simple system of equations that resulted in a very complicated dynamical object now known as the Lorenz attractor.[4]
[edit]See also

Systems science portal
Attractor
Butterfly effect
Chaos theory
Experimental mathematics
Lorenz attractor
Meteorology
Numerical weather prediction

NOTA said...

Silver,

Perhaps, but there's another wrinkle--mutual funds get paid based on their fees. It is spectacularly hard to argue for the reasonableness of fees when your mutual fund simply keeps an investment in the S&P 500 with occasional rebalancing of their portfolio. Most mutual funds aren't index funds, but rather bundles of stocks picked by a well-paid fund manager whose job is, essentially, justifying those fees. The data on these fund managers is that their performance is very hard to distinguish from luck. There are always a few people on a winning streak, but that's what you'd expect from luck.

For those who think there is some widespread secret information that lets you make consistent above-market returns, look at the Madoff case. Bernie Madoff was as connected and plugged in and informed about the markets as anyone you can imagine. Among his victims were seriously sophisticated people, including a number of hedge fund investors and wealthy people with experience and access to expert financial advice. His best available strategy was to run a Ponzi scheme. The explanation I have heard for this was that many of his sophisticated investors assumed he was doing a kind of inside-trading (front-running trades from his brokerage house), since otherwise, they knew he couldn't be getting the consistency of returns he was claiming.

NOTA said...

I think Steve's original point is that there's an important difference between places where a better science of prediction either does or doesn't affect the behavior of the thing being predicted. There may well be fundamental limits on how well anyone can predict weather, based on the combination of chaos theory and the impossibility of including every detail into your model. But one problem that doesn't exist for the weatherman is that his predictions don't alter the weather's behavior.

By contrast, in finance, economics, and politics, among many other things, the best available predictions are incorporated into everyone's future actions.

For example, the Romney and Obama campaigns are using the best predictive models they can find to look ahead at the outcome of the election in November, and then they are reacting to that. They're saying "wow, simulations say we're liable to have the election decided in Florida" and then dumping tons of resources in Florida. Or they're saying "my poll numbers and what I know is on the horizon say I have little chance of winning, time to take big gambles and swing for the fences" as with McCain and his Palin gambit. And this makes the predictions less accurate. (I imagine avoiding this problem is why all kinds of stories where some kind of oracle tells your future don't allow the future to be changed--otherwise, you mess up the oracle.)

This happens even when there's not a human adversary. Doctors benefit a lot from data mining that leads to some simple way of predicting which patients are most likely to have some complication or problem. Once they get the quickly-calculated score that says "this patient is 3x as likely as others to be back to the hospital in a week with pneumonia," they can prescribe antibiotics pre-emptively, messing up the predictive power. I know a woman with both breast cancer genes and a family history of breast cancer, who had a voluntary double mastectomy + plastic surgery to make the whole thing hard to notice. This seriously messes up the predictive model.

NOTA said...

Mr Anon:

A professor of mine in college told us this bit of doggerel, allegedly an old banker's saying (he probably made it up--he was a ham):

He who sells
what isn't his'n
must buy it back
or go to pris'n


What is your objection to shorts? I mean, you can get the same kind of investment in the futures markets, but there, someone else has to commit to the other side of the contract (I promise to buy this stock for that price on December 15, you promise to sell it to me for that price). But the guy promising to sell it doesn't have to own it now, he just has to keep enough money on deposit to make it likely he can afford to buy it. What's the difference?

My sense is always that people object to short selling for the same reason they object to anyone betting against their football team, or making investments that will prosper only if bad things happen in the world. (Be worried when all the people in the know are long body bag companies.)

One good reason to worry about this stuff is when it's possible for me to bet against someone and also to do something to make their prospects worse. There's a reasonable argument that CDS have been used this way at times, and if someone you never met takes out a big life insurance policy on you, it might be time to look to your home security system, buy a bulletproof vest, and spend more time practicing at the gun range.

NOTA said...

Another datapoint: The high speed trading firms are getting around the EMH in another way, I think. They're basically accepting that they can't out-trade the market on the basis of better understanding or knowledge at human time scales, and so instead are trying to out-trade the market based on faster algorithms. And that is a rich and interesting area of adversarial behavior--if my algorithm can predict your trades it can frontrun them, and take a little money right out of your pocket. And if your algorithm knows what I'm doing, it can fool mine into making trades that lose me money, or even frontrun *my* trades trying to take advantage of *your* trades. If I had to guess what the first truly devastating cyber attack will look like, I think it will be the assassination of some huge financial company by either compromising their high-speed trading algorithms, or by predicting them. (The widespread claim is that the Knight crash was a software error, and probably it was, but if someone was trying to wreck them, they could hardly have chosen a better tool.)

Being able to make money on average when you're much faster or bigger tnan other investors doesn't necessarily violate the EMH--the other traders know what you know, but most of them don't have the resources to compete with you. If there were some big concentration of plankton in the middle of the ocean, a bunch of hungry fish would somehow find a way to eat them. But big filter feeders (like a lot of whales) can still make a living on the low-concentration of plankton, by processing a huge volume of seawater.

Of course, in reality, the high speed trading guys know a lot that nobody else does, because they have operational experience in a domain where few other people have it. Similarly, the EMH seems to me to be a useful model in many cases, but it's clearly not true.

Dutch Boy said...

Correction: the Weather Man is not a moron, it is the Economist!

Anonymous said...

many of his sophisticated investors assumed he was doing a kind of inside-trading (front-running trades from his brokerage house), since otherwise, they knew he couldn't be getting the consistency of returns he was claiming.

It brings to mind the memorable advertising slogan: "We cheat the other guy and pass the savings on to you!"

Anonymous said...

One good reason to worry about this stuff is when it's possible for me to bet against someone and also to do something to make their prospects worse.

Gee, ya think?!?

Clutch cargo cult said...

There is so much misinformation here about short selling it would take too long to address. Suffice it to say you don't have a fucking clue how it works, the effect on the market, clearing and custody of stocks, FINRA and SEC rules. Nor how deep and liquid the US stock market is.

Jon Corzine said...

There is so much misinformation here about short selling it would take too long to address. Suffice it to say you don't have a fucking clue how it works, the effect on the market, clearing and custody of stocks, FINRA and SEC rules. Nor how deep and liquid the US stock market is.

LOSERS!!!

Anonymous said...

And don't forget, Vanguard has the option of juicing its returns by investing some portion of that $100 billion dollars to smaller hedge funds,

Um, no. The Vanguard 500 invests in S&P 500 stocks, period, aside from a little cash parked in short term bonds. You can look up their holdings. Their goal is to exactly match the S&P. And it's not clear just how they'd juice anything by sending money to hedgies.

Mr. Anon said...

"NOTA said...

Mr Anon:

What is your objection to shorts?"

My objection to it is just what I said: MY broker lending MY stock to someone else for the purpose of making money off of me by depressing it's value. What if I don't want my stock lent out - do I have that option? If not, why not? Is it mine, or is it not? When I park my car in a pay-lot, I don't expect the lot-operators to rent it out. Because it's MY car, not theirs.

Would you be cool with with your bank raiding your safe-deposit box and loaning the deed to your house to some third party so that they could sell your house in the expectation that your house value would go down, then repurchase it, and give it back to the bank to put back in your box?

The argument for short selling always seems to be: why shouldn't we f**k you over? We've always f**ked people over. F**king people over is what we do. It's good for you.

Silver said...

Dave hackensack,

Indexing is a huge business which benefits from the promotion of EMH. A lot of financial planners benefit from it too: if none of your clients expect you to find active managers that can beat the market, then you can just out them in passive investments; that frees up time for you to find more clients. It also frees up room for you to charge more in fees, since passive/index funds tend to have lower fees.

I don't disagree. But the managed fund industry has historically been more closely identified with "the financial industry" than indexers and financial planners, and it's that managed fund industry which is alleged here to be pushing EMH to further its own interests.

Mr Anon,

My objection to it is just what I said: MY broker lending MY stock to someone else for the purpose of making money off of me by depressing it's value. What if I don't want my stock lent out - do I have that option? If not, why not? Is it mine, or is it not? When I park my car in a pay-lot, I don't expect the lot-operators to rent it out. Because it's MY car, not theirs.

Shares are fungible.

Mr. Anon said...

"Silver said...

Shares are fungible."

So? Money is fungible. May I steal yours?

Anonymous said...

Shares are fungible.

Wow.

I know from copious hard-earned experience over the years that there's no way in Hades my visceral reaction would ever make it past Komment Kontrol.

So let's leave it at that: Just wow.

Silver said...

So? Money is fungible. May I steal yours?

No one's stealing your shares. If you own 500 shares then you own 500 shares. They're yours to keep or dispose of as you see fit. Short-selling doesn't alter this state of affairs.

If -- if -- short-selling causes excess downward reactions in price it also causes excess upward reactions (a "short squeeze"), meaning it all basically evens out as far as the stock-owning public is concerned. Some will lose a little more than they would have if selling while prices are depressed in excess by short-sales, others will gain more than they would have when prices are boosted by shorts covering.

I really fail to see why some of you are in such a moral panic about it all.

Anonymous said...

Excuse me, there was a weatherman in Florida about 10 years ago who WAS a moron. His forecasts were always 180 degrees out of phase with the weather that actually occurred. Or maybe he was just an evil genius, I don't know.

DaveinHackensack said...

Silver,

"I don't disagree. But the managed fund industry has historically been more closely identified with "the financial industry" than indexers and financial planners, and it's that managed fund industry which is alleged here to be pushing EMH to further its own interests."

Why indexing isn't as closely identified with "the financial industry", I have no idea. It's a huge part of it. I don't see how promoting EHM would be in the interests of the managed fund industry.

"Shares are fungible."

That's not why brokers can lend them out. They can lend them out because they're held "in street name" (i.e., in the name of the broker, to facilitate transactions), and because you agree to let them lend them out in the paperwork you sign when you open the account.

In the past, you could hold shares in certificate form, in which case they couldn't be lent out, but I don't think you always have that option today. I think it would be reasonable for investors to have the option of not allowing their shares to be loaned out, and, in return, paying slightly higher commissions or fees (since the broker won't be generating any fees from lending the shares out).

Mr. Anon said...

"Silver said...

If you own 500 shares then you own 500 shares. They're yours to keep or dispose of as you see fit."

And whoever owns them is hurt by the short sale. The broker is not just hurting one of his customers, he's hurting all of his customers who own that stock. Why should I agree to my broker doing this? Aren't brokers supposed to, at the very least, not undermine their clients?

"I really fail to see why some of you are in such a moral panic about it all."

I really fail to see why some of you are excusing an unethical business practice.

Mr. Anon said...

"DaveinHackensack said...

That's not why brokers can lend them out. They can lend them out because they're held "in street name" (i.e., in the name of the broker, to facilitate transactions), and because you agree to let them lend them out in the paperwork you sign when you open the account."

Perhaps people should not agree to such terms - demand different terms - when they open a brokerage account. Seems like an opportunity for an enterprising broker: "Morgan Lynch - we won't help you much, but at least we won't screw you over as much as those other guys".

In the past, you could hold shares in certificate form, in which case they couldn't be lent out, but I don't think you always have that option today. I think it would be reasonable for investors to have the option of not allowing their shares to be loaned out, and, in return, paying slightly higher commissions or fees (since the broker won't be generating any fees from lending the shares out)."

I.e. he won't be generating any fees by ripping off his clients. Well, that would be an improvement, however I still think I shouldn't have to pay a premium for the privelege of not being given the business by the guy I'm giving my business too. But I suppose one has to make allowances for the ethical deficiencies of stock brokers.

Silver said...

That's not why brokers can lend them out.

I didn't say it was. But he was making such a big fuss about them being HIS SHARES (dammit!) that it's worth pointing out the practice could be defended on fungibility alone.

And whoever owns them is hurt by the short sale.

As I thought I explained, you can also be helped. If the stock goes up rather than down, short-sellers have to scramble to cover their positions (their losses are potentially unlimited) and this sometimes has the effect of pushing prices even higher. Some owners of the stock will be selling out at this point in time (for whatever reason) and they receive a higher price than they otherwise would have gotten. So there's potential upside as well as downside. All in all it's just not something I think is worth worrying about all that much. I can't for the life of me figure out the "wow, just wow" reaction the "Komment Kontrol" whiner had to it.

Mr. Anon said...

"Anonymous Silver said...

All in all it's just not something I think is worth worrying about all that much."

Perhaps you just don't worry much about piddly things like ethics. And after all, sharp business practice - making a buck by any ruse possible - is what made this country great, isn't it?