David Brooks tries his hand at explaining regression toward the mean

Here's the coin-flipping scene from Rosencrantz and Guildenstern Are Dead,
with Gary Oldman blithely on an epic hot streak, much to the
dismay of the more scientific-minded Tim Roth.

Recently, we were kicking around Galton's paradoxical concept of regression toward mean. Galton discovered that there'a a wholly mathematical side to regression. And yet, it's also worth looking at examples of how human decisionmaking can increase or decrease the rate of regression. Regression toward the mean is such an important concept for understanding how the world works that it's worth unpacking the idea so that people don't get wrong ideas stuck in their heads.

Now David Brooks is giving regression toward the mean a try.

From the NYT:

For example, every person who plays basketball and nearly every person who watches it believes that players go through hot streaks, when they are in the groove, and cold streaks, when they are just not feeling it. 

But Thomas Gilovich, Amos Tversky and Robert Vallone found that a player who has made six consecutive foul shots has the same chance of making his seventh as if he had missed the previous six foul shots. 

There's got to be a better way to phrase this, right? Brooks is presumably talking not about two players you've never seen before, but about one player with years of NBA free throw shooting experience, who is widely acknowledged to have reached what appears to be a career plateau, and has no obvious problems contributing to a cold streak. Brooks' next paragraph is better:

When a player has hit six shots in a row, we imagine that he has tapped into some elevated performance groove. In fact, it’s just random statistical noise, like having a coin flip come up tails repeatedly. Each individual shot’s success rate will still devolve back to the player’s career shooting percentage.

If we are just talking about undefended free-throw shooting (not field goal shooting where the distance and defense varies constantly, especially in reaction to the results of the last few shots taken), an we're talking about an NBA veteran who has plateaued over several years at around, say, a 75% free throw success rate, well, then don't get too excited about him making six in a row. There's an 18 percent chance that from pure randomness, a "true" 75% shooter would make six in a row.

On the other, say you are the coach. With one second left in a tied game, the refs call a technical foul on the other team and rewards your team with one free throw. You get to pick the free throw shooter who will have one chance to win the game. Your two best available potential shooters are both veterans with career percentages of 75%. But one has made his last six free throws and the other has missed his last six. Which do you pick, or are you indifferent (as Brooks implies)?

Well, of course you go with the guy who looks like he's on a hot streak. Maybe he's not really on a hot streak, but at least he's not on a true cold streak. Maybe hot streaks are just the absence of cold streaks, but cold streaks caused by very real detrimental factors definitely exist.

There's only a 1/4096th chance of a 75% free throw shooter missing six in a row out of pure bad luck. So, missing six in a row could very well be a sign that he has a secret injury he's not telling you about, or that he's developed a hitch in his shot that he needs some extra free-throw shooting practice to work out, or that he's mentally flustered. So, as coach, you do something. The first thing you do is you don't assign the cold streak guy to shoot the technical foul shot (unless it's some mind game strategy you have of improving his self-confidence by showing your confidence in the cold shooter, but you'd better have your playoff spot clinched before you do that).

Hot streaks and cold streaks in field goal shooting (in the regular run of the game) are more complicated because of defense, which often shifts in response to streaks.

If you look at enough basketball statistics, you see that NBA players generally shoot poorly from the floor their rookie year, then maintain a fairly consistent level, except for off years (presumably caused by minor injuries, divorces, cocaine addictions, spats with coach or media, or whatever), until they hit a decline phase near the end of their careers.

On the other hand, strange things do happen. For example, veteran center Wilt Chamberlain shot .563 in his first three seasons with the Lakers, but then shot .683 over the last two seasons of his career, because his new coach, Bill Sharman, talked him into emphasizing defense like his more successful old rival, Bill Russell, and thus only take easy shots. (But, not too many conclusions should be drawn from anecdotes about Wilt. We still talk about him so much because he was so sui generis.)

But it takes a fair amount of coaching to keep NBA players near their career plateaus.

Say you have two starting guards, one (O) who is good at offense but not defense, and the other (D) is vice-versa. If they took exactly the same shots from the floor, O would make 60% and D 40%. So, of course, you devise game plans where O takes more shots, especially more of the hard shots, and D takes fewer shots, especially fewer of the hard shots. That moderates their respective shooting percentages to, say, 55% and 45%. (Under some conditions, the smart strategy is to push this all the way until both have the same shooting percentage. You want to keep arbitraging marginal advantages down to the vanishing point.

Or, consider the effects of defense on a single player. Say, Jeremy Lin starts off a game making two shots in a row against the Lakers last year. Is this just luck? Maybe. Or maybe he's being nominally defended by 37-year-old Derek Fischer, and so Lin can probably get open looks all night, and, indeed winds up with a career-high 38 points. A little while later, Lin starts a game off missing shots. Cause for panic? Or should he keep firing away because he'll be bailed out by regression toward the mean? If he's being guarded by, say, LeBron James, it's likely time for an agonizing reappraisal of the shoot-Jeremy-shoot tactics that worked so well against Fischer.

In general, if a player is 6 for 6 in the first half of a game because he owns a mismatch over his defender, at halftime the coach will probably tell him he should be shooting more. Assuming, say, a 50% breakeven point, the team would be better off if he went 10-13 in the second half rather than 6 for 6 again, because going 4 for 7 on the incremental shots would be to the team's advantage.

In general, coaches actively encourage players to regress toward their means and the team's means. If a player is missing hard shots, the coach will run plays where he gets fewer hard shots and fewer shots overall, but achieves a higher percentage because he's more limited to taking easy shots like open-court layups and offensive rebound dunks. If a player is hitting shots at a rate above the expected percentage, especially if he's enjoying a defensive mismatch, the coach will try to get him the ball more and have him take harder shots. The coach wants his hot hand to regress toward his mean, just not quite all the way. Meanwhile, the opposing coach is tearing his hair out trying to come up with a way to stop the man with the hot hand.

These kinds of defensive adjustments that encourage regression toward the mean happen at all levels from the most minutely tactical (shading a player a few inches more in one direction) to the most front-office strategic (trading for a defender to stop an archrival's best shooter). The classic paper cited above about the 1980s Philadelphia 76ers mentions that guard Andrew Toney was universally known as a "streak shooter," but there was no evidence that he went on longer streaks of makes or misses that his teammates. Instead, he was a great outside shooter (before a severe injury in his sixth season wrecked his career) who had the talent to make memorable strings of shots against the mighty Celtics in big games. (His nickname was The Boston Strangler.) To stop Toney, the Celtics traded for Dennis Johnson, one of the greatest defensive players of all time.

So, regression toward the mean just doesn't happen, it's often actively encouraged.

Other questions involve which mean a player should aim to regress toward: his natural mean or his current team's mean.

For example, in 2006 Kevin Garnett averaged 22 ppg on .526 shooting, while Kobe Bryant averaged 35 ppg on .450 shooting. Both teams had bad supporting casts (ladies and gentlemen, Laker's point guard Smush Parkerrrrrrrrrr!), but Kobe's team won 12 more games, in part because he took so many more hard shots than Garnett.

Put Garnett on a good team, however, and he's a wonderful team player. Put Kobe on a good team, and he wins a lot, but you know it will be a soap opera.