A century ago or so, each college had its own admissions tests, but the inefficiencies of that became apparent. Eventually the country came up with two competing college admissions tests -- the SAT and the the ACT -- with colleges free to pile on eccentric essay questions or whatever they feel like. A few colleges may still have their own unique entrance exam (All Souls College * at Oxford has a famous one), but in the U.S. customized admissions tests are quite rare.
Now you might think that if national tests are good enough for Yale, they ought to be good enough for, say, the New Haven Fire Department. And there are companies that publish tests for fire departments across the country to use. Yet, as we saw with the 2009 Ricci Supreme Court case, nobody thought it odd that New Haven had spent $100,000 on a consulting firm to dream up a customized test just for New Haven. That's pretty common in the fraught world of hiring firemen.
In fact, much of the defense against Mr. Ricci's reverse discrimination lawsuit consisted of the allegation that the test wasn't customized enough: the consulting firm had borrowed a question from earlier tests that referred to "downtown" when there is no downtown in New Haven (or something like that -- all I remember is that the lack of local customization of one question was a big deal in the national press for a couple of weeks in 2009.)
In fact, much of the defense against Mr. Ricci's reverse discrimination lawsuit consisted of the allegation that the test wasn't customized enough: the consulting firm had borrowed a question from earlier tests that referred to "downtown" when there is no downtown in New Haven (or something like that -- all I remember is that the lack of local customization of one question was a big deal in the national press for a couple of weeks in 2009.)
How, exactly, is the New Haven FD so different from, say, the New Canaan FD that New Haven needs to spend one hundred grand on its own test? Caltech, Yeshiva, the Air Force Academy, and Smith are probably more different from each other than most municipal fire departments are different from each other, and yet they all find the SAT and ACT helpful.
Now there are some nationally available general purpose job hiring tests like the Wonderlic IQ test famously used by the NFL. Still, a glance at the 17,000 words of federal guidelines on whether or not the EEOC will come down on you like a ton of bricks if you use formal testing as part of the hiring process demonstrates clearly that the answer to any legal question involving testing and hiring is Maybe. (Or Consult Your Attorneys or You'll Find Out.)
If you started your own college and then, as it got more successful, you announced that you were going to mandate the SAT and/or ACT as part of the admissions process, nobody would blink an eye. But if you start a business that grows big enough to get on the EEOC's radar, can you assume that you can just use some battle-tested national test? Or do you have to validate that the national test specifically works at your not-so-unique company?
Maybe.
Over the last century, industrial/organizational psychologists have put a lot of effort into understanding the mysteries of testing. One of their major findings is that you don't need all that many different tests. The kind of things that can be measured well by testing aren't very unique to one college or one company or one job or whatever. Test optimization runs into pretty severe diminishing returns.
And yet, almost nobody outside of the profession is aware of this major discovery of 20th Century social science. Why not?
One reason is because a large number of the people who understand this -- professional I/O psychologists -- are employed to come up with new tests that will do what 100 years of I/O research says can't be done.
One reason is because a large number of the people who understand this -- professional I/O psychologists -- are employed to come up with new tests that will do what 100 years of I/O research says can't be done.
Hey, it's a living.
------------------------
* By the way, once every century the learned dons of All Souls College stage a drunken torchlight parade while singing the "Mallard Song" about a giant duck devoured by their predecessors in the 15th Century. They are led by the specially chosen "Lord Mallard" (typically a distinguished classicist or a future Archbishop of Canterbury) who waves a duck on a pole. The last Hunting the Mallard was in 2001, while the next one will be in 2101.
------------------------
* By the way, once every century the learned dons of All Souls College stage a drunken torchlight parade while singing the "Mallard Song" about a giant duck devoured by their predecessors in the 15th Century. They are led by the specially chosen "Lord Mallard" (typically a distinguished classicist or a future Archbishop of Canterbury) who waves a duck on a pole. The last Hunting the Mallard was in 2001, while the next one will be in 2101.
IQ is not the only useful psychometric. For the majority of positions, knowing if someone is psychologically suited to the type of work they are applying for is equally critical. Ironically, for many positions, having a high IQ is a determent. Einstein would not have made a good assembly line worker. He would have immediately gotten bored, day dreamed, and would have quit as soon as something better came along. Job interviewees can be hard to gauge. They are usually prepared to give the presentation of their lives because the pressure is on and will say whatever it is they need to say to land the job. Even the smartest, most aggressive recruit will fall back to the level of comfort and competence after a couple of weeks once the daily grind sets in.
ReplyDeleteThis is why it is so critical to fire bad hires immediately. Nothing will kill a company quicker. Pay and employee benefits only help a company retain employees; but sometimes that is the last thing you want to do if the employee is not a good fit. If the employee is not intrinsically motivated they won't work out, irrespective of how smart they are.
the consulting firm had borrowed a question from earlier tests that referred to "downtown" when there is no downtown in New Haven (or something like that -- all I remember is that the lack of customization of one question was a big deal in the national press for a couple of weeks in 2009.)
ReplyDeleteHow, exactly, is the New Haven FD so different from, say, the New Canaan FD that New Haven needs to spend one hundred grand on its own test?
Not to be picky or anything, but there certainly is a downtown New Haven, in fact it's reasonably large.
As for New Canaan, like many towns in the area its fire department consists of a small paid staff, probably no more than 10 to 15 people, supplemented by volunteers. Hiring for the paid jobs would be very infrequent to say the least. As for the volunteers, I am quite sure the department will take just about anyone, it must be very difficult to recruit volunteers in an upscale community with a high percentage of commuters.
Peter
ReplyDeleteIsn't it likely that the Diversity-Grievance Industry is what keeps a lot of test designers employed? After all, as the twentieth century's two landmark tests have always functioned optimally to identify optimal job candidates, why else is there a need for evermore novel tests?
I'm an I/O, and I generally strongly agree with you on testing issues. But I'm not 100% with you on your reason for why this wasted test development work goes on.
ReplyDeleteOne thing you are in fact exactly right about is that we don't need specialized ability tests, and that developing them is for show (to the DOJ, the EEOC, people who might sue companies, etc.). The part I have to disagree with is that the reason this nonsense goes on is because I/Os are making money developing all these specialized tests, so they want to keep the facts a secret.
First, we couldn't keep it secret; others can read too, and many companies have I/Os on staff who would have an incentive to keep external consulting costs down. Also, the company I work for, which publishes a lot of off-the-shelf ability tests, would be happy if people just accepted that any heavily g-loaded assessment is good enough, no custom test development or validation needed. A lot of companies are afraid to use those tests now due to these kinds of misunderstandings or legal risks, and we'd sell a lot more tests if those risks for employers did not exit. I guess there are some independent consultants who like all the wasted extra work, but it's not all I/Os.
The real driver for this wasted test development effort is that lawyers, judges, and the EEOC and DOJ -- who are either ignorant of the facts about testing or chose to ignore them -- like to see custom test development work with local job analyses and local validation. So the extra cost is worth the reduced legal and PR risks for large companies, even though all competent I/Os as well as most HR departments commissioning the work know it’s a waste of time.
I presented a paper at a conference a while back outlining a method to do something called validity transport, which means showing that a validation study done in one context applied to another context in a way consistent with the Uniform Guidelines. I started out the paper by saying, in effect, that I realized this approach was a complete waste of time from the standpoint of actually saying whether the test is valid in that second context (if it's a g-loaded test, it will be) and that this method is all window dressing for the lawyers. No one batted an eyelid, everyone understood. The method was a hit, and I know of at least two major consulting firms that adopted it.
Thanks. Most informative.
ReplyDeleteIf the EEOC didn't exist, how many more employers would use tests for hiring purposes? Not many. People want to hire friends. And that's in the best case scenario. In dysfunctional societies people only want to hire relatives and members of their own tribe/caste/clan.
ReplyDeleteThe government and shareholders of large corporations have an interest in enforcing merit-based hiring, but even if they were allowed to push for it by reigning ideology, they would still encounter resistance from government and corporate bureaucracies, whose members always want to hire friends instead.
IQ is important, but as Steve himself has said, it's not everything. It doesn't tell you how extroverted a person is. Lots of jobs require extroversion and some even require its opposite. The relationship between IQ and getting along with others isn't linear either. The 100 - 130 group is better at it than either idiots or really smart people. Above 100 IQ honesty and objectivity are probably largely unrelated to intelligence.
When government and corporate bureaucrats make hiring decisions, they can underperform IQ tests by hiring their dumb loser friends and relatives, but they can also occasionally outperform tests by rejecting people with the wrong psychological profiles for the job.
People are often friends with others of similar ability. So some companies (e.g., Cantor Fitzgerald) have prospered by hiring lots of friends & relatives of employees.
DeleteHey, it's a living.
ReplyDeleteThat's an expression that is rarely heard nowadays. Other examples would be "it's a free country" and "there oughta be a law". The latter two are obsolete because it's not a very free country and there are laws against almost everything; "it's a living" is seldom heard because most English-speakers are embarrassed by menial labor. Reading it from Steve takes me back to The Flintstones and cartoon animals that performed mundane mechanical household tasks.
The Flintstones: It's a Living
It just occurred to me: How do other countries handle this?
ReplyDeleteI have never heard or read about how all our agony over police and fire hiring is a unique American experience.
It is, isn't it?
Pretty amazing that nobody has pointed out that this is voluntary pain and how other First World countries select people for these positions.
At least two things are going on with selection and hiring:
ReplyDelete1. Not even trying
2. Incompetence
By not even trying, I mean that selection of people on grounds of functional competence/ 'meritocracy' has almost disappeared. Ability is a constraint in selection, but no longer the major motivation. Since people are not trying to select the most competent, they won't get them.
The age of meritocracy (i.e. selection primarily by functional ability) was - we can now see - merely a blip in the mid twentieth century. However, we are no longer honest enough to admit the fact.
And the evidence is they don't get them - if you do what leftists do and look at the outcomes - such as sex ratios, you can see that across the board elite selection is not choosing the best. Near fifty percent male female sex ratios in elite college admissions is conclusive that general intelligence is not the major factor in selection.
But incompetence is a factor - because (presumably due to the secular decline in intelligence) nowadays few people are smart enough (or have a sufficient attention span) to understand the explanations; although this is exaggerated by the political correctness induced confusion which gets greater with increasing education - so that most of the people smart enough to understand IQ, have also had their minds destroyed by political correctness.
For example, it seems clear that most testers (and most economists) don't realize that cognitive ability is IQ - and that a cognitive ability test is (insofar as it is valid) an IQ test.
This goes up to the top. A while ago I looked at the Cambridge University specimen examination papers which are available on line (this has good claim to be the most academically selective university in Europe). It is obvious that the Cambridge testers don't know what they are doing - the tests are a mixture of traditional IQ with subject-specific made-up garbage of completely conjectural validity.
http://www.study.cam.ac.uk/undergraduate/apply/tests/
This happens because Cambridge is closely monitored for its sex/ class admissions (not so much race) - and the near 50 percent selection of women shows that strong preferences for women are in place.
If Cambridge was seriously trying to get the best, and if they knew their stuff - which clearly they do not, they would compensate for the decline in the g-loading of school examinations by quick, simple, straightforward generic IQ testing.
*
On the other hand, there is still some good IQ type testing going on rather quietly between individual providers and clients, under the radar - for example in relation to educational progress monitoring (in situations where people really want to know the true answers).
Again the testers tend to avoid using or referencing the IQ word, but it is clear that the testers know what they are doing.
one reason iq testing is treated with skepticism is that the highest scorers tend to be weird and unimpressive, especially to women. i've always kind of assumed that IQ lacks cachet because the wrong kinds of people win the game: if pasty-faced bookworms are the best at something, that something can't be too important, can it?
ReplyDeletei think prestigious schools and firms are selecting for "interpersonal impressiveness", and IQ has a fairly weak correlation with impressiveness from 125-140 and probably an inverse one thereafter. there aren't many geniuses who haven't invested their attention in a socially debilitating way.
The old saying is that "He who pays the piper calls the tune". It's still as operative today as it was when it was first devised, which I guess was sometime in the middle ages.
ReplyDeleteThe upshot is, of course, employers employ who they *want* to employ. Screw everything else. Ignore the EEOC ignore all the garbage and the the 'fixed' tests.
If your face fits the employ hires you - if not, then it's tough titty. Such is the reality of free-wheeling American capitalism that is exulted in these pages.
Which employers, apart from 'public' institutions and private employers that rely on the government, take all this EEOC bullcrap seriously? - as you must know, in the vast majority of openings in the small private business sector, (where most Americans happen to be employed), most jobs aren't even advertised in the first place. Yes, the big corps are targets for the EEOC, but there are ways and means around it. Note how the auto industry has abandoned Detroit and located to the rural south - why is that I ask, (hint, hint).
Corporate America puts on a big show of tokenism, like a Potemkin village, but do the tokens really pull the strings. Look what happened to all those 'diverse' finance CEOS after the financial f*ck-up.
The second axiom is that every big corp. has one objective and one objective only - its own survival. If AAs are in the main incompetent, then the firm will hire them to comply, but will 'cope' with them with all sorts of ways. It's another hidden tax, basically.
Government and government controlled jobs are another kettle of fish altogether. Here the payer of the piper *wants* as much 'diversity' as possible, regardless of the consequences - which to the government are of minor importance compared to tribal politics - hence whte men are basically cut-out of the pensionable, cushy shindig.
At the risk of being a bore, the 'golden rule' is that 'he who has the gold makes the rules'.
@Bruce Charlton
ReplyDeleteThank you for your very interesting posts.
May I add that along with 'meritocracy', the whole middle class is also a blip in the mid-late 20th century ? It is now fading away, it's disappearance partly camouflaged by a mountain of debt
I'd think it would be hard to construct a test to cull the better candidates given the unusual demands of the job. Testing for a certain minimally accepted cognitive ability and upper body stress would be easy.
ReplyDeleteBut how to test for bravery and conscientiousness? There's the greatest challenge.
Which employers, apart from 'public' institutions and private employers that rely on the government, take all this EEOC bullcrap seriously?
ReplyDeleteYou'd be surprised.
For that matter, who does not "rely on the government" in this day and age? The Cathedral is pretty ubiquitous.
OT, but previously covered at iSteve = The Decline of Wikipedia
ReplyDelete“The biggest issue is editor diversity,”
...
Jimmy Wales, now just a regular Wikipedian but still influential with editors and the Wikimedia Foundation, dismisses suggestions that the project will get worse. But he believes it can’t get significantly better without an influx of new editors who have different interests and emphases. “When you look at the article on the USB standard, you see it is really amazing and core to our competency as a tech geek community, but look at an entry about somebody famous in sociology, or Elizabethan poets, and it is quite limited and short and could be improved,” he says. “That’s not likely to happen until we diversify the community.” Wales hopes Visual Editor will do that by attracting people who are similar to those already editing the site but have interests beyond the male- and tech-centric—as he puts it, “geeks who are not computer geeks.” But he admits to worrying that making Wikipedia simpler to edit could instead confirm that the project doesn’t appeal to people who are not computer geeks.
Once again, it's not good enough for something great to be created. It must be made "better" through diversity. Like Wikipedia would have done better if there were more articles about Elizabethan poets.
The most remarkable feature of Einstein was his perfect fit to theoretical physics. He'd never have made an experimental physicist nor a mathematician, for instance, never mind a Master of the Mint.
ReplyDeleteThis fit was not achieved by HR Professionals, but by leaving one young man to find his own way. The portraits on his office wall were of the earlier theoretical physicists of whom he thought best: Newton, Maxwell and Faraday. They didn't benefit from HR professionals either.
I know it's more work, but these posts would be a lot better if you threw in some citations to controversial points. It would make me a lot more comfortable sharing your articles with people.
ReplyDeleteYes, the big corps are targets for the EEOC, but there are ways and means around it. Note how the auto industry has abandoned Detroit and located to the rural south - why is that I ask, (hint, hint).
ReplyDeleteYou don't know what you're talking about.
Federal EEOC rules and laws also apply in southern US states. Car factories in Dixie still have to hire their quota of Affirmative Action peepul.
The American auto industry is moving to southern states which have right to work laws. Why? To avoid the United Awful Workers Union.
“”””So some companies (e.g., Cantor Fitzgerald) have prospered by hiring lots of friends & relatives of employees.”””
ReplyDeleteI wonder which side, Cantor or Fitzgerald are most likely to be hired?
Off-topic and it's probably already been posted but worth repeating if so
ReplyDeletehttp://www.dailymail.co.uk/news/article-2513653/Sexual-violence-gang-neighbourhoods-like-war-zones-girls-young-11-groomed-raped.html
"Britain's worst gang hit neighbourhoods are seeing levels of sexual violence as bad as in war zones, it was claimed today."
The exact same thing has happened in every bluecollar US neighborhood that has been "diversified" over the last 60 years. The scale and intensity varies by ethnic group but the basic pattern is always the same - bluecollar "diversity" equals youth gangs engaging in the stoneage politics of using gang violence to dominate territory and monopolize breeding age females.
The number of victims in the US over the last 60 years will number in the millions but the media covered it all up.
I should add it doesn't stop when the white population has been cleansed as by then the gangs have become a dominant part of the culture so they carry on as before. So although 95% of the victims are white at the beginning of the process by the time the white population is all gone 95% of the victims are black or hispanic.
ReplyDeleteThere is a testing issue specific to firefighters that seems to always be forgotten.
ReplyDeleteI speak of the ability to carry a man out of a burning building. I think the physical tests that bore on this ability were phased out when they tried to bring more women into firefighting.
Both men and women are getting heavier for reasons which remain cloudy. So it would seem logical that our physical standards for firefighters should increase not decrease. If a woman can't carry me while unconscious from smoke inhalation out of a burning building what good is she? Why should I have to pay taxes to pay her a salary? This is more important than any ability to conjugate a verb or rotate a matrix.
Albertosaurus
@Prof. Woland: " Ironically, for many positions, having a high IQ is a determent."
ReplyDeleteTrue. Police and Fire Departments seek to screen out and eliminate high-IQ job applicants. There was one big federal case in CT regarding this several years ago.
while the next one will be in 2101
ReplyDeleteIn theory.
@pat: "... the ability to carry a man out of a burning building."
ReplyDeleteIt's no longer necessary for firefighters to have this ability. 98% of all calls are medical not fires. As fire safety, public awareness, building code, and construction improved over the last forty years, fire departments had to find work. They did that by putting private ambulance companies out of business.
The fact that 98% of all calls (proof?) are medical is irrelevant to the fact that the other 2% aren't, and do require tremendous physical strength.
ReplyDeleteIronic that this especially stupid comment came on the day that we had a really terrible rail accident. All of the first responders were men, including divers. The accident was right near the Harlem River (and the Hudson). Response crews included waterborne personnel on pontoon boats. All men.
"You can't fool nature."
@d(ick): Set aside your homoerotic fetish for men in uniform. Yes, having upper body strength can be useful for some of the rare events that firewhiners face, but that doesn't mean they all must be gym rats to the exclusion of common sense and sufficient intelligence.
ReplyDeleteDave Pinsen:
ReplyDeleteIn re Cantor Fitzgerald: you have absolutely no idea how right you are.
DJF:
ReplyDeleteNeither family is involved. The mover and shaker is a guy named Lutnick, who started out as a personal assistant to Cantor and, when his boss went into a long, irreversible coma, managed to dominate and take over the firm, eventually winning in court over Cantor's family (wife) and many of the firm's executives.
A datum, or anecdote: Norfolk Southern, the railroad, required me to take an online IQ test this year as a precondition to flying me in for an engineering interview.
ReplyDelete