Be Skeptical of Statisticians (Comparing Common Methods)
Managers, by the end of this article, you will find that many of the statistics on which you rely could have the wrong conclusions, even if the sample is random and representative, and even if the analyst uses an academically-accepted method!
If there is nothing else I impart to you, I want to give you the idea that we help our stakeholders the best by insisting on the 2nd/3rd/nth opinion.
In this case study, you will find that 3 types of very brilliant, highly paid analysts will draw the WRONG conclusion, and it is up to people like you to offer new perspective towards a MORE ACCURATE conclusion. We get the best conclusions not with lone champions but with Champion Teams.
A Quick, Provocative Case
In the following data set, we will approach a familiar question: Do abrasive people earn more money? You have probably heard the bromide, “Nice guys [and gals] finish last.” It is a topic of decades-long (if not millennia-long) research.
Although I am using fictional data in this table, they might not be far from the truth in a number of organizations.
For the most thorough experience, please download the MS Excel Spreadsheet (XLSX, 2010+)
20 employees are randomly selected from a Strategic Business Unit (SBU). We will assume for the sake of argument that this sample is large enough and representative of the SBU. Salaries are in $1,000s, and an Abrasive person is marked with a “1,” while a Non-Abrasive person is marked with a “0.” So respondent #6 is an abrasive person who earns $43,000 annually.
Looking at these data, do you think abrasive behavior is associated with higher pay? Notice that most abrasive folk are at the top, and they predominantly have high salaries. Let’s see what 3 highly-paid analysts (possibly your hires) think.
Analyst #1: The Descriptive Statistics Analyst (Most Common Type)
- Abrasive people on average earn $42K in this sample.
- Non-Abrasive people on average earn $29.5K in this sample.
- Abrasive behavior is associated with $12.5K higher pay on average.
The typical business analyst at a typical consultancy (paid $50K+) might argue that people make more money because of abrasive behavior. What about someone more rigid?
Analyst #2: The More Rigid Statistician (More Academic, Higher Paid)
- Analyst #1 has failed to account for whether that $12.5K difference is significant. If salaries are volatile (have high standard deviations), then the pay differences could simply be by random chance!
- A 2-population hypothesis test indicates that there is only 1.2% chance that Non-Abrasives make less money through sheer dumb luck.
- Therefore there is overwhelming evidence to infer that Non-Abrasives do earn less money than Abrasives.
The Statistician who conducted an hypothesis test also thinks that Abrasive behavior is associated with higher pay. What about someone who uses computer-based skills, like linear regression?
Analyst #3: The Linear Regression Analyst (Might Work in your Sales Ops or Marketing Department or Thereabouts)
- Although a low R-squared accounts for only 25% of actual data variation, a variable P-value suggests the Abrasive variable has a strong linear relationship with Salary (never mind; your analyst might be trying to sound smart).
- It is very likely that being abrasive has a relationship with salary, and on average, being abrasive is associated with a $12.5K gain in salary, all else equal.
The Linear Regression Analyst also thinks that people make more money because of being abrasive (yes, I know none of these things actually prove cause). And yes, I have read a very expensive report in which someone used linear regression in similar fashion.
Well, 3 highly paid folks with educations and different methods all converge on the same conclusion. What do you think?
Here’s a Hint: They’re All Wrong!
Although these 3 are experts of numbers and methodologies, you the manager have your own expertise. Since you might focus on things besides data, what do you think warrants further inquiry?
Could it be possible that some of those people in the SBU were paid more simply because of different professions or positions?
In this case, we simply have 10 managers and 10 non-managers. Otherwise, no major professional difference exists. It seems that 9 out of 10 managers are abrasive anyway. Are we confounded (unable to distinguish being a manager or being abrasive), or can we do something more?
Since you thought of another variable, or factor, call back Analyst #3, who works in linear regression (which deals with variables). Inform the analyst of your input.
The New Results
- Being a manager is associated with $34.7K gain in salary, strictly for being a manager.
- Being abrasive is associated with $15.3K…loss in salary!
It seems that the managers were making money “in spite of” being abrasive, not “because of” being abrasive. At the risk of having mere outliers, the nice manager (respondent #10) was paid the highest here, and the nastiest non-manager (respondent #20) was paid the least.
The Taketa Takeaway
Analysts are expensive and educated, but taking their conclusions at face value, in spite of good sampling and their generally accepted methodologies, is dangerous. We need to employ diverse perspectives, including yours, in order to help our stakeholders the best. Analysis is not about an expert’s work but about an expert team. Why not have me on board to help you create it?
Efficient But Not Effective
The average (or absent-minded) analyst is paid to be efficient at the expense of your destruction.
What do these 3 have in common?
- The United States’ War on Drugs
- Prostitution in Europe
- Kim Kardashian
Many analysts assume that if we destroy the supplier, we solve the problem (common sense dictates that you cannot buy what nobody sells).
I make no judgments here about the virtues/vices of anything; we will assume simply for the sake of argument that you have an incentive to reduce these three.
Is it easier to:
- Catch 1 illicit drug merchant or the 10-1000 users?
- Arrest 1 prostitute or the 10-100 “johnnies” (buyers)?
- Cancel a contract with Kim or ask millions of viewers/buyers to boycott?
Most interventions have focused on the supply side to be efficient. However, this efficiency has not yielded significant effectiveness (the social return on investment is egregious). We have yet to estimate effectively the incremental help from the 4-decades-running War on Drugs, though public perception is that 84% of American adults from a Rasmussen Poll believe we are losing the war. The overall cost of fighting the War on Drugs in 2011 had multiplied to 150 times what it cost in 1971 in nominal dollars (if we account for inflation, $1 in 1971 is about $6 in 2014, so real costs multiplied by about 25). Do you think we are 25 times better off in terms of drug abuse than if we did not spend on these programs?
Although programs do successfully arrest pushers and pimps, when demand is constant, it really is not that hard for another pusher/pimp/Kardashian to take the risk and enter. The input variable has an unforeseen “regenerative” property. If you killed 10 cockroaches, and 6 new ones hatched, then you really only destroyed net 4.
Nobel economics laureate Gary S. Becker and Kevin M. Murphy, both economics professors at the University of Chicago, write in the Wall Street Journal that in spite of cutting down supply, a constant demand would yield higher prices. This in turn either leads to new entrants tempted by the higher prices, or the incumbent drug gangs who can achieve higher profits get violent to avoid arrest.
While attacking supply seems efficient and common sensible, it often fails when the supply is regenerative and renewable.
[End Preview. To read the rest, please Download the PDF].
The Taketa Takeaway: We Need to Stop this Bias for Efficiency and Focus on Effectiveness. We do this by Employing Experimenters/Doers and Not Just Thinkers.
Nice Guys and Gals Finish in the Middle, for Now
Here’s our Dilemma: recent management studies claim from hard data that being a nicer (collaborative, giving) boss produces far greater net results for an organization than being tougher. However, another study indicates that being “agreeable” (friendly/compassionate from the Big 5 personality traits) is tied with lower income, more so for men than women.
So being nice pays off for the company but not for the you (the leader).
At this time, the typical advice from experts is: “Be nice but not always nice” (selectively straddle), or “Be nice but not too nice” (moderation).
This is a case of Absent-Minded Analysis. The results are reasonably supported, but the advice is not.
- If you have an agreeable personality, developing tough behaviors goes against your grain. Personalities are predominantly immutable by adulthood.
- If you should succeed in tougher behaviors, the other party might not know how genuine you are.
- If you should succeed in tougher behaviors in front of someone who has consistently seen your warm behaviors, the gain from being tough ephemerally might be small compared to the losses. For example, if a usually kind teacher gets harsh one day, the students might be too fearful for several days, creating a net loss.
Taketa’s Tertiary Solution
Instead of “fixing the person,” why not fix the pay?
- If nice guys & gals generate better long-term results, then they should generate better operating net present value or profits.
- This suggests that they should benefit more than stock-based compensation (restricted stock, options, etc.) more than their tougher counterparts.
- In addition, if nice guys & gals sustain a great team consistently, then they should remain with the company long enough to satisfy the vesting requirements.
The study regarding pay disparity might be from the lens of negotiated pay upon hiring or promotion. Tougher people do negotiate higher pay upon hiring, but nicer people tend to recapture their losses when asking for pay raises, when their superior results justify the increase. This observation has historically held with gender pay gaps in which women underbid at hiring and ask for pay raises later when results are obvious.
Granting nice leaders more pay raise opportunities helps motivate the right behaviors. Of course, stock-based compensation is an implicit pay raise if the company overall is growing.
Of course, nice leaders behave this way for reasons besides money. Famous consultant Alan Weiss writes in The Consulting Bible that real wealth is discretionary time. Nice leaders gain immensely by transforming 8 routine hours a day to 8 more pleasant hours a day. How much would a vacation cost to achieve that same gain?
If 360-degree feedback indicates that you are dealing with a nice leader, by all means, offer a higher proportion of stock-based compensation if the leader is sheepish about asking for higher pay compared to tougher counterparts. The return to the company is fabulous, and it costs you no upfront cash. If stock based compensation is not possible, make a conscious decision to allow for more pay raise opportunities, including at the end of successful projects.