Beware: HR Analytics leads to overconfidence

21/10/2014 at 14:32 5 comments

Overconfidence is a term used in psychology to describe a person bias for being right. The overconfidence effect is a well-established bias and describes a person’s subjective confidence in his or her judgments is reliably greater than the objective accuracy of those judgments, especially when confidence is relatively high.

Daniel Kahneman writes in his epic book ‘thinking fast and slow’ that “neither the quantity nor the quality of the evidence count for much in subjective confidence. The confidence that individuals have in their beliefs depends mostly on the quality of the story they can tell about what they see, even if they see little…our associative system tends to settle on a coherent pattern of activation and suppress doubts and ambiguity”. What this means in practical terms is that one would assume that

How is this relevant to HR Analytics you may ask? In many important ways. Imagine that you present a fantastic ‘Predictive Employee Turnover Analysis’ to your boss who is the head of HR. You have collected a ton of data, used complex algorithms and found the following conclusion: female leaders at level 3 and 4 who have been performing among the 10% best leaders (at those levels) for three straight years and have not received a promotion are 60% likely to leave the organization within one year. Wow. And based upon this analysis you suggest a targeted effort towards that group of leaders. You calculate that this effort alone will save the organization more than $3m a year.

Your leader is truly impressed and now ask you how confident you are that your result is correct. That this group of employees are that likely of leaving and that they effort will save that much money. This is a fair question because your boss has to make a decision based in part on your findings. What do you respond?

According to the theory of the overconfidence effect, you are likely to say a very high number. Perhaps even 90%. In any case, the number will be much higher than an objective accuracy of likelihood. Frankly this number is much lower than we think it is. And this is a problem because your boss then makes a decision based on wrong input from you. If he thinks you are right, he will underestimate the risk of the project – the risk of being wrong – hence estimating a too high ROI.

Why will you make a higher estimate of the correctness of your finding? According to the theory and what I have experienced many times in reality is that because when we work with data over a stretch of time, as such a project mentioned above would imply, we are slowly developing a story in our minds. A working hypothesis if you will. And slowly we are only finding data which supports our hypothesis. This is a natural inclination. We confirm our hypothesis when we work with data.

The solution to this problem, which is quite a significant problem; always have a steering group for your projects with many types of people, continuously present your findings and assign someone to be the devil’s advocate and make it that persons responsibility to ask the stupid questions and challenge your beliefs and hypothesis’.

Overconfidence is by the way only one of many biases we carry with us and which work against us. A couple of others such as cognitive dissonance, illusory correlation, availability and anchoring. They are subtle but important and possible devastating to good analytics.

My main message here is as follows; we would like to believe that because we work with data and facts we then automatically make decisions that are more rational. We do not. We have a fantastic ability as human beings to disregard facts and make decisions based on irrational biases. If we can overcome this natural tendency then working with data and facts will make our decisions so much better. Exponentially better. But we must continue to work on our psychology.

Entry filed under: Analytics. Tags: , , , .

How important is leadership for business success? Soccer may provide an answer. Workforce Analytics should do the rest 5 reasons HR Analytics should not be located in HR

5 Comments Add your own

  • 1. Jeyaganesh  |  23/10/2014 at 15:50

    Well, that’s so true. I second that. Hat-tip!

    Reply
  • 2. John Mattox  |  23/10/2014 at 17:27

    Great article! It is essential do apply diligence and skepticism throughout the data analysis and reporting process. If possible, test alternative hypotheses with the data or even look toward other data sets to refute (or support) the results. Thanks for raising the awareness of our often-broken decision making processes.

    Reply
  • […] Overconfidence is a term used in psychology to describe a person bias for being right. The overconfidence effect is a well-established bias and describes a person's subjective confidence in his or …  […]

    Reply
  • 4. Pasha Roberts  |  27/10/2014 at 22:54

    Yes, this would be true for results from any research or science – overconfidence is a common cognitive bias. But the solution to bias is education, not ignorance. We need to teach managers that predictions are not forecasts, and to give results with error bars.

    The unfortunate conclusion that some will make is that we must not look at our data, and that we must not think about the dynamics in our organizations. Wallowing in this mire of preconceived notions is a worse bias for hiring, industrial control, marketing, finance, or any human endeavor. It leaves the door open to subjective prejudice.

    There is more hubris in feeling that “we already know” than in the idea “we can find out more if we look.” As you say, we need to keep working on how we handle the results, not to stop thinking.

    Reply
  • […] skills, iv) visualization skills, v) strong psychological skills to understand terms such as bias and heuristics, and vi) the ability to truly understand the business. Very few in HR master these and fewer yet […]

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 1,186 other followers

Latest Tweets

Feeds


%d bloggers like this: