Beware: HR Analytics leads to overconfidence
Overconfidence is a term used in psychology to describe a person bias for being right. The overconfidence effect is a well-established bias and describes a person’s subjective confidence in his or her judgments is reliably greater than the objective accuracy of those judgments, especially when confidence is relatively high.
Daniel Kahneman writes in his epic book ‘thinking fast and slow’ that “neither the quantity nor the quality of the evidence count for much in subjective confidence. The confidence that individuals have in their beliefs depends mostly on the quality of the story they can tell about what they see, even if they see little…our associative system tends to settle on a coherent pattern of activation and suppress doubts and ambiguity”. What this means in practical terms is that one would assume that
How is this relevant to HR Analytics you may ask? In many important ways. Imagine that you present a fantastic ‘Predictive Employee Turnover Analysis’ to your boss who is the head of HR. You have collected a ton of data, used complex algorithms and found the following conclusion: female leaders at level 3 and 4 who have been performing among the 10% best leaders (at those levels) for three straight years and have not received a promotion are 60% likely to leave the organization within one year. Wow. And based upon this analysis you suggest a targeted effort towards that group of leaders. You calculate that this effort alone will save the organization more than $3m a year.
Your leader is truly impressed and now ask you how confident you are that your result is correct. That this group of employees are that likely of leaving and that they effort will save that much money. This is a fair question because your boss has to make a decision based in part on your findings. What do you respond?
According to the theory of the overconfidence effect, you are likely to say a very high number. Perhaps even 90%. In any case, the number will be much higher than an objective accuracy of likelihood. Frankly this number is much lower than we think it is. And this is a problem because your boss then makes a decision based on wrong input from you. If he thinks you are right, he will underestimate the risk of the project – the risk of being wrong – hence estimating a too high ROI.
Why will you make a higher estimate of the correctness of your finding? According to the theory and what I have experienced many times in reality is that because when we work with data over a stretch of time, as such a project mentioned above would imply, we are slowly developing a story in our minds. A working hypothesis if you will. And slowly we are only finding data which supports our hypothesis. This is a natural inclination. We confirm our hypothesis when we work with data.
The solution to this problem, which is quite a significant problem; always have a steering group for your projects with many types of people, continuously present your findings and assign someone to be the devil’s advocate and make it that persons responsibility to ask the stupid questions and challenge your beliefs and hypothesis’.
Overconfidence is by the way only one of many biases we carry with us and which work against us. A couple of others such as cognitive dissonance, illusory correlation, availability and anchoring. They are subtle but important and possible devastating to good analytics.
My main message here is as follows; we would like to believe that because we work with data and facts we then automatically make decisions that are more rational. We do not. We have a fantastic ability as human beings to disregard facts and make decisions based on irrational biases. If we can overcome this natural tendency then working with data and facts will make our decisions so much better. Exponentially better. But we must continue to work on our psychology.