Blog

Over the past 30 years, businesses have spent billions on talent assessments. Many of these are now being used to understand job candidates.

Increasingly, businesses are asking how (or if) a predictive talent acquisition strategy can include the use of pre-hire assessments. As costs of failed new hires continue to rise, recruiters and hiring managers are looking for any kind of pre-hire information to increase the probability of making a truly great hire.

How can you know if it’s a real predictive solution vs. just marketing fluff?

What is a real predictive solution?

For all of the marketing hype, Predictive Analytics boils down to three very simple steps.

1. A system reads “input” data — perhaps assessment scores or CV information.
2. The system then does some math to apply a “predictive model” to the input data.
3. Finally, the results of the model are shown as “output” data of the model — perhaps the likelihood of the candidate achieving a certain level of sales performance or another KPI. At heart, it takes “inputs” and turns them into “outputs,” or predicted business outcomes. But to build and validate a model, you need a healthy, logical set of both input and output data for that role in your company.

If you are using a talent assessment alone this is just input data. The talent assessment is just one piece of the system. There are two more pieces (as I’ve detailed above).

For most companies, their current pre-hire talent assessments are wasted data. Results are delivered in an individual report that cannot be analyzed or aggregated.

For most “legacy” talent assessments, it’s difficult or impossible to determine what positive (or negative) business inpact the assessments are having. It often comes down to the question of “how much the HR person believes the results.”

This is a bad measure of success.

But it doesn’t have to be that way. At Talent Analytics, we include talent assessment data, generated from our own proprietary assessments, as an additional data point in every predictive project. In predictive-speak, “our assessment data has proven to be a very strong independent variable for our predictive models.”

We repeatedly prove that our Talent Analytics scores, predict business performance, such as the probability of someone making their sales quota, or the probability of someone lasting in a contact center role for at least 12 months, the probability of a truck driver making accidents  … and so on for most quantifiable KPIs.

If you’d like to begin a predictive talent acquisition project using talent assessments, it can be daunting to figure out what solutions are smoke and mirrors, and what solutions will actually deliver a predictive solution.

To help, I wanted to share important factors to consider to help you sort through “pretend predictive solutions” and “real, rigorous predictive solutions” that can deliver significant bottom line results. This decision can dramatically affect your business’s bottom line.

Yes, it’s important.

Selecting a Predictive Talent Assessment solution

1. The Predictive company itself — Are you dealing with an assessment company, who is trying to learn how to be predictive? Or is it a predictive company that also uses assessment data? How long have they been doing predictive work? Are they invited to speak at predictive conferences or at basic HR conferences?

2. Their Predictive team — Ideally the company will have Data Scientists on staff as well as IO Psychologists. This is important because Data Scientists tend to utilize more modern and rigorous methods for prediction and validation. IO Psychologists tend to be focused on the instrument, while Data Scientists tend to be concerned with predictive validity and business results.
3. Are they predicting for your company, or for everyone? — There are companies that create industry benchmarks,” that is, a general performance predictions for general industry categories – such as retail sales or customer service. These predictions are significantly less accurate, because they are based on companies different from your own, with different cultures, goals, and regions. Not all “customer service” is the same. Modern computing methods enable leading providers to create and validate predictive models for your roles in your own company alone, and to continuously update the model over time.
4. Do they care about your outcome data? — Generally these solutions predict attrition or performance for a candidate or employee. Has the assessment company asked you for the attrition or KPI data for your employees in your target role? If they don’t know your employee outcomes, how can they predict your outcomes? They can’t.

Most job roles have multiple KPIs that describe performance — do they predict each of these separately? For KPIs that naturally contradict each other, e.g. speed vs. accuracy, how does the predictive solution resolve the contradiction? Just getting a “green light” isn’t good enough in many cases.

What sample size did they ask for? Real predictions require a reasonable sample to properly validate that you aren’t being fooled by randomness. If they only ask for 15 top performers, your sample is too small to create a real prediction.

5. Does the solution base predictions on outcome data or a job fit, job match or a job blueprint survey? — Data Science predicts what you ask it to predict. If you want lower attrition or higher KPIs, the models must be trained and validated with that data alone. The process looks for fact-based patterns to drive your business.

Surprisingly, many solutions don’t use this approach, but fall back to managerial bias. These solutions ask well-meaning committees of managers to list competencies that they believe are needed for success in a role.

The resulting criteria are not predictive at all — they just find candidates that match the laundry list of beliefs and biases held by that committee. Nowhere in this process is a connection to actual attrition or KPI outcomes.

Again, if the system doesn’t know about your outcomes, how can the process predict them? Start with data, not bias.

6. Does the solution use machine learning to recalibrate your predictive models? How often?
   — Business needs, role descriptions, and culture changes over time. Local labor conditions change. For example, service representatives may be incentivized to cross-sell related products, or new regulations may require new compliance to be performed. It is important to update and re-validate your predictive models 2-4 times a year to keep up to date with seen and unseen trends. Some solutions have not changed their models for 30 years; do you expect these to find great sales reps for you?
7. The new validation question: Criterion Validation? — HR has been taught to ask if the assessment is validated. The first level of validation checks whether the assessment measures are self-consistent. Continue to ask this question.

But ultimately you care about whether the assessment feeds predictions that accurately correspond to improved business outcomes. That is, are the predictions actually working? This level is called “Criterion Validation” and is a high bar that is not commonly reached by vendors.

A top tier predictive talent assessment vendor will perform Criterion Validation for the solutions several times a year – with every client. Criterion validation is the highest level of validation possible, and is the most preferred by regulatory agencies.

8. Can you easily access/download your company’s Talent Assessment Data? — Talent assessment data is a critical data set for your company. If your Talent Assessment vendor makes it difficult or impossible to access your talent assessment data – this is a good indication they are using pre-predictive technology and that they don’t appreciate that this data is your asset.

True predictive solutions know that your workforce data scientists will want to use your talent assessment data to find correlations and predictions in many areas of your business. You need to insist on easy and direct access to the raw assessment scores.

9. How easy is it to deploy the solution into the talent acquisition process and use the predictions — How much training is required? Do your talent acquisition professionals need to read long text reports, or get out a calculator to use the predictions? The complexity of a prediction should be kept out of the way of daily operations.If your team still needs to “think” about what the answer is, it is probably not a predictive solution.
10. Is there a different assessment for every role? Or one assessment with multiple predictive models? — Multiple assessments make it impossible to predict one candidate’s performance against multiple roles. This may also be a signal that you are working with an older, legacy (less predictive) talent assessment supplier.
11. Is there an answer key for their solution on the web? — For many assessments, there are answer keys and guides on how to fool or pass the test. One example is here: http://on.wsj.com/29Che0n. When you see this, it means two things:
    • A – That the test is easily fooled, lacking internal controls to prevent spoofing; and,
    • B – It means that that you are looking at an “industry benchmark” with one clear set of answers.

A data science-driven model would be custom to your role in your company, and be continuously evolving – therefore very difficult for answer keys and spoofing to catch.

12. Does the company itself (i.e. Myers Briggs) specifically tell you NOT to use their solution for hiring/talent acquisition? — Some assessments, notably the Myers Briggs survey, specifically implore users to not use the tool for talent acquisition: “It is not ethical to use the MBTI instrument for hiring or for deciding job assignments.”
13. Ask to see their company policy on employee predictive modeling, discrimination, disparate impact and fairness — It is important that a predictive solution has thought through the specific outcomes of their models and how they fit into creating fair opportunity for all applicants. In particular it is vital for the solution to satisfy or exceed any government requirements for hiring and selection.
14. Do your own (internal) data scientists approve of this predictive solution? — We recommend asking one of your own data scientists (from HR, marketing, or another area inside your own company) to accompany you in your evaluation. They know what is a rigorous approach and what is marketing fluff.
15. How does the predictive solution regularly prove that the models are working? — Ideally the company you select will be able to show you 2 to 4 times a year how your predictions are working (i.e. turnover is going down, sales are going up, calls are going up, errors are going down etc.,)

Only use a predictive model during talent acquisition if the predictions are accurate.

If they’re not, you should stop using the models. You need this feedback.

Remember: Predictive talent assessments can have a prominent place in your Predictive Talent Acquisition process, but you need to be careful in choosing the real predictive solution vs. a legacy talent assessment with a predictive marketing wrapper.