Seven steps to using analytics to improve the evaluation of learning.
In short, most organizations need a new approach. Simply doing more of the same—which typically means using Kirkpatrick Levels 1-4, with an emphasis on the lower levels—isn't the answer. Leading-edge talent development executives want not only an understanding of what's working (and what isn't) in talent development; they want fact-based guidance on how to target limited resources on the most promising initiatives for driving business results. This is now possible.
The more modern approach moves beyond traditional Kirkpatrick evaluation and capitalizes on advances in analytics tools and methodologies now being applied to an increasingly broad spectrum of the business landscape—including the business of talent development.
We have been using analytics to help organizations improve their return on people for more than 20 years. Here are the key lessons we've learned about how to apply analytics to improve your learning evaluation strategy, and more importantly, to improve your return on learning investments.
Create an "authentic" learning impact evaluation by embedding it in a more holistic framework
The fundamental problem with traditional learning evaluation is that it's done in a vacuum. While some Kirkpatrick evaluations definitely have their place, a more authentic way to evaluate the impact of learning is to embed it in the larger context of measuring the overall management and development of a company's people.
What do we mean by authentic? We're looking for evaluation that occurs within the context of how people actually get work done. In particular, you should seek out opportunities to learn from naturally occurring variations in employees' work, learning, and leadership environments. This requires putting together disparate pieces of data—those from the "people side" of your business as well as data on business outcomes (see Figure 1 below).
These various data elements can then be linked together. This allows you, for example, to identify which managerial competencies are associated with lower regretted turnover among employees, higher sales, or better employee engagement. Similarly, data from your learning management system that identify which managers and leaders have been exposed to more or different types of development investments is an important source of naturally occurring variation in your organization. By linking that data into the analysis, you can identify the impact of these investments on a variety of factors—including the business outcomes that your CFO and CEO most care about.
These analyses, in turn, provide the foundation for making a compelling business case for where and how much your organization should be investing in talent development.
Doing this type of analysis, however, is a radical departure from traditional learning evaluation. It requires that the talent development function collaborate with an analytics "center of excellence" within your organization. If one doesn't already exist, we suggest that the chief talent development officer take the lead in creating one (see sidebar for guidance).
Stop waiting for the perfect data warehouse
At their core, traditional data warehouses are designed for reporting—not for analytics. So the sooner you realize that your organization's data warehouse (whether current, pending, or hoped-for) is never going to enable you to do the analysis that you need, the better off you'll be.
As we mentioned above, the very essence of analytics is putting together disparate pieces of data for the purpose of creating actionable insights. This will enable you to:
- Take a big step toward embedding your learning evaluation in the larger context of evaluating people management and development.
- Analyze why you are getting the impact that you are getting (not only what the impact is).
- Produce actionable insights about what levers to pull to create better business results through learning.
Using this more holistic framework will enable you to understand and document the drivers of your organization's business outcomes (as opposed to more traditional, but less business-critical, HR or talent development outcomes). And the information and insight you'll produce will arm you with the evidence you need to convince a skeptical CFO about the business impact of investing in talent development. It's a game-changer.
Don't let the perfect become the enemy of the good
The current reality is that most organizations have a very low level of maturity with regard to learning analytics. Many have lots of data and reports. But actionable, analytics-enhanced insights that can provide guidance for optimizing the allocation of tight budgets? That's typically lacking.
In the early days of creating an analytics-enhanced learning evaluation strategy, you will almost certainly not be able to obtain all of the disparate pieces of data noted above and successfully integrate them into a unified analysis file. But don't let that become an excuse for inaction. Instead, start small—really small.
Just pick two of these pieces of information, and then put them together. For example, you might begin with merging LMS data together with employee engagement scores (or turnover data, or customer satisfaction data, or sales data). See what you learn, go from there, and keep moving yourself and your organization in the right direction.
Starting with modest progress is much better than continuing with the (unsatisfying) status quo.
Choose your initial analytics project carefully
Even if you're starting small, you still need to decide exactly what your initial focus should be. Almost always, the best possible place to start is with a burning business issue. Examples might include one or more of the following:
- customer satisfaction problems
- lackluster sales
- safety
- high levels of regretted turnover
- failure to achieve diversity and inclusion goals
- stagnant or declining employee engagement.
For many organizations, sales is a great place to start; every organization has data on it and every CEO (and CFO) cares about it. What explains the variation in sales productivity between your high-performing teams (or individuals) and those who are less effective? Is it the quality of the onboarding experience? Ongoing training and development? Tenure? Managerial effectiveness? Span of control? Engagement?
And once you've done your statistical homework to identify the source(s) of naturally occurring variation within your organization that explain variations in sales, then you'll also be in good position to provide sound advice on the solution(s) that will have the biggest return on investment.
For any burning issue, there are likely to be a lot of competing and conflicting hypotheses within your organization—but little by way of evidence. Doing the type of linkage analysis outlined above will help you bring evidence to the table and enable you to help your organization avoid "management by myth," because your evidence will help point to the solutions that will have the biggest impact on sales.
Be forewarned, however, that the solution might not be training. But best to know that so you don't end up advocating for an increase in training/development in situations where it's unlikely to have the desired result.
If your organization suffers from high levels of regretted turnover, that's another good place to start because that business outcome also is typically fairly easy to measure.
The main point to keep in mind is that you should design your initial analytics project to provide actionable insight on issues that are front and center for senior executives. This will shift the conversation within your organization, create a lot of credibility for the talent development function, and in so doing, elevate its strategic role.
Start under the radar
Making progress toward better, more powerful, analytics-enhanced learning measurement does not have to take a great deal of time or money.
Choose the right initial project without fanfare. Just do it. What you're looking for is any critical business outcome (sales, safety, etc.) that is of real concern to executives and about which your organization has good data. By that we mean that the data are of (relatively) high quality and are measured regularly and consistently across a variety of groups within your organization.
After you've completed your analysis, begin by rolling out the findings from this first project to a carefully selected audience. Then, assuming it's properly presented and creates enthusiasm among some or all of the recipients of the results, it's time to take advantage of that enthusiasm. Seek to capitalize on it to bootstrap up your learning analytics budget and capability—while increasing your organization's return on earnings.
Remember to think about this from the perspective of the CEO and CFO. Although you want to launch your learning analytics strategy with little (if any) public proclamation, at the same time you need to have high aspirations. The old saying "Speak softly, but carry a big stick" is a good one to keep in mind.
Insightful reporting trumps data dumps
Senior executives do not need more reports; they probably already have too many of those. What they don't have enough of is insightful analysis that leads to actionable insights. Keep this thought front of mind as you think through how to present the findings and recommendations that emerged from your analysis.
Analytics is both a science and an art. The science is how you do the analysis. The art is in how you present the findings from your analysis. Less is often more. So focus on what the CEO needs to know to drive better business results, and avoid the temptation to share every nuance and cool thing you might have learned in the course of the analysis.
You need to give considerable thought to the visual display of findings: when to use pie charts, histograms, scatter plots, etc. Cross-section data need to be visualized differently from time-series data. Central tendencies in the data need to be delineated. Use color-coding to help make selected key points jump out visually from your tables and charts.
Check out this simple chart (Figure 2); can you tell instantly which team needs the most work? Remember that clever analysis without equally clever presentation of the analysis is likely to fail to have the hoped-for impact.
Finally, don't underestimate the power of a well-chosen story. Although analysis is the language of business, it is stories that ultimately touch people's hearts. A presentation that speaks both to executives' heads and their hearts is likely to be a powerful one.
Use learning evaluation to improve the effectiveness of learning
In closing, we leave you with this word of caution: Avoid the temptation to use your newfound analytics capability to prove that "learning is working." It's far better to use analytics to develop actionable insights for improving the business impact of learning. Doing so will generate significant returns in terms of the enhanced credibility and support that learning enjoys within your organization.
How to Become Better at Talent Analytics
In many talent development departments, the heat is on—or soon will be. Senior executives are increasingly demanding that you provide actionable insights for driving better business results through targeted improvements in the management and development of people.
There is a growing demand that the talent development function provide analysis about how to cost-effectively improve outcomes such as sales productivity, customer service, managerial effectiveness, employee well-being, and diversity and inclusion. So how do you get from where you are currently to where you need to be?
In the end, advanced talent analytics capabilities can't simply be parachuted into your organization; you have to build it through the following steps:
- Create organizational and broad-based executive support by beginning to produce insightful, succinct reports and analyses.
- Develop an analytics strategy that's aligned with your organization's overall business strategy. Produce business intelligence and actionable insights that help leaders at all levels in your organization drive better business results through focused, targeted, and achievable improvements in the management and development of people.
- Grow the size and skills of your analytics staff within talent development (or find a trusted external analytics consultant). Don't focus exclusively on technical skills. The business acumen, collaboration, consulting, and presentation skills of your analysts are all critical elements as well.
- Expand the scope of your analytics initiatives to encompass all of the essential aspects of people management and development. This includes:
- recruiting and onboarding
- learning and development
- performance and career management
- rewards and recognition
- engagement and retention.
Where to start? If you're like many talent development departments, you have limited resources, but your executives have high expectations. So you need a plan that enables you to focus on the right things and to get them done in the right order. And that, of course, will be shaped by your organization's current talent analytics capability.
We've designed a quick (and free) online assessment tool to objectively measure your current level of talent analytics maturity along with recommendations on the most important areas for focus in light of your relative strengths and weaknesses. The assessment takes less than 10 minutes to complete, so why not give it a try right now?
Read more from CTDO magazine: Essential talent development content for C-suite leaders.
