In short, a different approach is needed in most organizations. More of the same (which typically means using Kirkpatrick levels 1-5, with an emphasis on the lower levels) won’t get CLOs where they want to be—understanding what’s working (and what isn’t) in learning and development initiatives and targeting resources at the most fruitful areas for improving business results. Instead, a more modern, “analytics-enhanced” approach is necessary.
Here’s what we’ve learned about how to transform an organization’s learning analytics:
#1: Create an “authentic” learning impact evaluation by embedding it in a more holistic framework. The fundamental problem with traditional (Kirkpatrick) learning evaluation is that it’s done in a vacuum. While some Kirkpatrick evaluations definitely have their place, a more authentic way to evaluate the impact of learning is to embed it in the larger context of measuring the overall management and development of a company’s people.
In particular, companies should seek to identify opportunities to learn from “naturally occurring variations” in people’s work, learning and leadership environments. All of those variations (not just learning-related ones) should be used to assess and predict variations in business outcomes. This requires that the CLOs office collaborate with—or take the lead to create, if necessary—an analytics “center of excellence” within the organization.
#2: Stop waiting for the perfect data warehouse. Instead, create a “data hut.” At their core, data warehouses are designed for reporting—not analytics. So the sooner you realize that your organization’s data warehouse (whether current, pending, or hoped-for) is never going to enable you to do the learning analytics that you need, the better off you’ll be. To undertake analytics, you need to start by putting together heretofore disparate pieces of data (see Figure 1).
Figure 1: Build a HR Analytics “Data Hut”
This will enable you to do the following:
- Take a big step toward embedding your learning evaluation in the larger context of evaluating people management and development (step #1 above).
- Analyze why you are getting the impact that you are getting (not just what the impact is).
- Produce actionable insights about what levers to pull to create better business results through learning.
#3: Don’t let the perfect become the enemy of the good. Generalizing #2 above, in the early days of creating an analytics-enhanced learning evaluation, you will almost certainly not be able to obtain all of these disparate pieces of data and integrate them into a unified analysis file. But don’t let that become an excuse for inaction.
Start by putting two of these pieces of information together; for example, data from your LMS with employee engagement, or turnover data, or customer satisfaction data, or sales data. The important point is to begin to make progress, rather than to continue with the (unsatisfying) status quo.
#4: Choose your initial analytics project carefully. The best place to start is with a burning business issue. Examples might include one or more of the following:
- customer satisfaction problems
- lackluster sales
- high levels of regretted turnover
- failure to achieve diversity and inclusion goals
- stagnant or declining employee engagement.
Design your initial analytics project to provide actionable insight on issues that are front-and-center for senior executives, and you will find yourself in a much better place.
#5: Start under the radar. Making progress toward better, more powerful, analytics-enhanced learning measurement does not have to take a lot of time or money. Choose the right initial project (#4) without fanfare. Just go do it. Use the findings from your first project and capitalize on the enthusiasm it will engender when properly presented (see #6) in order to bootstrap up your learning analytics budget and capability.
#6: Remember: insightful reporting trumps data dumps. Learning analytics is both a science and an art. The art comes in how you present the findings from your analysis. Less is often more. So, focus on what executives need to know to drive better business results, and avoid the temptation to share every nuance and cool thing you might have learned in the course of the analysis.
#7: Use learning evaluation to improve the effectiveness of learning. It will also be tempting to use your new-found analytics capability to prove that “learning is working.” It’s far better, however, to use it to develop actionable insights for improving the business impact of learning. Doing so will generate significant returns in terms of the enhanced credibility and support that learning enjoys within your organization.
Want to learn more about improving learning evalution? Join us for the next offering of our Evaluating Learning Impact Certificate program.