ATD Blog
7 Steps to Using Analytics to Improve the Evaluation of Learning
Wed May 27 2015
Content
A recent CLO article reported that nearly 60 percent of CLOs are dissatisfied with their internal learning analytics capability. The article states that, “This reflects an ongoing trend: The state of measurement in learning and development is falling behind other areas of the business. CLOs are more dissatisfied with their organizational approach to measurement this year than last, continuing a trend of the past three years.”
A recent CLO article reported that nearly 60 percent of CLOs are dissatisfied with their internal learning analytics capability. The article states that, “This reflects an ongoing trend: The state of measurement in learning and development is falling behind other areas of the business. CLOs are more dissatisfied with their organizational approach to measurement this year than last, continuing a trend of the past three years.”
Content
In short, a different approach is needed in most organizations. More of the same (which typically means using Kirkpatrick levels 1-5, with an emphasis on the lower levels) won’t get CLOs where they want to be—understanding what’s working (and what isn’t) in learning and development initiatives and targeting resources at the most fruitful areas for improving business results. Instead, a more modern, “analytics-enhanced” approach is necessary.
In short, a different approach is needed in most organizations. More of the same (which typically means using Kirkpatrick levels 1-5, with an emphasis on the lower levels) won’t get CLOs where they want to be—understanding what’s working (and what isn’t) in learning and development initiatives and targeting resources at the most fruitful areas for improving business results. Instead, a more modern, “analytics-enhanced” approach is necessary.
Content
Here’s what we’ve learned about how to transform an organization’s learning analytics:
Here’s what we’ve learned about how to transform an organization’s learning analytics:
Content
#1: Create an “authentic” learning impact evaluation by embedding it in a more holistic framework. The fundamental problem with traditional (Kirkpatrick) learning evaluation is that it’s done in a vacuum. While some Kirkpatrick evaluations definitely have their place, a more authentic way to evaluate the impact of learning is to embed it in the larger context of measuring the overall management and development of a company’s people.
#1: Create an “authentic” learning impact evaluation by embedding it in a more holistic framework. The fundamental problem with traditional (Kirkpatrick) learning evaluation is that it’s done in a vacuum. While some Kirkpatrick evaluations definitely have their place, a more authentic way to evaluate the impact of learning is to embed it in the larger context of measuring the overall management and development of a company’s people.
Content
In particular, companies should seek to identify opportunities to learn from “naturally occurring variations” in people’s work, learning and leadership environments. All of those variations (not just learning-related ones) should be used to assess and predict variations in business outcomes. This requires that the CLOs office collaborate with—or take the lead to create, if necessary—an analytics “center of excellence” within the organization.
In particular, companies should seek to identify opportunities to learn from “naturally occurring variations” in people’s work, learning and leadership environments. All of those variations (not just learning-related ones) should be used to assess and predict variations in business outcomes. This requires that the CLOs office collaborate with—or take the lead to create, if necessary—an analytics “center of excellence” within the organization.
Content
#2: Stop waiting for the perfect data warehouse. Instead, create a “data hut.” At their core, data warehouses are designed for reporting—not analytics. So the sooner you realize that your organization’s data warehouse (whether current, pending, or hoped-for) is never going to enable you to do the learning analytics that you need, the better off you’ll be. To undertake analytics, you need to start by putting together heretofore disparate pieces of data (see Figure 1).
#2: Stop waiting for the perfect data warehouse. Instead, create a “data hut.” At their core, data warehouses are designed for reporting—not analytics. So the sooner you realize that your organization’s data warehouse (whether current, pending, or hoped-for) is never going to enable you to do the learning analytics that you need, the better off you’ll be. To undertake analytics, you need to start by putting together heretofore disparate pieces of data (see Figure 1).
Content
Figure 1: Build a HR Analytics “Data Hut”
Figure 1: Build a HR Analytics “Data Hut”
Content
Content
This will enable you to do the following:
This will enable you to do the following:
Content
Take a big step toward embedding your learning evaluation in the larger context of evaluating people management and development (step #1 above).
Take a big step toward embedding your learning evaluation in the larger context of evaluating people management and development (step #1 above).
Content
Analyze why you are getting the impact that you are getting (not just what the impact is).
Analyze why you are getting the impact that you are getting (not just what the impact is).
Content
Produce actionable insights about what levers to pull to create better business results through learning.
Produce actionable insights about what levers to pull to create better business results through learning.
Content
#3: Don’t let the perfect become the enemy of the good. Generalizing #2 above, in the early days of creating an analytics-enhanced learning evaluation, you will almost certainly not be able to obtain all of these disparate pieces of data and integrate them into a unified analysis file. But don’t let that become an excuse for inaction.
#3: Don’t let the perfect become the enemy of the good. Generalizing #2 above, in the early days of creating an analytics-enhanced learning evaluation, you will almost certainly not be able to obtain all of these disparate pieces of data and integrate them into a unified analysis file. But don’t let that become an excuse for inaction.
Content
Start by putting two of these pieces of information together; for example, data from your LMS with employee engagement, or turnover data, or customer satisfaction data, or sales data. The important point is to begin to make progress, rather than to continue with the (unsatisfying) status quo.
Start by putting two of these pieces of information together; for example, data from your LMS with employee engagement, or turnover data, or customer satisfaction data, or sales data. The important point is to begin to make progress, rather than to continue with the (unsatisfying) status quo.
Content
#4: Choose your initial analytics project carefully. The best place to start is with a burning business issue. Examples might include one or more of the following:
#4: Choose your initial analytics project carefully. The best place to start is with a burning business issue. Examples might include one or more of the following:
Content
customer satisfaction problems
customer satisfaction problems
Content
lackluster sales
lackluster sales
Content
safety
safety
Content
high levels of regretted turnover
high levels of regretted turnover
Content
failure to achieve diversity and inclusion goals
failure to achieve diversity and inclusion goals
Content
stagnant or declining employee engagement.
stagnant or declining employee engagement.
Content
Design your initial analytics project to provide actionable insight on issues that are front-and-center for senior executives, and you will find yourself in a much better place.
Design your initial analytics project to provide actionable insight on issues that are front-and-center for senior executives, and you will find yourself in a much better place.
Content
#5: Start under the radar. Making progress toward better, more powerful, analytics-enhanced learning measurement does not have to take a lot of time or money. Choose the right initial project (#4) without fanfare. Just go do it. Use the findings from your first project and capitalize on the enthusiasm it will engender when properly presented (see #6) in order to bootstrap up your learning analytics budget and capability.
#5: Start under the radar. Making progress toward better, more powerful, analytics-enhanced learning measurement does not have to take a lot of time or money. Choose the right initial project (#4) without fanfare. Just go do it. Use the findings from your first project and capitalize on the enthusiasm it will engender when properly presented (see #6) in order to bootstrap up your learning analytics budget and capability.
Content
#6: Remember: insightful reporting trumps data dumps. Learning analytics is both a science and an art. The art comes in how you present the findings from your analysis. Less is often more. So, focus on what executives need to know to drive better business results, and avoid the temptation to share every nuance and cool thing you might have learned in the course of the analysis.
#6: Remember: insightful reporting trumps data dumps. Learning analytics is both a science and an art. The art comes in how you present the findings from your analysis. Less is often more. So, focus on what executives need to know to drive better business results, and avoid the temptation to share every nuance and cool thing you might have learned in the course of the analysis.
Content
#7: Use learning evaluation to improve the effectiveness of learning . It will also be tempting to use your new-found analytics capability to prove that “learning is working.” It’s far better, however, to use it to develop actionable insights for improving the business impact of learning. Doing so will generate significant returns in terms of the enhanced credibility and support that learning enjoys within your organization.
#7: Use learning evaluation to improve the effectiveness of learning. It will also be tempting to use your new-found analytics capability to prove that “learning is working.” It’s far better, however, to use it to develop actionable insights for improving the business impact of learning. Doing so will generate significant returns in terms of the enhanced credibility and support that learning enjoys within your organization.
Content
Want to learn more about improving learning evalution? Join us for the next offering of our Evaluating Learning Impact Certificate program.
Want to learn more about improving learning evalution? Join us for the next offering of our Evaluating Learning Impact Certificate program.