Advertisement
Advertisement
ATD Blog

7 Steps to Using Analytics to Improve the Evaluation of Learning

Wednesday, May 27, 2015
Advertisement

analytics_shutterstock_122951974.jpg
A recent CLO article reported that nearly 60 percent of CLOs are dissatisfied with their internal learning analytics capability. The article states that, “This reflects an ongoing trend: The state of measurement in learning and development is falling behind other areas of the business. CLOs are more dissatisfied with their organizational approach to measurement this year than last, continuing a trend of the past three years.” 

In short, a different approach is needed in most organizations. More of the same (which typically means using Kirkpatrick levels 1-5, with an emphasis on the lower levels) won’t get CLOs where they want to be—understanding what’s working (and what isn’t) in learning and development initiatives and targeting resources at the most fruitful areas for improving business results. Instead, a more modern, “analytics-enhanced” approach is necessary. 

Here’s what we’ve learned about how to transform an organization’s learning analytics: 

#1: Create an “authentic” learning impact evaluation by embedding it in a more holistic framework. The fundamental problem with traditional (Kirkpatrick) learning evaluation is that it’s done in a vacuum. While some Kirkpatrick evaluations definitely have their place, a more authentic way to evaluate the impact of learning is to embed it in the larger context of measuring the overall management and development of a company’s people. 

In particular, companies should seek to identify opportunities to learn from “naturally occurring variations” in people’s work, learning and leadership environments. All of those variations (not just learning-related ones) should be used to assess and predict variations in business outcomes. This requires that the CLOs office collaborate with—or take the lead to create, if necessary—an analytics “center of excellence” within the organization.

#2: Stop waiting for the perfect data warehouse. Instead, create a “data hut.” At their core, data warehouses are designed for reporting—not analytics. So the sooner you realize that your organization’s data warehouse (whether current, pending, or hoped-for) is never going to enable you to do the learning analytics that you need, the better off you’ll be. To undertake analytics, you need to start by putting together heretofore disparate pieces of data (see Figure 1). 

Figure 1: Build a HR Analytics “Data Hut”

Advertisement

Bassi-Analytics-Data-Hut.jpg

 

This will enable you to do the following:

  • Take a big step toward embedding your learning evaluation in the larger context of evaluating people management and development (step #1 above).
  • Analyze why you are getting the impact that you are getting (not just what the impact is).
  • Produce actionable insights about what levers to pull to create better business results through learning. 

#3: Don’t let the perfect become the enemy of the good. Generalizing #2 above, in the early days of creating an analytics-enhanced learning evaluation, you will almost certainly not be able to obtain all of these disparate pieces of data and integrate them into a unified analysis file. But don’t let that become an excuse for inaction. 

Advertisement

Start by putting two of these pieces of information together; for example, data from your LMS with employee engagement, or turnover data, or customer satisfaction data, or sales data. The important point is to begin to make progress, rather than to continue with the (unsatisfying) status quo. 

#4: Choose your initial analytics project carefully. The best place to start is with a burning business issue. Examples might include one or more of the following:

  • customer satisfaction problems
  • lackluster sales
  • safety
  • high levels of regretted turnover
  • failure to achieve diversity and inclusion goals
  • stagnant or declining employee engagement. 

Design your initial analytics project to provide actionable insight on issues that are front-and-center for senior executives, and you will find yourself in a much better place. 
#5: Start under the radar. Making progress toward better, more powerful, analytics-enhanced learning measurement does not have to take a lot of time or money. Choose the right initial project (#4) without fanfare. Just go do it. Use the findings from your first project and capitalize on the enthusiasm it will engender when properly presented (see #6) in order to bootstrap up your learning analytics budget and capability. 

#6: Remember: insightful reporting trumps data dumps. Learning analytics is both a science and an art. The art comes in how you present the findings from your analysis. Less is often more. So, focus on what executives need to know to drive better business results, and avoid the temptation to share every nuance and cool thing you might have learned in the course of the analysis. 

#7: Use learning evaluation to improve the effectiveness of learning. It will also be tempting to use your new-found analytics capability to prove that “learning is working.” It’s far better, however, to use it to develop actionable insights for improving the business impact of learning. Doing so will generate significant returns in terms of the enhanced credibility and support that learning enjoys within your organization.

Want to learn more about improving learning evalution? Join us for the next offering of our Evaluating Learning Impact Certificate program.

About the Author

Laurie Bassi is the CEO of McBassi & Company, a leader in using behavioral economics to improve organizational performance. Laurie is a prolific author, with more than 90 published papers and books, including Good Company: Business Success in the Worthiness Era (Berrett-Koehler) and  The HR Analytics Handbook (Reed Business). She holds a PhD in economics from Princeton University, an MS in industrial and labor relations from Cornell University, and a BS in mathematics from Illinois State University. Follow Laurie on Twitter @goodcompanybook.

About the Author

Dan McMurrer is chief analyst at McBassi & Company. During the past 15 years, he has worked in the world of HR analytics, designing and deploying assessment tools for understanding the unique strengths and weaknesses of organizations’ work and learning environments, and analyzing how those are linked to business results. Prior to co-founding McBassi & Company, Dan worked in research positions at the Urban Institute, Saba Software, the American Society for Training & Development, and the U.S. Department of Labor. Dan also worked for more than 10 years as chief research officer at Bassi Investments, a groundbreaking investment company that generated above-market returns by investing in companies with superior human capital management. He is also the co-author of three books, including the  HR Analytics Handbook, as well as multiple articles. He holds a bachelor’s degree in politics from Princeton University and a master’s degree in public policy from Georgetown University.

3 Comments
Sign In to Post a Comment
Thanks for sharing such a valuable information. Above topics which have discussed for analytics will helps you to analyse the analytics easily.
Sorry! Something went wrong on our end. Please try again later.
Thanks for sharing such a valuable information. Above topics which have discussed for analytics will helps you to analyse the analytics easily.
Sorry! Something went wrong on our end. Please try again later.
Thanks for sharing such a valuable information. Above topics which have discussed for analytics will helps you to analyse the analytics easily.
Sorry! Something went wrong on our end. Please try again later.
Sorry! Something went wrong on our end. Please try again later.