What’s the Business Impact of Learning?

Content

Only 37 percent of survey respondents described their learning functions as highly or very highly effective at analyzing learning-related big data, according to the 2014 report Big Data, Better Learning? How Big Data Is Affecting Organizational Learning. What's more, although a high percentage of organizations express interest in big data and analytics (with one in five having initiatives under way), only 15 percent of survey respondents characterized their organizations' analytical capabilities as effective.

Only 37 percent of survey respondents described their learning functions as highly or very highly effective at analyzing learning-related big data, according to the 2014 report Big Data, Better Learning? How Big Data Is Affecting Organizational Learning. What's more, although a high percentage of organizations express interest in big data and analytics (with one in five having initiatives under way), only 15 percent of survey respondents characterized their organizations' analytical capabilities as effective.

Content

In short, most organizations need a new approach. Simply doing more of the same—which typically means using Kirkpatrick Levels 1-4, with an emphasis on the lower levels—isn't the answer. Leading-edge talent development executives want not only an understanding of what's working (and what isn't) in talent development; they want fact-based guidance on how to target limited resources on the most promising initiatives for driving business results. This is now possible.

In short, most organizations need a new approach. Simply doing more of the same—which typically means using Kirkpatrick Levels 1-4, with an emphasis on the lower levels—isn't the answer. Leading-edge talent development executives want not only an understanding of what's working (and what isn't) in talent development; they want fact-based guidance on how to target limited resources on the most promising initiatives for driving business results. This is now possible.

Content

The more modern approach moves beyond traditional Kirkpatrick evaluation and capitalizes on advances in analytics tools and methodologies now being applied to an increasingly broad spectrum of the business landscape—including the business of talent development.

The more modern approach moves beyond traditional Kirkpatrick evaluation and capitalizes on advances in analytics tools and methodologies now being applied to an increasingly broad spectrum of the business landscape—including the business of talent development.

Content

We have been using analytics to help organizations improve their return on people for more than 20 years. Here are the key lessons we've learned about how to apply analytics to improve your learning evaluation strategy, and more importantly, to improve your return on learning investments.

We have been using analytics to help organizations improve their return on people for more than 20 years. Here are the key lessons we've learned about how to apply analytics to improve your learning evaluation strategy, and more importantly, to improve your return on learning investments.

Create an "authentic" learning impact evaluation by embedding it in a more holistic framework

Content

The fundamental problem with traditional learning evaluation is that it's done in a vacuum. While some Kirkpatrick evaluations definitely have their place, a more authentic way to evaluate the impact of learning is to embed it in the larger context of measuring the overall management and development of a company's people.

The fundamental problem with traditional learning evaluation is that it's done in a vacuum. While some Kirkpatrick evaluations definitely have their place, a more authentic way to evaluate the impact of learning is to embed it in the larger context of measuring the overall management and development of a company's people.

Content

What do we mean by authentic? We're looking for evaluation that occurs within the context of how people actually get work done. In particular, you should seek out opportunities to learn from naturally occurring variations in employees' work, learning, and leadership environments. This requires putting together disparate pieces of data—those from the "people side" of your business as well as data on business outcomes (see Figure 1 below).

What do we mean by authentic? We're looking for evaluation that occurs within the context of how people actually get work done. In particular, you should seek out opportunities to learn from naturally occurring variations in employees' work, learning, and leadership environments. This requires putting together disparate pieces of data—those from the "people side" of your business as well as data on business outcomes (see Figure 1 below).

Content

[

[

Content

Content

Seven steps to using analytics to improve the evaluation of learning.

Seven steps to using analytics to improve the evaluation of learning.

Content

These various data elements can then be linked together. This allows you, for example, to identify which managerial competencies are associated with lower regretted turnover among employees, higher sales, or better employee engagement. Similarly, data from your learning management system that identify which managers and leaders have been exposed to more or different types of development investments is an important source of naturally occurring variation in your organization. By linking that data into the analysis, you can identify the impact of these investments on a variety of factors—including the business outcomes that your CFO and CEO most care about.

These various data elements can then be linked together. This allows you, for example, to identify which managerial competencies are associated with lower regretted turnover among employees, higher sales, or better employee engagement. Similarly, data from your learning management system that identify which managers and leaders have been exposed to more or different types of development investments is an important source of naturally occurring variation in your organization. By linking that data into the analysis, you can identify the impact of these investments on a variety of factors—including the business outcomes that your CFO and CEO most care about.

Content

These analyses, in turn, provide the foundation for making a compelling business case for where and how much your organization should be investing in talent development.

These analyses, in turn, provide the foundation for making a compelling business case for where and how much your organization should be investing in talent development.

Content

Doing this type of analysis, however, is a radical departure from traditional learning evaluation. It requires that the talent development function collaborate with an analytics "center of excellence" within your organization. If one doesn't already exist, we suggest that the chief talent development officer take the lead in creating one (see sidebar for guidance).

Doing this type of analysis, however, is a radical departure from traditional learning evaluation. It requires that the talent development function collaborate with an analytics "center of excellence" within your organization. If one doesn't already exist, we suggest that the chief talent development officer take the lead in creating one (see sidebar for guidance).

Stop waiting for the perfect data warehouse

Content

At their core, traditional data warehouses are designed for reporting—not for analytics. So the sooner you realize that your organization's data warehouse (whether current, pending, or hoped-for) is never going to enable you to do the analysis that you need, the better off you'll be.

At their core, traditional data warehouses are designed for reporting—not for analytics. So the sooner you realize that your organization's data warehouse (whether current, pending, or hoped-for) is never going to enable you to do the analysis that you need, the better off you'll be.

Content

As we mentioned above, the very essence of analytics is putting together disparate pieces of data for the purpose of creating actionable insights. This will enable you to:

As we mentioned above, the very essence of analytics is putting together disparate pieces of data for the purpose of creating actionable insights. This will enable you to:

• Content

  Take a big step toward embedding your learning evaluation in the larger context of evaluating people management and development.

  Take a big step toward embedding your learning evaluation in the larger context of evaluating people management and development.

• Content

  Analyze why you are getting the impact that you are getting (not only what the impact is).

  Analyze why you are getting the impact that you are getting (not only what the impact is).

• Content

  Produce actionable insights about what levers to pull to create better business results through learning.

  Produce actionable insights about what levers to pull to create better business results through learning.

Content

Using this more holistic framework will enable you to understand and document the drivers of your organization's business outcomes (as opposed to more traditional, but less business-critical, HR or talent development outcomes). And the information and insight you'll produce will arm you with the evidence you need to convince a skeptical CFO about the business impact of investing in talent development. It's a game-changer.

Using this more holistic framework will enable you to understand and document the drivers of your organization's business outcomes (as opposed to more traditional, but less business-critical, HR or talent development outcomes). And the information and insight you'll produce will arm you with the evidence you need to convince a skeptical CFO about the business impact of investing in talent development. It's a game-changer.

Don't let the perfect become the enemy of the good

Content

The current reality is that most organizations have a very low level of maturity with regard to learning analytics. Many have lots of data and reports. But actionable, analytics-enhanced insights that can provide guidance for optimizing the allocation of tight budgets? That's typically lacking.

The current reality is that most organizations have a very low level of maturity with regard to learning analytics. Many have lots of data and reports. But actionable, analytics-enhanced insights that can provide guidance for optimizing the allocation of tight budgets? That's typically lacking.

Content

In the early days of creating an analytics-enhanced learning evaluation strategy, you will almost certainly not be able to obtain all of the disparate pieces of data noted above and successfully integrate them into a unified analysis file. But don't let that become an excuse for inaction. Instead, start small—really small.

In the early days of creating an analytics-enhanced learning evaluation strategy, you will almost certainly not be able to obtain all of the disparate pieces of data noted above and successfully integrate them into a unified analysis file. But don't let that become an excuse for inaction. Instead, start small—really small.

Content

Just pick two of these pieces of information, and then put them together. For example, you might begin with merging LMS data together with employee engagement scores (or turnover data, or customer satisfaction data, or sales data). See what you learn, go from there, and keep moving yourself and your organization in the right direction.

Just pick two of these pieces of information, and then put them together. For example, you might begin with merging LMS data together with employee engagement scores (or turnover data, or customer satisfaction data, or sales data). See what you learn, go from there, and keep moving yourself and your organization in the right direction.

Content

Starting with modest progress is much better than continuing with the (unsatisfying) status quo.

Starting with modest progress is much better than continuing with the (unsatisfying) status quo.

Choose your initial analytics project carefully

Content

Even if you're starting small, you still need to decide exactly what your initial focus should be. Almost always, the best possible place to start is with a burning business issue. Examples might include one or more of the following:

Even if you're starting small, you still need to decide exactly what your initial focus should be. Almost always, the best possible place to start is with a burning business issue. Examples might include one or more of the following:

• Content

  customer satisfaction problems

  customer satisfaction problems

• Content

  lackluster sales

  lackluster sales

• Content

  safety

  safety

• Content

  high levels of regretted turnover

  high levels of regretted turnover

• Content

  failure to achieve diversity and inclusion goals

  failure to achieve diversity and inclusion goals

• Content

  stagnant or declining employee engagement.

  stagnant or declining employee engagement.

Content

For many organizations, sales is a great place to start; every organization has data on it and every CEO (and CFO) cares about it. What explains the variation in sales productivity between your high-performing teams (or individuals) and those who are less effective? Is it the quality of the onboarding experience? Ongoing training and development? Tenure? Managerial effectiveness? Span of control? Engagement?

For many organizations, sales is a great place to start; every organization has data on it and every CEO (and CFO) cares about it. What explains the variation in sales productivity between your high-performing teams (or individuals) and those who are less effective? Is it the quality of the onboarding experience? Ongoing training and development? Tenure? Managerial effectiveness? Span of control? Engagement?

Content

And once you've done your statistical homework to identify the source(s) of naturally occurring variation within your organization that explain variations in sales, then you'll also be in good position to provide sound advice on the solution(s) that will have the biggest return on investment.

And once you've done your statistical homework to identify the source(s) of naturally occurring variation within your organization that explain variations in sales, then you'll also be in good position to provide sound advice on the solution(s) that will have the biggest return on investment.

Content

For any burning issue, there are likely to be a lot of competing and conflicting hypotheses within your organization—but little by way of evidence. Doing the type of linkage analysis outlined above will help you bring evidence to the table and enable you to help your organization avoid "management by myth," because your evidence will help point to the solutions that will have the biggest impact on sales.

For any burning issue, there are likely to be a lot of competing and conflicting hypotheses within your organization—but little by way of evidence. Doing the type of linkage analysis outlined above will help you bring evidence to the table and enable you to help your organization avoid "management by myth," because your evidence will help point to the solutions that will have the biggest impact on sales.

Content

Be forewarned, however, that the solution might not be training. But best to know that so you don't end up advocating for an increase in training/development in situations where it's unlikely to have the desired result.

Be forewarned, however, that the solution might not be training. But best to know that so you don't end up advocating for an increase in training/development in situations where it's unlikely to have the desired result.

Content

If your organization suffers from high levels of regretted turnover, that's another good place to start because that business outcome also is typically fairly easy to measure.

If your organization suffers from high levels of regretted turnover, that's another good place to start because that business outcome also is typically fairly easy to measure.

Content

The main point to keep in mind is that you should design your initial analytics project to provide actionable insight on issues that are front and center for senior executives. This will shift the conversation within your organization, create a lot of credibility for the talent development function, and in so doing, elevate its strategic role.

The main point to keep in mind is that you should design your initial analytics project to provide actionable insight on issues that are front and center for senior executives. This will shift the conversation within your organization, create a lot of credibility for the talent development function, and in so doing, elevate its strategic role.

Start under the radar

Content

Making progress toward better, more powerful, analytics-enhanced learning measurement does not have to take a great deal of time or money.

Making progress toward better, more powerful, analytics-enhanced learning measurement does not have to take a great deal of time or money.

Content

Choose the right initial project without fanfare. Just do it. What you're looking for is any critical business outcome (sales, safety, etc.) that is of real concern to executives and about which your organization has good data. By that we mean that the data are of (relatively) high quality and are measured regularly and consistently across a variety of groups within your organization.

Choose the right initial project without fanfare. Just do it. What you're looking for is any critical business outcome (sales, safety, etc.) that is of real concern to executives and about which your organization has good data. By that we mean that the data are of (relatively) high quality and are measured regularly and consistently across a variety of groups within your organization.

Content

After you've completed your analysis, begin by rolling out the findings from this first project to a carefully selected audience. Then, assuming it's properly presented and creates enthusiasm among some or all of the recipients of the results, it's time to take advantage of that enthusiasm. Seek to capitalize on it to bootstrap up your learning analytics budget and capability—while increasing your organization's return on earnings.

After you've completed your analysis, begin by rolling out the findings from this first project to a carefully selected audience. Then, assuming it's properly presented and creates enthusiasm among some or all of the recipients of the results, it's time to take advantage of that enthusiasm. Seek to capitalize on it to bootstrap up your learning analytics budget and capability—while increasing your organization's return on earnings.

Content

Remember to think about this from the perspective of the CEO and CFO. Although you want to launch your learning analytics strategy with little (if any) public proclamation, at the same time you need to have high aspirations. The old saying "Speak softly, but carry a big stick" is a good one to keep in mind.

Remember to think about this from the perspective of the CEO and CFO. Although you want to launch your learning analytics strategy with little (if any) public proclamation, at the same time you need to have high aspirations. The old saying "Speak softly, but carry a big stick" is a good one to keep in mind.

Insightful reporting trumps data dumps

Content

Senior executives do not need more reports; they probably already have too many of those. What they don't have enough of is insightful analysis that leads to actionable insights. Keep this thought front of mind as you think through how to present the findings and recommendations that emerged from your analysis.

Senior executives do not need more reports; they probably already have too many of those. What they don't have enough of is insightful analysis that leads to actionable insights. Keep this thought front of mind as you think through how to present the findings and recommendations that emerged from your analysis.

Content

Analytics is both a science and an art. The science is how you do the analysis. The art is in how you present the findings from your analysis. Less is often more. So focus on what the CEO needs to know to drive better business results, and avoid the temptation to share every nuance and cool thing you might have learned in the course of the analysis.

Analytics is both a science and an art. The science is how you do the analysis. The art is in how you present the findings from your analysis. Less is often more. So focus on what the CEO needs to know to drive better business results, and avoid the temptation to share every nuance and cool thing you might have learned in the course of the analysis.

Content

You need to give considerable thought to the visual display of findings: when to use pie charts, histograms, scatter plots, etc. Cross-section data need to be visualized differently from time-series data. Central tendencies in the data need to be delineated. Use color-coding to help make selected key points jump out visually from your tables and charts.

You need to give considerable thought to the visual display of findings: when to use pie charts, histograms, scatter plots, etc. Cross-section data need to be visualized differently from time-series data. Central tendencies in the data need to be delineated. Use color-coding to help make selected key points jump out visually from your tables and charts.

Content

Check out this simple chart (Figure 2); can you tell instantly which team needs the most work? Remember that clever analysis without equally clever presentation of the analysis is likely to fail to have the hoped-for impact.

Check out this simple chart (Figure 2); can you tell instantly which team needs the most work? Remember that clever analysis without equally clever presentation of the analysis is likely to fail to have the hoped-for impact.

Content

Finally, don't underestimate the power of a well-chosen story. Although analysis is the language of business, it is stories that ultimately touch people's hearts. A presentation that speaks both to executives' heads and their hearts is likely to be a powerful one.

Finally, don't underestimate the power of a well-chosen story. Although analysis is the language of business, it is stories that ultimately touch people's hearts. A presentation that speaks both to executives' heads and their hearts is likely to be a powerful one.

Use learning evaluation to improve the effectiveness of learning

Content

In closing, we leave you with this word of caution: Avoid the temptation to use your newfound analytics capability to prove that "learning is working." It's far better to use analytics to develop actionable insights for improving the business impact of learning. Doing so will generate significant returns in terms of the enhanced credibility and support that learning enjoys within your organization.

In closing, we leave you with this word of caution: Avoid the temptation to use your newfound analytics capability to prove that "learning is working." It's far better to use analytics to develop actionable insights for improving the business impact of learning. Doing so will generate significant returns in terms of the enhanced credibility and support that learning enjoys within your organization.

How to Become Better at Talent Analytics

Content

In many talent development departments, the heat is on—or soon will be. Senior executives are increasingly demanding that you provide actionable insights for driving better business results through targeted improvements in the management and development of people.

In many talent development departments, the heat is on—or soon will be. Senior executives are increasingly demanding that you provide actionable insights for driving better business results through targeted improvements in the management and development of people.

Content

There is a growing demand that the talent development function provide analysis about how to cost-effectively improve outcomes such as sales productivity, customer service, managerial effectiveness, employee well-being, and diversity and inclusion. So how do you get from where you are currently to where you need to be?

There is a growing demand that the talent development function provide analysis about how to cost-effectively improve outcomes such as sales productivity, customer service, managerial effectiveness, employee well-being, and diversity and inclusion. So how do you get from where you are currently to where you need to be?

Content

In the end, advanced talent analytics capabilities can't simply be parachuted into your organization; you have to build it through the following steps:

In the end, advanced talent analytics capabilities can't simply be parachuted into your organization; you have to build it through the following steps:

1. Content

   Create organizational and broad-based executive support by beginning to produce insightful, succinct reports and analyses.

   Create organizational and broad-based executive support by beginning to produce insightful, succinct reports and analyses.

2. Content

   Develop an analytics strategy that's aligned with your organization's overall business strategy. Produce business intelligence and actionable insights that help leaders at all levels in your organization drive better business results through focused, targeted, and achievable improvements in the management and development of people.

   Develop an analytics strategy that's aligned with your organization's overall business strategy. Produce business intelligence and actionable insights that help leaders at all levels in your organization drive better business results through focused, targeted, and achievable improvements in the management and development of people.

3. Content

   Grow the size and skills of your analytics staff within talent development (or find a trusted external analytics consultant). Don't focus exclusively on technical skills. The business acumen, collaboration, consulting, and presentation skills of your analysts are all critical elements as well.

   Grow the size and skills of your analytics staff within talent development (or find a trusted external analytics consultant). Don't focus exclusively on technical skills. The business acumen, collaboration, consulting, and presentation skills of your analysts are all critical elements as well.

4. Content

   Expand the scope of your analytics initiatives to encompass all of the essential aspects of people management and development. This includes:

   Expand the scope of your analytics initiatives to encompass all of the essential aspects of people management and development. This includes:

• Content

  recruiting and onboarding

  recruiting and onboarding

• Content

  learning and development

  learning and development

• Content

  performance and career management

  performance and career management

• Content

  rewards and recognition

  rewards and recognition

• Content

  engagement and retention.

  engagement and retention.

Content

Where to start? If you're like many talent development departments, you have limited resources, but your executives have high expectations. So you need a plan that enables you to focus on the right things and to get them done in the right order. And that, of course, will be shaped by your organization's current talent analytics capability.

Where to start? If you're like many talent development departments, you have limited resources, but your executives have high expectations. So you need a plan that enables you to focus on the right things and to get them done in the right order. And that, of course, will be shaped by your organization's current talent analytics capability.

Content

We've designed a quick (and free)]( https://files.astd.org/CTDO-Images/Spring2016/ProveltSB1-L.png ) online assessment tool to objectively measure your current level of talent analytics maturity along with recommendations on the most important areas for focus in light of your relative strengths and weaknesses. The assessment takes less than 10 minutes to complete, so why not give it a try right now?

We've designed a quick (and free)](https://files.astd.org/CTDO-Images/Spring2016/ProveltSB1-L.png) online assessment tool to objectively measure your current level of talent analytics maturity along with recommendations on the most important areas for focus in light of your relative strengths and weaknesses. The assessment takes less than 10 minutes to complete, so why not give it a try right now?

Content

Read more from CTDO magazine : Essential talent development content for C-suite leaders.

Read more from CTDO magazine: Essential talent development content for C-suite leaders.