Do You Measure Up?

Content

Here’s the truth: I wasn’t measuring up when it came to consistently measuring and demonstrating the business value of our L&D programs. And I’m fairly confident that I’m not the only learning or talent development leader who has felt that way.

Here’s the truth: I wasn’t measuring up when it came to consistently measuring and demonstrating the business value of our L&D programs. And I’m fairly confident that I’m not the only learning or talent development leader who has felt that way.

Content

In fact, according to the Association for Talent Development’s 2019 research report Effective Evaluation: Measuring Learning Programs for Success , only 40 percent of talent development professionals surveyed believe their learning evaluation efforts helped them meet their organization’s business goals. And only 16 percent measured the return on investment of learning programs.

In fact, according to the Association for Talent Development’s 2019 research report Effective Evaluation: Measuring Learning Programs for Success, only 40 percent of talent development professionals surveyed believe their learning evaluation efforts helped them meet their organization’s business goals. And only 16 percent measured the return on investment of learning programs.

Content

So, while I may not be alone, as the chief learning officer of an urban, pediatric healthcare system, it is my responsibility to get better at measuring the learning impact on business outcomes. Here’s how I did that and the lessons I have learned along the way.

So, while I may not be alone, as the chief learning officer of an urban, pediatric healthcare system, it is my responsibility to get better at measuring the learning impact on business outcomes. Here’s how I did that and the lessons I have learned along the way.

The path to measurement

Content

My journey to get better at measurement started in 2014, when my team and I codified the first iteration of our “Measurement and Reporting Guidelines.” The document outlined an easy-to-follow, three-step process to categorize a project, plan the measurement, and determine reporting needs.

My journey to get better at measurement started in 2014, when my team and I codified the first iteration of our “Measurement and Reporting Guidelines.” The document outlined an easy-to-follow, three-step process to categorize a project, plan the measurement, and determine reporting needs.

Content

The first step is to categorize a project as high, mid, or low profile based on strategic alignment, visibility, and audience size. The second step is to determine the measurement requirements based on those categories.

The first step is to categorize a project as high, mid, or low profile based on strategic alignment, visibility, and audience size. The second step is to determine the measurement requirements based on those categories.

Content

High-profile projects are measured at the business outcomes or return on investment level, while low-profile projects can be measured solely at the mastery level (or Level 2 in Kirkpatrick’s model). The final step suggests reporting methods, whom to report to, and the frequency of reporting based on the project category.

High-profile projects are measured at the business outcomes or return on investment level, while low-profile projects can be measured solely at the mastery level (or Level 2 in Kirkpatrick’s model). The final step suggests reporting methods, whom to report to, and the frequency of reporting based on the project category.

Content

As we rolled out the measurement guidelines, it became clear that my team and I needed upskilling first. Many of our organization’s learning professionals have a first career in a healthcare-related specialty, and some do not have academic preparation in adult education.

As we rolled out the measurement guidelines, it became clear that my team and I needed upskilling first. Many of our organization’s learning professionals have a first career in a healthcare-related specialty, and some do not have academic preparation in adult education.

Content

We created a system of development workshops, one-on-one coaching, and consulting practice integration to help us develop the skills necessary to be better at measuring project and program outcomes. We made the case for change to our team and provided exemplars of effective measurement plans as well as a library of measurement options for each of Kirkpatrick’s levels. From a leadership perspective, we had everything in place: clear expectations plus the skills and tools needed to execute. We were well on our way.

We created a system of development workshops, one-on-one coaching, and consulting practice integration to help us develop the skills necessary to be better at measuring project and program outcomes. We made the case for change to our team and provided exemplars of effective measurement plans as well as a library of measurement options for each of Kirkpatrick’s levels. From a leadership perspective, we had everything in place: clear expectations plus the skills and tools needed to execute. We were well on our way.

Measuring success

Content

During the past several years, my team and I have achieved some success. For example, we developed a great measurement plan for a large suicide-screening training initiative for the nursing staff. Through automated reports from the electronic medical record, we could quickly identify that inpatient nurses were performing the suicide screening for greater than 95 percent of the new admissions.

During the past several years, my team and I have achieved some success. For example, we developed a great measurement plan for a large suicide-screening training initiative for the nursing staff. Through automated reports from the electronic medical record, we could quickly identify that inpatient nurses were performing the suicide screening for greater than 95 percent of the new admissions.

Content

We also saw that the emergency department was struggling with the new process, which prompted us to further analyze and modify the process for that team.

We also saw that the emergency department was struggling with the new process, which prompted us to further analyze and modify the process for that team.

Content

We began to hear feedback as well. For example, a teenager came into the hospital for ankle surgery and shared their thoughts of self-harm with the nurse during the screening. The nurse was able to get the individual the help and resources they needed. Without that screening, the teen may not have received the necessary support.

We began to hear feedback as well. For example, a teenager came into the hospital for ankle surgery and shared their thoughts of self-harm with the nurse during the screening. The nurse was able to get the individual the help and resources they needed. Without that screening, the teen may not have received the necessary support.

Content

All those metrics show our value in changing behaviors and helping the organization meet its healing mission.

All those metrics show our value in changing behaviors and helping the organization meet its healing mission.

Content

Another success was the annual publication of a learning dashboard. It was largely focused on activity metrics, such as learning hours and formats, but we started publishing outcomes for high-profile projects and programs too. The most recent iteration includes compliance, learner reaction, and a few outcomes for our diversity, equity, and inclusion work as well as for our emerging leader program.

Another success was the annual publication of a learning dashboard. It was largely focused on activity metrics, such as learning hours and formats, but we started publishing outcomes for high-profile projects and programs too. The most recent iteration includes compliance, learner reaction, and a few outcomes for our diversity, equity, and inclusion work as well as for our emerging leader program.

The learning curve

Content

Yet, although my team and I have made great strides, the journey has not been seamless and painless. As a leader, I have failed along the way. But those failures have been opportunities to learn and grow.

Yet, although my team and I have made great strides, the journey has not been seamless and painless. As a leader, I have failed along the way. But those failures have been opportunities to learn and grow.

Content

For example, I had completely failed to appreciate the learning curve and nuance to the talent analytics space. All the industry articles made talent analytics sound so easy. Business literature could consolidate the process down to four simple steps or five innovative ways, but putting meaningful measurement into practice was much more complicated.

For example, I had completely failed to appreciate the learning curve and nuance to the talent analytics space. All the industry articles made talent analytics sound so easy. Business literature could consolidate the process down to four simple steps or five innovative ways, but putting meaningful measurement into practice was much more complicated.

Content

First, my team and I didn’t own the data. Even our engagement survey data was “owned” outside our team, so simply getting to the data involved some work and building bridges.

First, my team and I didn’t own the data. Even our engagement survey data was “owned” outside our team, so simply getting to the data involved some work and building bridges.

Content

Second, our systems didn’t talk to each other easily, and merging learning and program data with other people measures has been an ongoing challenge.

Second, our systems didn’t talk to each other easily, and merging learning and program data with other people measures has been an ongoing challenge.

Content

Further, I had the stark realization that I had an entire new lexicon and body of knowledge to learn with talent analytics and systems integrations. Elements such as data mining, data visualizations, and xAPI integrations were blind spots where I hadn’t appreciated my own ignorance.

Further, I had the stark realization that I had an entire new lexicon and body of knowledge to learn with talent analytics and systems integrations. Elements such as data mining, data visualizations, and xAPI integrations were blind spots where I hadn’t appreciated my own ignorance.

The bandwidth dilemma

Content

Another key failure in our measurement journey was the bandwidth dilemma.

Another key failure in our measurement journey was the bandwidth dilemma.

Content

Measurement takes time and effort. So often, individuals are pushed quickly to the next project or deliverable.

Measurement takes time and effort. So often, individuals are pushed quickly to the next project or deliverable.

Content

I frequently found myself asking, “Is the juice worth the squeeze?” Is pushing on measurement follow-up worth risking the next project timeline or team member engagement?

I frequently found myself asking, “Is the juice worth the squeeze?” Is pushing on measurement follow-up worth risking the next project timeline or team member engagement?

Content

I vividly recall the look a direct report gave me when I asked for an outcomes report on a program after she’d just listed about a dozen items she was actively working on. That was not my proudest leadership moment.

I vividly recall the look a direct report gave me when I asked for an outcomes report on a program after she’d just listed about a dozen items she was actively working on. That was not my proudest leadership moment.

Content

Even our learning dashboard was a bandwidth drain. Condensing data from multiple sources into an engaging, visual presentation was a tedious manual process.

Even our learning dashboard was a bandwidth drain. Condensing data from multiple sources into an engaging, visual presentation was a tedious manual process.

Content

Working on automating our data integrations and visualizations using business intelligence software took away time from keeping our dashboard updated and vice versa. Consequently, the learning dashboard that is still posted is a year outdated (although the pandemic takes some of that blame).

Working on automating our data integrations and visualizations using business intelligence software took away time from keeping our dashboard updated and vice versa. Consequently, the learning dashboard that is still posted is a year outdated (although the pandemic takes some of that blame).

Content

Leaders face hard choices, and frequently, the bandwidth dilemma is one of the hardest to navigate.

Leaders face hard choices, and frequently, the bandwidth dilemma is one of the hardest to navigate.

Content

For me and my team, many times our measurement failures fell into three categories: measurement plans were delayed, altered, or forgotten due to bandwidth challenges; they were killed or altered by technology or data challenges; or over time, as we raced to keep up with the pace of change, we experienced a delay in our learning team identifying meaningful outcomes at the onset of projects.

For me and my team, many times our measurement failures fell into three categories: measurement plans were delayed, altered, or forgotten due to bandwidth challenges; they were killed or altered by technology or data challenges; or over time, as we raced to keep up with the pace of change, we experienced a delay in our learning team identifying meaningful outcomes at the onset of projects.

Lessons learned

Content

As I reflect on mine and my team’s continuing measurement journey, I have identified several lessons that, hopefully, other leaders can use in their own measurement efforts.

As I reflect on mine and my team’s continuing measurement journey, I have identified several lessons that, hopefully, other leaders can use in their own measurement efforts.

Content

Talent leaders need to upskill themselves in the discipline of data science. As I mentioned, getting more sophisticated in program measurement is not as easy as “four simple steps.”

Talent leaders need to upskill themselves in the discipline of data science. As I mentioned, getting more sophisticated in program measurement is not as easy as “four simple steps.”

Content

Options for your personal upskilling include everything from free vendor-supplied webinars and materials to inexpensive massive open online courses in analytics from such providers as Coursera and edX.

Options for your personal upskilling include everything from free vendor-supplied webinars and materials to inexpensive massive open online courses in analytics from such providers as Coursera and edX.

Content

Leading business schools offer more expensive but deeper academic certificate programs in business analytics. And industry associations, including the Association for Talent Development, offer meaningful certificate programs.

Leading business schools offer more expensive but deeper academic certificate programs in business analytics. And industry associations, including the Association for Talent Development, offer meaningful certificate programs.

Content

Following execution, leaders need to provide space for measurement activities. The bandwidth dilemma is real—plan accordingly.

Following execution, leaders need to provide space for measurement activities. The bandwidth dilemma is real—plan accordingly.

Content

Ditch the grand, multipronged measurement plans for a realistic plan that focuses on a few key measures. Build measurement activities into project plans, and allot hours for post-implementation measurement where possible.

Ditch the grand, multipronged measurement plans for a realistic plan that focuses on a few key measures. Build measurement activities into project plans, and allot hours for post-implementation measurement where possible.

Content

Watch the cart and the horse. I tried to evolve my team’s measurement and reporting capabilities before I had a fuller understanding of the discipline and before we had the systems and data capabilities to make it easy.

Watch the cart and the horse. I tried to evolve my team’s measurement and reporting capabilities before I had a fuller understanding of the discipline and before we had the systems and data capabilities to make it easy.

Content

In many ways, I contributed to a level of disdain for higher-level measurement because it was so hard to combine data into real insights. I put the cart before the horse.

In many ways, I contributed to a level of disdain for higher-level measurement because it was so hard to combine data into real insights. I put the cart before the horse.

Content

Get your systems in place to the greatest extent possible before you place unrealistic expectations on your team.

Get your systems in place to the greatest extent possible before you place unrealistic expectations on your team.

Content

It’s a journey, not a destination. Our measurement journey has been complete with potholes and delays along the way. But we have made progress and learned other valuable lessons.

It’s a journey, not a destination. Our measurement journey has been complete with potholes and delays along the way. But we have made progress and learned other valuable lessons.

Content

For example, my team and I now place a higher value on different types of measurement such as anecdotes and stories that add color to the numbers. Don’t forget to pause on your journey and assess how far you’ve come and what you’ve learned.

For example, my team and I now place a higher value on different types of measurement such as anecdotes and stories that add color to the numbers. Don’t forget to pause on your journey and assess how far you’ve come and what you’ve learned.

Content

It’s true—none of us always measure up. There are certainly many things my team and I have to be proud of, and we will continue to reflect and learn from our shortcomings.

It’s true—none of us always measure up. There are certainly many things my team and I have to be proud of, and we will continue to reflect and learn from our shortcomings.

Content

For next steps, I will keep looking for ways to be better and to use the lessons I have encountered to learn and move the needle forward. I feel no shame in failure when pushing into uncertain areas unless I repeat mistakes I should have learned from previous experiences.

For next steps, I will keep looking for ways to be better and to use the lessons I have encountered to learn and move the needle forward. I feel no shame in failure when pushing into uncertain areas unless I repeat mistakes I should have learned from previous experiences.

Content

Read more from CTDO magazine : Essential talent development content for C-suite leaders.

Read more from CTDO magazine: Essential talent development content for C-suite leaders.