ATD Blog
Mining for Insights
Thu Dec 14 2023
Brought to you by
Content
How a Canadian energy company upskilled its learning team and built a model program evaluation in nine months.
How a Canadian energy company upskilled its learning team and built a model program evaluation in nine months.
Content
Learning initiatives fuel strategic progress. Leaders need actionable, timely information about program impact to keep their goals on track. However, many organizations lack effective learning evaluation programs that could provide these insights. Of the 556 respondents to TiER1 Performance’s 2023 Learning Trends Report , just 5 percent report that they generally use research-supported evaluation practices, while 64 percent feel unable to do the kinds of learning evaluation that they want to do.
Learning initiatives fuel strategic progress. Leaders need actionable, timely information about program impact to keep their goals on track. However, many organizations lack effective learning evaluation programs that could provide these insights. Of the 556 respondents to TiER1 Performance’s 2023 Learning Trends Report, just 5 percent report that they generally use research-supported evaluation practices, while 64 percent feel unable to do the kinds of learning evaluation that they want to do.
Content
What will help us better our evaluation process and create successful learning programs? Let’s learn from a Canadian energy company that developed both the capabilities of its learning team and a suite of powerful evaluation tools in less than nine months.
What will help us better our evaluation process and create successful learning programs? Let’s learn from a Canadian energy company that developed both the capabilities of its learning team and a suite of powerful evaluation tools in less than nine months.
Breaking Ground
Content
In mid-2022, the learning team at this company approached our company, TiER1, for help with evaluating a key program intended to provide foundational training for shovel supervisors in one of their mines.
In mid-2022, the learning team at this company approached our company, TiER1, for help with evaluating a key program intended to provide foundational training for shovel supervisors in one of their mines.
Content
To get the most from the partnership, we adopted a hybrid model where the company’s team could learn from and work alongside Will Thalheimer, PhD, MBA, an internationally recognized learning evaluation expert and author of an award-winning book on performance-focused learner surveys . Through this hybrid model, the company’s team would gain three main takeaways:
To get the most from the partnership, we adopted a hybrid model where the company’s team could learn from and work alongside Will Thalheimer, PhD, MBA, an internationally recognized learning evaluation expert and author of an award-winning book on performance-focused learner surveys. Through this hybrid model, the company’s team would gain three main takeaways:
Content
Actionable, high-impact insights to improve the pilot course
Actionable, high-impact insights to improve the pilot course
Content
The skills to repeat this process for other courses
The skills to repeat this process for other courses
Content
Valuable experience with this learning evaluation approach to share with others across the company
Valuable experience with this learning evaluation approach to share with others across the company
Digging In
Content
To become full partners in evaluation, the core team spent time building shared values and a common language.
To become full partners in evaluation, the core team spent time building shared values and a common language.
Content
We conducted research and interviews with key stakeholders, instructors, and recent learners to better understand the needs of shovel supervisors and the goals for the course. Digging in further, the team engaged in a series of six two-hour collaborative workshops to develop foundational skills in the design of metrics, studies, surveys, and scenarios and to apply these skills to the project context.
We conducted research and interviews with key stakeholders, instructors, and recent learners to better understand the needs of shovel supervisors and the goals for the course. Digging in further, the team engaged in a series of six two-hour collaborative workshops to develop foundational skills in the design of metrics, studies, surveys, and scenarios and to apply these skills to the project context.
Content
We then developed appropriate metrics and measuring tools, using the Learning Transfer Evaluation Model (LTEM) developed by Will Thalheimer to guide our work. LTEM helps teams make decisions about the kinds of data to collect for a multidimensional picture of a learning program’s effectiveness. For instance, to measure job performance, we identified key behavioral indicators (KBIs) that would show whether course objectives were really being applied on the job.
We then developed appropriate metrics and measuring tools, using the Learning Transfer Evaluation Model (LTEM) developed by Will Thalheimer to guide our work. LTEM helps teams make decisions about the kinds of data to collect for a multidimensional picture of a learning program’s effectiveness. For instance, to measure job performance, we identified key behavioral indicators (KBIs) that would show whether course objectives were really being applied on the job.
Content
We planned a time-series study, in which we would compare the data before training and at multiple points after the training, to assess the impact the training had on the learners.
We planned a time-series study, in which we would compare the data before training and at multiple points after the training, to assess the impact the training had on the learners.
Content
What we wanted to measure:
What we wanted to measure:
Content
Job performance
Job performance
Content
Task competence
Task competence
Content
Decision-making skills
Decision-making skills
Content
Learner perceptions of performance impact
Learner perceptions of performance impact
Content
How we measured it:
How we measured it:
Content
Anonymous self-evaluations of KBIs
Anonymous self-evaluations of KBIs
Content
Capability checklists
Capability checklists
Content
Scenario questions
Scenario questions
Content
Performance-focused learner surveys
Performance-focused learner surveys
Content
Developing the evaluation instruments was an iterative process, involving multiple stages of team discussion, review, and revision. For example, we validated scenario questions with both experts and nonexperts. Listening to experts “think aloud” through their answers helped us ensure that the questions assessed decision making based on the intended learning points. Testing with nonexperts helped us identify and remove wording that could make the answers easier to guess.
Developing the evaluation instruments was an iterative process, involving multiple stages of team discussion, review, and revision. For example, we validated scenario questions with both experts and nonexperts. Listening to experts “think aloud” through their answers helped us ensure that the questions assessed decision making based on the intended learning points. Testing with nonexperts helped us identify and remove wording that could make the answers easier to guess.
Staying Grounded
Content
The project faced challenges, but the team persevered in excavating insights and expanding their learning evaluation capability.
The project faced challenges, but the team persevered in excavating insights and expanding their learning evaluation capability.
Content
We ran into obstacles with implementing the time-series study as envisioned, as the number of learners entering the program dropped significantly right before our planned start for the study. Because our sample size was too small to perform meaningful data analysis, we ultimately derived the most reliable insights from research-inspired benchmarking based on our initial interviews and observations.
We ran into obstacles with implementing the time-series study as envisioned, as the number of learners entering the program dropped significantly right before our planned start for the study. Because our sample size was too small to perform meaningful data analysis, we ultimately derived the most reliable insights from research-inspired benchmarking based on our initial interviews and observations.
Content
Despite this setback, we still achieved several important outcomes:
Despite this setback, we still achieved several important outcomes:
Content
Identifying strengths and opportunities for improvement in the shovel supervisor training program
Identifying strengths and opportunities for improvement in the shovel supervisor training program
Content
Developing evaluation tools that can be used in the future for additional mine training programs
Developing evaluation tools that can be used in the future for additional mine training programs
Content
Informing the learning evaluation strategy for programs at the company
Informing the learning evaluation strategy for programs at the company
Content
Gaining skill and confidence in evaluating learning programs that we were able to transfer to other contexts
Gaining skill and confidence in evaluating learning programs that we were able to transfer to other contexts
Fueling Discovery
Content
Research-based evaluation methods such as LTEM help us leverage learning and performance experiences of all kinds to uncover insights and achieve goals. This project illustrates how collaboration and learning science can be combined to create a solution that delivers high performance, develops teams, and drives business results.
Research-based evaluation methods such as LTEM help us leverage learning and performance experiences of all kinds to uncover insights and achieve goals. This project illustrates how collaboration and learning science can be combined to create a solution that delivers high performance, develops teams, and drives business results.
Content
Learning evaluation projects come in many shapes and sizes. Talk to TiER1 about your goals, and we can help design the right solution for your team.
Learning evaluation projects come in many shapes and sizes. Talk to TiER1 about your goals, and we can help design the right solution for your team.