Ginger Swart

With modern technology, accurately measuring the impact of virtual training programs is a very real possibility. When used properly, this kind of measurement catalyzes change in industries like healthcare, financial services, consulting, manufacturing, and others that rely heavily on effective training programs. But the ability to measure doesn’t guarantee success. You also need to know what metrics to track, how to avoid bias, and how to get buy-in from stakeholders and instructors.Let’s examine how best to use data and automation to save instructors and administrators time and improve the learner experience.3 Top Challenges to Effective AnalysisWhile there are several things to consider when implementing the most effective measurement program for virtual training, let’s take a closer look at three top challenges that we recommend tackling first, as well as our tips for clearing these hurdles with ease.1. Difficulty quantifying engagement. Look for a virtual training measurement platform that collects and analyzes the data for you, assessing factors like tardiness, the number of minutes instructors speak versus learners, and the number of learners raising their hands and using customized reactions, to name a few. Once you start measuring, set benchmarks based on your historical data, setting ambitious yet realistic goals for improvement. 2. Unintentional bias using qualitative measurement.Those responsible for measuring the effectiveness of training programs shouldn’t completely ignore qualitative methods of judging engagement, but must also introduce data to reinforce the findings of qualitative measurement to ensure accuracy.3. Lack of instructor focus on measuring engagement. Put plainly, the onus should not be solely on instructors here. The introduction of technology and data can be a welcome solution. Teachers can focus on teaching while reliable tech measures the results in the background, helping tell a complete story.Use Results to Improve InstructionTo improve instruction, it’s critical to analyze various metrics like total class time, instructor speaking time, participant speaking time—including the total number of times participants spoke as well as the average speaking time from all participants—and the total number of times participants raised their hands or gave feedback through the platform. Technology solutions can help compile all these metrics into one easy-to-digest dashboard and allow you to view results for an entire session, as well as drill down into individual participant results. Use the insights from your results to inform the changes you make to your training programs, and continue to iterate as more results come in.Technology can also automate time-consuming tasks like attendance recording, time in, time out, and tardiness, which both increases the accuracy of this data and frees instructors up to focus on quality education, improving the learning experience overall.Demonstrating Program EffectivenessIf you’re trying to effect change in your training programs by determining what’s working and not working, having quantifiable results and trends can go a long way in showing stakeholders the impact. Remember to utilize automation in your reporting technology, which can help expand reporting efforts and demonstrate effectiveness at a larger scale. When you’re communicating the value of your reporting to stakeholders, it also helps to have a framework for your metrics that’s backed by research and proven success. The Kirkpatrick Model for evaluating learner engagement lends itself perfectly here. This model distills training evaluation down to four distinct levels: The reaction of learners, including their thoughts on the training experienceThe learners’ increase in knowledge after the training experienceThe learners’ behavioral change and performance after applying the skills learnedThe results the learners’ performance has on the business or organizationMake sure whatever metrics you’re tracking to measure effectiveness follow a similar model to the Kirkpatrick Model. The key is persistent tracking to evaluate success at several points in time, allowing you to present a complete picture with your results.Access to accurate data and smart automation is the backbone of a successful training program. A powerful virtual classroom platform that’s purpose-built for learning and development, like Class, ensures that you can always keep your finger on the pulse of how your programs are performing, allowing you to focus on innovating and providing the best possible education for your learners.

Measuring the Impact of Your Virtual Training Programs

December 2022 - TD Magazine

Reduce the friction that is preventing learners from engaging with your learning.Businesses spend billions of dollars trying to understand consumer behavior, resulting in marketing strategies that aim to get consumers to buy their products. However, in those companies, many L&D functions are not leveraging the same insights and strategies to reach their learners. Treat your learning solutions like products, and minimize the friction between the solutions and learners. One way to do so is using BJ Fogg's behavior model to evaluate your learning initiatives.Identify the frictionThe Fogg behavior model looks at three conditions that must be present for someone to engage in a behavior: motivation, ability, and a prompt. When one of them is absent or diminished, so is the likelihood that the desired behavior will occur.For example, let's say you have a missed call on your cell phone. Why didn't you answer the phone? Maybe you heard it ring, looked down and saw who was calling, and decided you didn't want to talk to that person right now. You heard the ring—that was the prompt. Nothing was keeping your from answering the call (that is to say, you had the ability). You just didn't want to—you didn't have the motivation to engage in the behavior.Now let's say the phone rings, and you're expecting the call and want to take it. But you can't find your phone, or it's in another room and you can't get there in time. Again, you heard the ring (the prompt), and this time you wanted to take the call (motivation), but you weren't able to get to the phone in time (ability).Finally, let's say you're expecting the call, and your phone is right next to you. However, the ringer is set to silent. The phone rings, you want to answer and are able to do so, but you never hear the ring—you received no prompt to answer it.That's how Fogg's behavior model works. The presence or absence of any of the variables can contribute directly to the likelihood that you'll engage in a given behavior.What if we look at learner behavior through that lens? You've created a great learning initiative and spent a lot of time and money on the learning ecosystem. However, you don't seem to be getting the traction you're hoping for. Here are some questions to consider:Are learners aware of the learning solution? Are there prompts that alert learners to its availability?Are learners motivated? Do they want to engage in learning?Are they able to engage with the learning solution?Is there anything preventing them or making it more difficult for them to access it?If you have opportunities in any of those areas, then you have friction between learners and the learning solution. Friction, in this context, is the absence of prompts or is anything that impedes motivation or ability when it comes to a learning initiative.PromptsIf learners don't know about your learning initiative, it may as well not exist. These days, people are inundated with prompts—pop-ups on smartphones, ads on social media, digital billboards, and the like. How can we ensure that learners are prompted to our initiatives? The traditional methods are static and push prompts.A static prompt is one that is in a fixed physical location, such as a poster in a breakroom. It works fine when your audience tends to be isolated in one particular location. But if someone is not in that exact location, they won't see it.A push prompt is much more targeted. It entails identifying your audience members and sending a prompt to them, such as an email reminder that a course is available. However, you often can't customize that prompt by audience member—everyone gets the same message. And you can't time the prompt so that it reaches an individual right when they need it most. Thus, there are some limitations to what we have traditionally used as prompts in our learning ecosystems.This is an area we should pay close attention to because it has the potential to change rapidly. Imagine if we could activate a prompt for a specific learner at a specific time and in a specific location with a learning solution that is designed precisely for the activity that they are engaging in in that moment? It would be similar to finding an exclusive offer from Starbucks popping up on your phone whenever you're within 50 yards of a store.MotivationMotivation should be the easiest variable in this equation. If your initiative meets a learning need, learners should be inherently motivated to engage with it. However, that isn't always the case. In my experience, there are two major reasons. The first is that the solution does not meet a learning need (which is a subject for another article). The second, and more egregious, reason is that learning solutions get bogged down with demotivators that dilute learners' eagerness.Given the massive increase in video use for learning, there are some lessons to be learned from how people engage with YouTube and how we think about the videos and other learning initiatives we create.I often use the example of having a hole in my drywall and turning to YouTube to see how to fix it. Not surprisingly, there are numerous options from which to choose. We've all become accustomed to using filters to narrow down internet search results, although we often do our own visual filtering first. For example, I can take a quick look at a results list on YouTube for drywall repair and immediately rule out, or at least deprioritize, some of the videos. The first thing I'll look at is time. I expect that this should be a relatively simple task, so I'm not looking for a long video. The shorter the better.Likewise, learners have an expectation of how simple or complex a task is and at least some sense of how long it should take to learn. If they see that a video is 30 minutes long, and their perception of the task doesn't line up with that amount of time, you may lose them right away.I start playing a six-minute video from the list. After a lengthy introduction, a bio of the host, and several examples that don't apply to what I'm looking for, the host eventually starts the process of explaining how to execute the repair. It took over one minute to get there. That is an informal measurement I call "time to learning"—or how long it takes before I get to the actual learning. Time to learning is especially important when considering whether our goal is knowledge retention or knowledge acquisition. In other words, are learners in this for the long haul, or do they just need a quick one-off fix that they will rarely if ever need again?When looking at learning from a knowledge retention standpoint, time to learning is slightly less important. We know we're going to get deep, and participants learning about the instructor's credibility can be helpful to set the tone. But if we seek knowledge acquisition, or disposable knowledge, time to learning matters a lot—especially if there are several options from which to choose.For example, imagine I need to generate a report for my manager, who is out sick. I've never done the task before, and it's highly unlikely that I'll ever have to do it again. After some digging, I find a few videos that will help me figure out what to do. However, the videos don't cover only the report I need to generate; they cover all the reports in the entire program and explain where the data from those reports comes from and why they're important to the business.Are those bad things for me to know? Of course not. But they're not relevant to what I need to do right now. Getting what I need is like searching for that how-to needle in a haystack of video instruction. I may resort to other options, such as trial and error (risky) or asking a colleague (now two of us are wasting time trying to figure this out).Going back to the drywall video: I stuck with it for more than a minute only to see how long it would take for the purposes of this example. If I had a real need, I doubt I would have held out for more than 10–15 seconds.We can and should apply the same lessons to our video content libraries and consider where we are adding friction that we should remove. In most cases, those are demotivators that don't reflect the quality of our instructional design yet may deter learners from ever finding that out.AbilityAnother go-to source for information in people's nonwork lives is Google. Here's what happens if I search for "how to repair drywall"—let's count the clicks: I open the browser to Google and type the search term, then click Search. One click. What I see first are a few links to associated terms like "water damage" and "peeling" that enable me to refine my search. Below that is a frame from a video already queued to a specific timecode that is likely to address my need. Below that are links to a few more videos, all with descriptions and timecodes of key moments in the videos. Next up is the "People also ask" section, which displays several ways that others have posed their question that may be more in alignment with my need. And finally, I get a list of results that I would normally expect to see on Google. All from one click.Now think about your learners and, more importantly, your learning ecosystem. Most likely your version of Google is some sort of learning management system or learning experience platform. How many clicks does it take learners to get there? Once someone arrives at your version of Google, how many of the features described above does it have? How easy is it for learners to find what they need?Many of us are presumably still in a situation where learners must go through six, eight, or even 10-plus clicks to find what they need. Number of clicks and speed to solution are ways we can assess learners' ability to engage in the desired behavior.Nobody wants to click through a never-ending menu. Individuals have been conditioned that they shouldn't have to, and they are far more likely to give up somewhere along the way than they are to reach the end.Other ability barriers include bandwidth and hardware. Sometimes our instructional design comes with ability barriers. If you have motivated learners who know about your initiatives, make it as easy as possible for them to access and engage with learning.Enhance learning's impactThis article has numerous questions you can ask about any given learning solution. It's going to be difficult to get satisfactory answers to all of them. But what you can do is identify an area that you believe may be causing friction for your learners and is something that you can influence.In other words, we all want better bandwidth, but you probably have an extremely limited ability to influence that yourself. You may have a better chance implementing some prompts for learners that can move the needle. Or you may be able to reduce the number of clicks required to access content with some adjustments to your internal websites or portals. Start small by looking for areas to reduce friction with existing learning solutions, and then incorporate the ideas into the design phase of future ones.If you treat your learning program like a product and view your learning ecosystem as a mechanism for reaching your audience and providing that product, it may change your perspective.