Jack E. Smith

Presents steps to confirm the validity of competency examination in training programs. Questions that should be answered by competency tests; Development of learning objectives; Outline of examination content areas; Establishment of examination format; Construction of test items; Evaluation of content validity; Analysis and feedback.

Using Competency Exams for Evaluating Training

 I first heard about the Association for Talent Development and its Certified Professional in Learning and Performance (CPLP) program in 2005, during my first year as a doctoral student, while looking for material for my literature review. The program intrigued me, and I decided at the time that I would one day come back and try to obtain the CPLP certification.In 2008, I graduated with a PhD in educational leadership and passed the Texas principal certification test. It was one of the proudest days of my life. I was on top of the world, the more so because the road to my doctorate had been anything but straight and smooth. It was many years—and many dilemmas—in the making. As it later turned out, the road to my CPLP certification would be equally rough, but now that I have it, I’m proud to have persevered and prevailed.My Long Journey BeginsEver since I was in grade school, I knew that I wanted to teach. While other kids my age were always eager to escape school for their summer breaks, I spent mine in my parents’ garage, pretending to be a teacher! That was my life plan, but then life happened: I married early and quickly had four daughters. I assumed that was the end of my plan to someday attend college—that is, until it became obvious to me that a divorce was in my near future, and that I would need to make some big changes quickly in order to support my family as a single parent.It’s amazing what you can accomplish when you know that you have no other choice. While working and solo-parenting full time, I also managed to attend school full time and graduate with a BS in mathematics in four years. While attending school, I had been working as a teacher’s aide, but armed with my new college degree, I became a high school math teacher. I still had to get certified as a teacher by the state, however, which meant passing a two-part test: one part on teaching, the other on math. I passed the teaching part on my first try, but I kept missing the math portion by one point!Feeling embarrassed and pressured, I went to study groups and got one-on-one tutoring, but still was not successful. It was then that I realized that it might be a good idea to develop a plan B, so while I was teaching, I also started sending out my resume.A New EndeavorAt the end of the summer, just before I was to go back and teach, I got a call from a NASA contractor. I started working at Johnson Space Center as a station robotics instructor. Soon I became a certified senior robotics instructor, training astronauts to assemble the space station using a robotic arm, and training engineers on how to be trainers. Then the shuttle blew up.We were told that training was going to be slow for about two years, so we used that time to redevelop lessons and to get employees certified. During that time, I also went back to school during the weekends to obtain my master’s degree in education.Then, just a few months after completing my master’s degree, I started my doctorate in educational leadership. Training at NASA was beginning to ramp back up—I was training shuttle and station astronauts again—but I still managed to complete my doctorate in exactly three years and become a manager of operational engineers and subject matter experts. Then the shuttle program ended.My company lost its contract for the space station, dissolved, and the majority of its employees were laid off. The few remaining employees, like myself, were hired by the company that had won the station contract. I was reassigned to robotics.Unexpected Twists In 2013, a job opening in a talent development department within a school district became available. I thought it was the perfect job for me. They didn’t. I’ve always believed that if you lose, don’t lose the lesson, so after being told I didn’t get the job, I asked for some feedback. The hiring manager suggested I look into the CPLP. CPLP? Oh yeah, I had forgotten about that. I thought the CPLP might be the certification I needed to complement my educational background and advance my professional goals. I went to the ATD website and read everything I could about the CPLP program. I became an ATD member and registered for the CPLP. I purchased the Learning System (the official study guide for the CPLP). Once again, I was on that merry-go-round of working days and studying at night, but I’d been there before, and I knew I could do it. I studied by myself. I thought the material was easy for me to understand because of my formal studies in education, but then I took the Knowledge Exam and failed. Immediately, the self-doubt and negative thoughts that I’d had when I took my teacher certification exam years before came rushing back, telling me that even though I knew the material, I just wasn’t good at test-taking. But after a month of feeling depressed, I pushed those negative thoughts aside and resolved to try again. Now that I knew how the test was set up, I studied differently: still by myself, but more focused on areas in which I knew I had strengths, and which were scored at a higher percentage. I was a math major, remember, so I put that training to good use: I played the averages! The second time I took the test, I passed—thank God I didn’t give up! Putting together the Work Product was almost like writing another doctoral dissertation. Using the area of expertise key action list and the scoring rubrics appendix as my guides, I read and reread everything pertaining to the Work Product and made sure everything was covered. My goal was to have it finished two months before it was due and I met that goal. I made copies of everything before mailing, marked the week I would get the notification on my calendar, and then settled in to wait. After a couple of months, I got the email stating that I had passed! Happy Ending Attaining the CPLP validated my childhood dreams. I know I have a gift for teaching, so professionally, I finally feel as if I really belong. I don’t know what will happen with my new CPLP certification, but I know when the next opportunity knocks, I’ll be ready. Learn more about the CPLP Certification.

CPLP Journey Taught Me That Determination Defeats Dilemmas

This summer, the Science of Learning Community began the discussion of Eduardo Salas’ and fellow authors’ essential work, The Science of Training and Development in Organizations: What Matters in Practice, which used meta-analyses (statistical methods for contrasting and combining results from multiple research studies) to find the best ways to design, deliver, and implement training so performance outcomes are as good as possible.In July, I reviewed their analyses on the critical elements before training. Last month, I reviewed critical elements during training. This month, I’ll complete this series by discussing what the meta-analyses uncovered about the most effective practices after training.Why Post-Training Is So ImportantMy mother-in-law used to say this about cooking: “You put good things in; it’ll taste good.” While this maxim sounds logical, it doesn’t always work in practice. For instance, chocolate and mustard don’t taste good when you mix them together. Likewise, you can ruin good foods by cooking them for too long.What does this have to do with training? Simply this: Good ingredients don’t ensure good training, and training “tastes” best when L&D pros “cook” it the right way. Certainly, it’s nice when learners earn a good score on a post-instruction test, but it’s inadequate if their performance doesn’t improve. In other words, transfer of training is the ultimate goal.In their research, Salas and his fellow authors come to the same conclusion. They explain that although organizations spend billions of dollars on training each year, even good training doesn’t always transfer to the workplace. Therefore, they assert that what happens after training “can have as great an impact on training effectiveness as what happens during training….” What’s more, lack of transfer happens for a variety of reasons, but their meta-analyses found that follow-up activities can have a positive impact on this challenge.Figure 1 shows Table 4 from the research, which is a checklist of steps L&D practitioners can take following training experiences to ensure maximum effectiveness. Support on the JobThe analyses shows the post-training environment has a significant impact on whether training will transfer to the workplace. For example, is the supervisor supportive of the skills covered in the training? Will the work environment allow people to use these skills? These factors have a huge impact on whether the skills practiced in training continue or die out. The paper also points to research that shows only 7 to 9 percent of skill acquisition comes from formal training. Instead, leaders (both formal and informal) are key factors in learning—as they greatly influence what people actually do on the job. Think about how people perform where you work and you’ll realize the truth of this statement. Salas et al. state that organizations must provide tools, training, and support to help these leaders coach others and reinforce desired training, as well. Debriefing The authors also say that organizations can institute debriefing as a powerful (but often underused) tool for reinforcing learning after training. A common—and effective—tool in the military, debriefing enables learners to reflect on the training experience and identify what went well and not-so well. In addition, when used after training, debriefing can help workers create performance agreements that tie measurable actions to their performance goals. EvaluationTraining evaluation typically means collecting data to determine whether the training met learning objectives. The Salas paper provides a good discussion of how best to do this so the organization can determine whether the accomplishment of the learning objectives resulted in improved job performance.  Training, Systems, and the Science of Learning The Science of Training and Development in Organizations: What Matters in Practice provides a list of important questions that L&D practitioners can use to ask about training programs—in general terms and specific aspects of individual training programs. (See  Figure 2, which is Table 6 from the research.)  The authors suggest that readers consider the analogy of investing large sums of money, such as retirement funds, into mutual funds without ever researching the fund’s past performance or reviewing how the investment does over time. Doing so would be akin to throwing away your money, right? Similarly, not trying to obtain the best outcomes from your training is irrational. That’s why they tell readers that “decisions about what to train, how to train, and how to implement and evaluate training should be informed by the best information science has to offer.” Indeed, that is exactly what this paper’s meta-analyses does for training. It uses the science of learning to outline how to gain the best outcomes from training. For instance, investments in training must use adequate analyses to ensure that the organization is addressing the right training needs, using the right training methods, and taking the right actions before, during, and after training. Simply stated, like retirement investments, we must analyze, monitor, and follow-up on our training investments. To be sure, science of learning provides evidence-based training design principles, and training developed with these principles provides better outcomes for our organizations. But design principles are not always enough. Training is a system issue. If learners don’t feel confident, even good training may not take. If learners return to an environment that doesn’t support the training, even good training will fail. Bottom line: L&D professionals involved in designing training need to understand the entire learning system. Sources 
-------- Salas, E., Tannenbaum, S.I., Kraiger, K and Smith-Jentsch, K.A. (2012). The Science of Training and Development in Organizations: What Matters in Practice, Psychological Science in the Public Interest, 13 (2), pp. 74-101. 42.

Jack E. Smith

About