Newsletter Article
Member Benefit
Who's Afraid of Kirkpatrick's Four Levels?
Content
Ann Armstrong defines evaluation, describes the benefits, identifies different evaluation models, explains why evaluation is often neglected, and espouses the need for internal evaluators with evaluator competencies.
Ann Armstrong defines evaluation, describes the benefits, identifies different evaluation models, explains why evaluation is often neglected, and espouses the need for internal evaluators with evaluator competencies.
Tue Feb 18 2014
Content
According to Kirkpatrick, the day of reckoning is here for training professionals. Senior management is demanding better evidence that training is cost effective; that there is in fact a positive payoff for the organization. Senior leaders want to know specifically what their employees learned (Level 2), exactly how they are applying what they learned on the job (Level 3), and the specific results to the organization from the training (Level 4).
According to Kirkpatrick, the day of reckoning is here for training professionals. Senior management is demanding better evidence that training is cost effective; that there is in fact a positive payoff for the organization. Senior leaders want to know specifically what their employees learned (Level 2), exactly how they are applying what they learned on the job (Level 3), and the specific results to the organization from the training (Level 4).
Content
Evaluation defined
Evaluation defined
Content
Often a hot topic for training departments, evaluation is defined by organizations in different ways. In Qualitative Research and Evaluation Methods, independent evaluation consultant Michael Quinn Patton explains the program evaluation began in the early 1900s under Thorndike’s leadership with a focus on educational testing.
Often a hot topic for training departments, evaluation is defined by organizations in different ways. In Qualitative Research and Evaluation Methods, independent evaluation consultant Michael Quinn Patton explains the program evaluation began in the early 1900s under Thorndike’s leadership with a focus on educational testing.
Content
In Utilization Focused Evaluation: The New Century Text, Patton defines evaluation as: “The systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.”
In Utilization Focused Evaluation: The New Century Text, Patton defines evaluation as: “The systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.”
Content
Program evaluation under the first part of this definition, focused on outcomes, became known as summative evaluation; evaluation for program improvement became known as formative evaluation. Today both forms of evaluation are important. In Evaluative Inquiry for Learning in Organizations, authors Preskill and Torres add to this definition by emphasizing that evaluation needs to be linked to organizational work practices, including:
Program evaluation under the first part of this definition, focused on outcomes, became known as summative evaluation; evaluation for program improvement became known as formative evaluation. Today both forms of evaluation are important. In Evaluative Inquiry for Learning in Organizations, authors Preskill and Torres add to this definition by emphasizing that evaluation needs to be linked to organizational work practices, including:
Content
personnel interests in using evaluation logic
personnel interests in using evaluation logic
Content
personnel involved in the organization’s evaluation processes
personnel involved in the organization’s evaluation processes
Content
focus on individual growth within the organization.
focus on individual growth within the organization.
Content
Benefits of evaluation
Benefits of evaluation
Content
In Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change , authors Darlene Russ-Eft and Hallie Preskill emphasize that for organizations to benefit from evaluation, evaluation must be integrated systematically into the organization’s work practices and processes. When this occurs several benefits accrue:
In Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change, authors Darlene Russ-Eft and Hallie Preskill emphasize that for organizations to benefit from evaluation, evaluation must be integrated systematically into the organization’s work practices and processes. When this occurs several benefits accrue:
Content
quality is improved
quality is improved
Content
workers become more knowledgeable
workers become more knowledgeable
Content
resources can be better prioritized
resources can be better prioritized
Content
planning and delivery of organizational initiatives is improved
planning and delivery of organizational initiatives is improved
Content
workers are held accountable
workers are held accountable
Content
recognition of program effectiveness is enhanced
recognition of program effectiveness is enhanced
Content
workers with evaluator competencies are in demand in the market.
workers with evaluator competencies are in demand in the market.
Content
In Evaluating Training Programs: The Four Levels, Donald Kirkpatrick and James Kirkpatrick propose that there are three primary reasons for evaluating a program: 1) future program improvement, 2) decision to keep or drop a program, and 3) to justify the training department.
In Evaluating Training Programs: The Four Levels, Donald Kirkpatrick and James Kirkpatrick propose that there are three primary reasons for evaluating a program: 1) future program improvement, 2) decision to keep or drop a program, and 3) to justify the training department.
Content
Meanwhile, Patton compares program evaluation to quality assurance and asserts that in program evaluation:
Meanwhile, Patton compares program evaluation to quality assurance and asserts that in program evaluation:
Content
the focus is specifically on program outcomes and process
the focus is specifically on program outcomes and process
Content
aggregated data is used
aggregated data is used
Content
judgments are goals based
judgments are goals based
Content
the intent of the evaluation is for decision makers.
the intent of the evaluation is for decision makers.
Content
Quality assurance on the other hand is focused on individual outcomes and processes, individual performance data, objective results, and intended use is individuals and their supervisors. Given the benefits of evaluation, why are many programs not evaluated or evaluated only at the individual quality assurance level?
Quality assurance on the other hand is focused on individual outcomes and processes, individual performance data, objective results, and intended use is individuals and their supervisors. Given the benefits of evaluation, why are many programs not evaluated or evaluated only at the individual quality assurance level?
Content
Methods of evaluation
Methods of evaluation
Content
According to Russ-Eft and Preskill_,_ in the late 1950s Kirkpatrick noticed that most evaluation of training programs could be placed in four categories: 1) reaction, 2) learning, 3) behavior, and 4) results. The simplicity of this model and ease of remembering catapulted its use in corporations.
According to Russ-Eft and Preskill_,_ in the late 1950s Kirkpatrick noticed that most evaluation of training programs could be placed in four categories: 1) reaction, 2) learning, 3) behavior, and 4) results. The simplicity of this model and ease of remembering catapulted its use in corporations.
Content
Indeed, Kirkpatrick’s model enjoyed the position of “state-of-the art training evaluation” through the mid-1990s, and has inspired the development of other models that incorporated its “look and feel.” Among the newer models are:
Indeed, Kirkpatrick’s model enjoyed the position of “state-of-the art training evaluation” through the mid-1990s, and has inspired the development of other models that incorporated its “look and feel.” Among the newer models are:
Content
Richey’s 1992 Systematic Model of Factors Predicting Training Outcomes
Richey’s 1992 Systematic Model of Factors Predicting Training Outcomes
Content
Five-Level Model of Evaluation by Kaufman, Keller, and Watkins
Five-Level Model of Evaluation by Kaufman, Keller, and Watkins
Content
Navy Civilian Personnel Transfer Model
Navy Civilian Personnel Transfer Model
Content
Hamblin’s Five-Level Model
Hamblin’s Five-Level Model
Content
Training Efficiency and Effectiveness Model (TEEM)
Training Efficiency and Effectiveness Model (TEEM)
Content
Holton’s HRD Evaluation Research and Measurement Model
Holton’s HRD Evaluation Research and Measurement Model
Content
Foxon’s Stages of Transfer Model
Foxon’s Stages of Transfer Model
Content
Brinkerhoff’s Stage Model
Brinkerhoff’s Stage Model
Content
Phillips and Phillips’s ROI Model—Kirkpatrick Level 5 Evaluation
Phillips and Phillips’s ROI Model—Kirkpatrick Level 5 Evaluation
Content
Garavaglia’s Transfer Design Model.
Garavaglia’s Transfer Design Model.
Content
In a qualitative embedded multi-case study of seven learning leaders (CLO, SVP, and VP) in the financial services and construction industries focused on the critical success factors in defining, designing, developing, and delivering online learning for adult professional development in corporations, all seven of the leaders used Kirkpatrick’s Four Levels of Evaluation for program evaluation. Of interest was the fact that while all 7 used Level 1 (Reaction) and Level 2 (Learning), only one of the learning leaders used Level 3 (Behavior) and only one used Level 4 (Results).
In a qualitative embedded multi-case study of seven learning leaders (CLO, SVP, and VP) in the financial services and construction industries focused on the critical success factors in defining, designing, developing, and delivering online learning for adult professional development in corporations, all seven of the leaders used Kirkpatrick’s Four Levels of Evaluation for program evaluation. Of interest was the fact that while all 7 used Level 1 (Reaction) and Level 2 (Learning), only one of the learning leaders used Level 3 (Behavior) and only one used Level 4 (Results).
Content
Three out of the four who were not using Level 3 asserted that they were in the process of trying to evaluate behavior changes in the programs evaluation or at least doing it anecdotally. One of the three was also in the process of trying to implement Level 4.
Three out of the four who were not using Level 3 asserted that they were in the process of trying to evaluate behavior changes in the programs evaluation or at least doing it anecdotally. One of the three was also in the process of trying to implement Level 4.
Content
Why is evaluation neglected?
Why is evaluation neglected?
Content
The reasons why evaluation is neglected is well-documented. From experience and observation delivering learning programs to learning leaders over three decades, Russ-Eft and Preskill offer their top 10 reasons why evaluation is neglected:
The reasons why evaluation is neglected is well-documented. From experience and observation delivering learning programs to learning leaders over three decades, Russ-Eft and Preskill offer their top 10 reasons why evaluation is neglected:
Content
lack of understanding of the goals and purposes of evaluation
lack of understanding of the goals and purposes of evaluation
Content
fear of the impact of evaluation results
fear of the impact of evaluation results
Content
lack of evaluator skills in the organization
lack of evaluator skills in the organization
Content
considered and add-on task
considered and add-on task
Content
belief that evaluation results will never be utilized
belief that evaluation results will never be utilized
Content
view that it is too time consuming and labor intensive
view that it is too time consuming and labor intensive
Content
cost versus benefit does not justify
cost versus benefit does not justify
Content
leaders think they already know the answer
leaders think they already know the answer
Content
bad prior experiences with evaluation
bad prior experiences with evaluation
Content
no one requires it.
no one requires it.
Content
In one instance the anecdotal reviews of a top program for financial advisors were so excellent that the learning leader refused to quantify the Level 3 and Level 4 results for fear that they would not live up to the organization’s beliefs about program success. In many cases there were no internal resources capable of performing the Level 3, Level 4, and Phillip’s Level 5 evaluation, and the cost of hiring an outside consulting firm to perform the evaluation was not perceived cost justified.
In one instance the anecdotal reviews of a top program for financial advisors were so excellent that the learning leader refused to quantify the Level 3 and Level 4 results for fear that they would not live up to the organization’s beliefs about program success. In many cases there were no internal resources capable of performing the Level 3, Level 4, and Phillip’s Level 5 evaluation, and the cost of hiring an outside consulting firm to perform the evaluation was not perceived cost justified.
Content
Enter evaluator competencies
Enter evaluator competencies
Content
Evaluation of training programs goes back to the early 1900s. For decades, Kirkpatrick’s model has been espoused as the model of choice for organizations evaluating training. Despite praise for Kirkpatrick’s model and many of its predecessors, training organizations rarely implement evaluation beyond Level 1 (Reaction) and Level 2 (Learning).
Evaluation of training programs goes back to the early 1900s. For decades, Kirkpatrick’s model has been espoused as the model of choice for organizations evaluating training. Despite praise for Kirkpatrick’s model and many of its predecessors, training organizations rarely implement evaluation beyond Level 1 (Reaction) and Level 2 (Learning).
Content
Given that top management and senior leadership are demanding evidence for results and the ROI associated with training, it is time to develop and standardize evaluator competencies and train both internal and external personnel on these competencies.
Given that top management and senior leadership are demanding evidence for results and the ROI associated with training, it is time to develop and standardize evaluator competencies and train both internal and external personnel on these competencies.
Content
The current list of ibstipi competencies was designed for both internal staff and external consultants across industries and organizations and includes 14 evaluator competencies in four domains. The competencies are further explained by 84 performance standards. The competencies were validated through a global sample of 450 practitioners and reflect the knowledge, skills, and attitudes needed by a practitioner to be a competent evaluator. For a complete list of evaluator competencies go to www.ibstpi.org/download-center-2 and select download evaluator competencies.
The current list of ibstipi competencies was designed for both internal staff and external consultants across industries and organizations and includes 14 evaluator competencies in four domains. The competencies are further explained by 84 performance standards. The competencies were validated through a global sample of 450 practitioners and reflect the knowledge, skills, and attitudes needed by a practitioner to be a competent evaluator. For a complete list of evaluator competencies go to www.ibstpi.org/download-center-2 and select download evaluator competencies.
Content
Further feading
Further feading
Content
Armstrong, A. W. (2007). Executive beliefs about the critical success factors in defining, designing, developing, and delivering e-learning for adult professional development in corporations. Teachers College, Columbia University). ProQuest Dissertations and Theses, , 334. Retrieved from http://search.proquest.com/docview/862347883?accountid=35812 . (862347883).
Armstrong, A. W. (2007). Executive beliefs about the critical success factors in defining, designing, developing, and delivering e-learning for adult professional development in corporations. Teachers College, Columbia University). ProQuest Dissertations and Theses, , 334. Retrieved from http://search.proquest.com/docview/862347883?accountid=35812. (862347883).
Content
Armstrong, A. W. (2008). Executive beliefs about the critical success factors in defining, designing, developing, and delivering e-learning for adult professional development in corporations. Online Submission . Retrieved at http://www.eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED501621
Armstrong, A. W. (2008). Executive beliefs about the critical success factors in defining, designing, developing, and delivering e-learning for adult professional development in corporations. Online Submission. Retrieved at http://www.eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED501621
Content
Ibstipi (2014). Evaluator competencies. Retrieved from http://www.ibstpi.org/evaluator-competencies/
Ibstipi (2014). Evaluator competencies. Retrieved from http://www.ibstpi.org/evaluator-competencies/
Content
Kirkpatrick, D.L. (2009). Are you REALLY using the Four Levels? Kirkpatrick Partners. Retrieved from Kirkpatrickpartners.com
Kirkpatrick, D.L. (2009). Are you REALLY using the Four Levels? Kirkpatrick Partners. Retrieved from Kirkpatrickpartners.com
Content
Kirkpatrick, D. L. & Kirkpatrick, J. D. (2006). Evaluating training programs: The Four Levels (3rd ed). San Francisco, CA: Berrett-Koehler
Kirkpatrick, D. L. & Kirkpatrick, J. D. (2006). Evaluating training programs: The Four Levels (3rd ed). San Francisco, CA: Berrett-Koehler
Content
Patton, M.Q. (1997) Utilization focused evaluation: The new century text. Thousand Oaks, CA: Sage.
Patton, M.Q. (1997) Utilization focused evaluation: The new century text. Thousand Oaks, CA: Sage.
Content
Patton, M. Q. (2002). Qualitative research and evaluation methods. (3rd ed.). Thousand Oaks, CA: Sage
Patton, M. Q. (2002). Qualitative research and evaluation methods. (3rd ed.). Thousand Oaks, CA: Sage
Content
Preskill, H. & Torres, R.T. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage
Preskill, H. & Torres, R.T. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage
Content
Russ-Eft, D; Bober, M.J., delaTeja, I., Foxon, M.J., & Koszalka, T.A. (2008). Evaluator competencies: Standards for the practice of evaluation in organizations. San Francisco, CA: John Wiley & Sons.
Russ-Eft, D; Bober, M.J., delaTeja, I., Foxon, M.J., & Koszalka, T.A. (2008). Evaluator competencies: Standards for the practice of evaluation in organizations. San Francisco, CA: John Wiley & Sons.
Content
Russ-Eft, D. & Preskill, H. (2001). Evaluation in organizations: A systematic approach to enhancing learning, performance, and change. Cambridge, MA: Basic Books
Russ-Eft, D. & Preskill, H. (2001). Evaluation in organizations: A systematic approach to enhancing learning, performance, and change. Cambridge, MA: Basic Books