Best Practices for Leadership Development ROI Studies

In the last decade, we’ve witnessed an increase in leadership development programs in talent development. This has sparked more evaluations at the Impact and ROI levels. This article describes why this is being pursued and the best practices for conducting these types of studies.

More than two decades ago, ATD entered into a partnership with ROI Institute. ATD’s motivation for educating members on the use of ROI was simple. If members could show the impact and ROI of major talent development programs, like leadership development, they would be able to get more support for talent programs and secure increased funding for talent development at budget approval time. This initiative involved four different areas:

Publications. Just this year, at the ATD conference, the 51st book authored or edited by Jack and Patti Phillips of ROI Institute was released. More than half of these books focused on the use of ROI and the different facets of ROI implementation.

Education. ROI Institute offered several educational programs to members and continues to offer them today. Measuring ROI (MROI), one of ATD’s first educational programs, is still being offered today.

A business arrangement to provide new products and services. Among the products was a new program, Evaluating Learning Impact, which has been a successful offering through ATD education. This program is still being offered globally in multiple languages.

More ROI Involvement with ATD. ATD acquired the global ROI network, offered ROI conferences and special programming, and increased involvement of ROI network members in ATD ICE. ROI programming has been a part of the ATD conference for almost 30 years.

This partnership, initiated by ATD, has helped talent development professionals show the value of what to do to senior executives and funders, use process improvement to make programs better, gain support for talent development, and secure more funding at budget time.

Developing ROI Studies

The ROI Methodology, a logic model with sequential steps, is now the most used evaluation system in the world. It originated in the learning and development space but is now used in all types of evaluations, including measuring the ROI of marketing, sustainability initiatives, and artificial intelligence. The most popular and frequent type of ROI studies conducted with this methodology is leadership development.

ROI evaluation studies come through two streams of activities. The first is the ROI Certification process, where individuals attend a comprehensive program to learn how to conduct evaluations at the Impact and ROI level. Certification participants must focus on a project and evaluate that project to impact and ROI to achieve their certification. This is a work-product certification that requires validation that the participants have actually used the methodology to evaluate a program. Almost 10,000 individuals have completed this process to become Certified ROI Professionals. The number one topic tackled by these participants is leadership development.

The second stream is through requests to evaluate the ROI for a program or project, providing an independent, objective assessment. Again, leadership development is at the top of these requests.

What is causing this? Leadership development is needed now more than ever, and there is so much great content that it is often expensive to implement on a wide-scale basis. At the same time, the funders of these programs, the senior executives and top leaders, are requiring that they see the actual impact of the program on the organization or the financial ROI, to make sure the investment was worthwhile. Some progressive leadership development providers are being proactive and not waiting for the request.

Along the way, we have identified best practices for the studies being conducted. These are consistent processes that are in place that help make leadership development programs successful and deliver amazing ROI numbers. This article describes the best practices.

First, the Value Chain

Leadership development can be developed along a value chain with five levels of outcomes. This chain is a logical flow of data that occurs in any type of program or project. Raymond Katzell brought this concept to the training field in 1952.1 It is particularly relevant to leadership development, as Katzell noted. Kirkpatrick promoted the four levels in four articles in 1959–1060.2 The fifth level of outcome is the latest version of the logic model, first reported in our book, Handbook of Training Evaluation and Measurement Methods, published in 1983. This was the first book on training evaluation in the USA.3

Here are the levels of outcomes:

Level 1, Reaction, ensures that participants involved in a leadership development program see that the program is relevant to their situation, important to them, something they would recommend to others, and are committed to making it a success. These four powerful reactions often serve as predictors of behavior change.

Level 2, Learning, is the classic acquisition or enhancement of knowledge, skills, behaviors, and competencies, making sure the participants can actually do what is desired. Skill practices, role-plays, simulations, and self-assessments often validate that learning has been achieved. Without learning, a program will not be successful.

Level 3, Application, involves the use of the leader behavior, applying what they’ve learned. This is often measured by the extent, frequency, and success of use of the desired competencies. Without application, there is no impact. At this point, barriers and enabling the use of the behaviors are captured for process improvement.

Level 4, Impact, is the consequences of the behavior change. These are usually key measures in the system that often reflect the work of the organization. Impact often includes hard measures related to output, quality, cost, and time, and softer measures, such as teamwork, engagement, and collaboration. These measures attract executive attention and need to be connected to the leadership development program. Typical impact measures for leadership development are retention, productivity, quality of work, and engagement. Impact is the number 1 measure desired by top executives. Without impact, the ROI will be very negative.

Level 5, Return on Investment (ROI), answers the question, Is it worth it? ROI shows the monetary benefits delivered compared to the cost of the program. This ultimate accountability should be considered for programs that are very expensive, important, strategic, and attract the executive’s attention for this level of accountability. This is the number 2 measure desired by top executives.

Next, the ROI Methodology

The ROI Methodology is a systematic, comprehensive measurement and evaluation process that collects, analyzes, and reports six types of measures: reaction, learning, application, impact, return on investment, and intangibles. Intangibles are the impacts not converted to money. This balanced approach to measurement includes a technique to isolate the effects of the program from other influences. Figure 1 shows this step-by step methodology. Other ATD references provide more detail.4

A set of operating standards with a fiscally conservative philosophy that serves as guiding principles to keep the process on track to ensure reliable results. The conservative standards build credibility with executives and top administrators as well as researchers. Figure 2 provides a list of the guiding principles.

The ROI Methodology is the most documented, published, and used measurement and evaluation system in the world. Major organizations have adopted and endorsed this methodology, including the United Nations and thirty central governments around the world.

Best Practices

From all of this use, we have identified the 10 best practices for leadership development ROI evaluations. Here are ten actions organizations are taking to ensure their leadership development programs are designed and delivered to achieve impact, are evaluated at the ROI level, and the results are leveraged for process improvement, funding, and support.

Make sure that leadership development starts with why. The why is not leader behavior, but rather the impact of the leader behavior. Impact measures are often described with key performance indicators. These measures are the beginning point because these are what executives want to see at the end. Starting with the end in mind helps to deliver the desired impact. This would require clever thinking and possibly some additional upfront analysis to ensure that the leadership development is connected to specific impact measures.

Customize the leadership program with impact objectives. It is important for participants in leadership development programs to own the program and be determined to use the behaviors to drive impact. One of the best ways to address this issue is to have the leaders who are attending the program select one or two team key performance indicators (KPIs) that they want to improve, but only if they can improve them with the use of the proposed leadership development competencies. When this is accomplished, the program is immediately connected to something the participants want—to improve the measures. You are now providing a means to the end to help them get there. These impact measures become impact objectives for the participants.

Deliver credible impact data. This best practice has two parts: showing the results by identifying the key impact measures that the leadership program has influenced and sorting out the effects of the program on those impacts. This means that a step has to be taken to isolate the effects of the leadership development program from other influences. With the ROI Methodology, it’s not only a step in the model, but it is one of the standards that must be followed. It can be accomplished with every program ROI evaluation. It has to be achieved to the credibility level that will win the approval of the chief financial officer.

Don’t be afraid of the outcome. Too many providers have a fear of the outcome. There is a concern that this program might not deliver an impact or a positive ROI. This concern keeps them from pursuing this level of evaluation. Unfortunately, they are often forced into this situation by an executive who demands it or requires it. We suggest that providers be proactive and pursue this evaluation, realizing that if the program is unsuccessful, it’s usually not the content that is the problem. It’s actually how the program was implemented and supported in the organization. This information is needed for the client to see what may be keeping the program from being successful. It’s an action to be taken, not something to be avoided.

Providers should encourage clients to pursue this level of accountability. The Blanchard Company offered a Podcast (The Business Case for Leadership Development) for their clients, suggesting that they show the impact and maybe even the ROI of their leadership programs. They recommended the ROI Methodology from ROI Institute, introducing one of our books and the methodology. This is helpful to the clients who will see this as a positive move, a provider suggesting that the client show the value of their programs.

Providers should conduct studies or pay for the studies to be conducted. Some leadership providers have developed this capability and show the impact and ROI for their leadership programs. This is often built into the program’s price, not an add-on charge that the client would be reluctant to pay. Alternatively, the provider can pay for it to be conducted. Recently, one organization, Insights Discovery, paid for an ROI study for one of its clients, Indeed. They wanted to show Indeed the value of the huge investment that they made in the Insights Discovery program.

Providers should educate their clients. Some leadership providers will make an extra effort to educate their clients on this kind of methodology. The Center for Creative Leadership, for example, has hosted the ROI Certification process in its facilities and invited its clients to attend. In these situations, it is impressive for the clients who not only see that the provider is encouraging this level of accountability but actually show them how it’s accomplished. The rationale: if the program is going to be evaluated, it needs to be evaluated using a credible process like the ROI Methodology. An ROI request often comes from senior executives, and if the client doesn’t know how to evaluate to Impact and ROI, they may try to it and not use a credible process. The results could be disastrous.

Make sure the analysis is credible. Worst-case scenarios need to be presented. You want to adjust for errors in the process. You need to understate the results. Every step of the way, a conservative approach is taken. The philosophy is this: If you have an executive briefing showing the results of an ROI study to a group of executives, you have to make sure that the CEO and CFO leave the meeting convinced that these results are credible.

Tell the story. When the results are presented, include both narrative and numbers. Narrative is in the form of stories that are collected along the way as reaction, learning, application, and impact are measured. There will be results showing six types of data: reaction, learning, application, impact, ROI, and intangibles. Together, this makes a convincing story that captures attention and provides impact and ROI data.

Leverage the results. These results can be used to improve programs, increase support, enhance funding, and earn respect. For leadership development providers, it provides great data that they can use to market to other prospective clients. It is the best strategic marketing data you can provide.

So, there you have it. How results are changing, and best practices are emerging—all with the good intention of making leadership development better, more successful, and a major driver of success in all types of organizations. For copies of leadership development ROI studies, please contact ROI Institute at [email protected].

Learn how to grow top talent and drive organizational success with ATD's creating a leadership development program certificate.

References

Katzell, R. (1952), "Can We Evaluate Training?" Industrial Management Institute, University of Wisconsin.

Kirkpatrick, D. L. (1959a). Techniques for evaluating training programs. Journal of the American Society Training Directors, 13(11), 3-9.

Kirkpatrick, D. L. (1959b). Techniques for evaluating training programs: Part 2-Learning. Journal of the American Society of Training Directors, 13(12), 21-26.

Kirkpatrick, D. L. (1960a). Techniques for evaluating training programs: Part 3-Behavior. Journal of the American Society of Training Directors, 14(1), 13-18.

Kirkpatrick, D. L. (1960b). Techniques for evaluating training programs: Part 4-Results. Journal of the American Society of Training Directors, 14(2), 28-32.

Phillips, J. J. (1983). Handbook of Training Evaluation and Measurement Methods. Gulf Publishing.

Phillips, P. P., Phillips, J. J., and Ray, R. (2020). Proving the Value of Soft Skills: Measuring Impact and Calculating ROI. ATD Press.