ATD Test Development Steps

All credible credentialing programs start with occupational research. This can be called a job analysis, role delineation, or competency study. This research outlines what it is important for a specific professional to know and be able to do. This foundational research ties directly to what is covered on an exam and how much of each topic is covered. The most recent research on which ATD CI has based its credentials is the 2019 study which resulted in Capabilities for Talent Development: Shaping the Future of the Profession, as well as the Talent Development Capability Model. Over 3,000 professionals contributed to the research in the largest response rate ever for an ATD competency study. The most recent study included nearly 200 knowledge and skill statements, but not all will end up being included on certification exams.

Each time the competency model research is updated, the breakdown of what is covered on the APTD and CPTD exams must be updated to reflect the results. The outline of what is covered on a certification exam is called a Detailed Content Outline or is referred to as a blueprint or test specifications. This document outlines what topics are covered and what percentage of each topic is covered.

To develop the detailed content outline, ATD CI convenes a panel of currently certified SMEs to review the data from that survey to identify what subset of knowledge and skill statements should be included on each exam. Response data from the target audience (three years of experience for APTD and five years for CPTD) are reviewed and the SMEs work with ATD CI’s testing partners (psychometricians) to identify what percentage of the exam should cover each content area. Each exam administered must conform to this blueprint. ATD CI cannot change what is covered on the exam, or the percentages of what is covered, without doing new research.

Once ATD CI knows what percentage of which topics needs to be included on an exam, test questions, called items, must be developed. There is both an art and a science to writing test questions. ATD CI trains currently certified subject matter experts to draft test questions. Each question must meet a specific format. Each question must be coded back to the detailed content outline to ensure it is testing a relevant topic. The correct answer to each question must be supported by a citation in commonly accepted and readily available professional texts. The same list of texts that ATD CI provides to candidates in the Candidate Handbook is used by item writers.

Draft items are then reviewed, edited, and approved by a separate group of currently certified SMEs. They must all agree that only one correct answer is listed and must verify the reference citation supports the correct answer. The content area it is testing is also confirmed. Approved items receive a final review by psychometricians, who are testing format experts. About 40% of drafted items never make it into the item bank. Those items approved and verified are now ready to be included on an exam form and to be pre-tested.

An exam form is what test developers call the total set of items delivered to a candidate during an exam. Each exam form must follow the blueprint, or detailed content outline, established for an exam. Most exam programs have more than one exam form in use at the same time. This is a security measure and also ensures candidates who are unsuccessful on one exam attempt do not receive the same exact exam on the second attempt.

ATD CI works with psychometricians to select test questions that meet acceptable statistical standards and meet the content breakdown. Once a psychometrician selects the items, ATD CI assembles a group of currently certified subject matter experts to review the form. They review each item again to make sure the correct answer is indicated as the key. They also make sure that there are not items that give away the answer to other items inadvertently. Items are substituted, as necessary. Each exam form must meet all the content and statistical properties established before it can be administered.