Certification Test Development Steps

All credible credentialing programs start with occupational research. This can be called a job analysis, role delineation, or competency study. This research outlines what it is important for a specific professional to know and be able to do. This foundational research ties directly to what is covered on an exam and how much of each topic is covered. The most recent research on which ATD CI has based its credentials is the 2019 study which resulted in Capabilities for Talent Development: Shaping the Future of the Profession, as well as the Talent Development Capability Model. Over 3,000 professionals contributed to the research in the largest response rate ever for an ATD competency study. The most recent study included nearly 200 knowledge and skill statements, but not all will end up being included on certification exams.

Each time the competency model research is updated, the breakdown of what is covered on the APTD and CPTD exams must be updated to reflect the results. The outline of what is covered on a certification exam is called a Detailed Content Outline or is referred to as a blueprint or test specifications. This document outlines what topics are covered and what percentage of each topic is covered.

To develop the detailed content outline, ATD CI convenes a panel of currently certified SMEs to review the data from that survey to identify what subset of knowledge and skill statements should be included on each exam. Response data from the target audience (three years of experience for APTD and five years for CPTD) are reviewed and the SMEs work with ATD CI’s testing partners (psychometricians) to identify what percentage of the exam should cover each content area. Each exam administered must conform to this blueprint. ATD CI cannot change what is covered on the exam, or the percentages of what is covered, without doing new research.

Once ATD CI knows what percentage of which topics needs to be included on an exam, test questions, called items, must be developed. There is both an art and a science to writing test questions. ATD CI trains currently certified subject matter experts to draft test questions. Each question must meet a specific format. Each question must be coded back to the detailed content outline to ensure it is testing a relevant topic. The correct answer to each question must be supported by a citation in commonly accepted and readily available professional texts. The same list of texts that ATD CI provides to candidates in the Candidate Handbook is used by item writers.

Draft items are then reviewed, edited, and approved by a separate group of currently certified SMEs. They must all agree that only one correct answer is listed and must verify the reference citation supports the correct answer. The content area it is testing is also confirmed. Approved items receive a final review by psychometricians, who are testing format experts. About 40% of drafted items never make it into the item bank. Those items approved and verified are now ready to be included on an exam form and to be pre-tested.

An exam form is what test developers call the total set of items delivered to a candidate during an exam. Each exam form must follow the blueprint, or detailed content outline, established for an exam. Most exam programs have more than one exam form in use at the same time. This is a security measure and also ensures candidates who are unsuccessful on one exam attempt do not receive the same exact exam on the second attempt.

ATD CI works with psychometricians to select test questions that meet acceptable statistical standards and meet the content breakdown. Once a psychometrician selects the items, ATD CI assembles a group of currently certified subject matter experts to review the form. They review each item again to make sure the correct answer is indicated as the key. They also make sure that there are not items that give away the answer to other items inadvertently. Items are substituted, as necessary. Each exam form must meet all the content and statistical properties established before it can be administered.

ATD CI pre-tests all of its test questions before they affect a candidate’s score. There are two ways to pre-test items. One way is to include pre-test items in an already established exam. Some of the items on an exam being routinely administered may be pre-test items. Questions are mixed into the exam so that ATD CI can collect data on the performance of the item. These items do not count for or against a candidate’s score. ATD CI does not let candidates know which are pre-test items because candidates would be less likely to give equal effort to answering them.

After items have been through pre-testing, performance statistics are reviewed. We look at: overall percentage of candidates answering it correctly. What percentage of candidates choose each of the options presented? How well are candidates who choose each option performing on the entire exam? We want to see a positive correction between those choosing the correct answer and performing well on the entire exam.

Once a test question is reviewed and its statistics meet acceptable standards, it goes into the item bank as a live item and can appear as a scored item on an exam. Questions that don’t meet acceptable statistical standards are sent back for editing and possible inclusion as a pre-test item at a future time.

Another way to pre-test items is to conduct a pilot exam. This is done when the format of an exam is new or changed, or a significant portion of the items have not been tested before. In this case, exams are administered but no scores are given. The data across the entire exam from a large pool of candidates is reviewed before scores are issued.

Both of these methods help ensure that candidates are not inadvertently penalized for flaws in test development and are part of best practices for developing and administering exams.

This is the process that is used to set the passing score for an exam. Sometimes it is referred to as a cut score study. ATD CI uses the Angoff Method to establish its passing score. A group of currently certified subject matter experts are brought together who have not been involved in previous test development steps. They work with ATD CI’s test development partners to review the overall difficulty level of the exam, question by question. These assessments are compared with actual candidate performance on the questions. This data is used to establish a relative difficulty level of the exam form and establish how many items must be answered correctly to pass. Once the score is set, it does not change until a new cut score study is done. This is called criterion referenced scoring. A candidate’s success on the exam is measured against the fixed cut score. (This is similar to how par on a golf course is set.)

Because there are different forms of the exam being used at the same time, a passing point (the number of items that must be answered correctly to pass) can sometimes differ from one exam form to the next simply because of the combination of questions on that form. Some forms are just a little bit easier or a little bit harder than another. For this reason, ATD CI issues scaled scores. Scores are issued on a scale from 200 – 800 with 500 equaling passing. 500 may equal 78 questions for one exam form and could equal 76 or 80 items for another form. By issuing scaled scores, candidates scores are comparable and which form they took does not matter.

Criterion referenced scoring is different than norm referenced scoring which is used for exams like the SATs or MCATs or GREs. In the case of these types of exams, a candidate’s performance is compared to the group of candidates that tested at the same time. Candidate results are put into an established curve and scores issued accordingly. This means that a candidate’s relative success depends on the group they tested with. This is NOT the case with the APTD and CPTD exams.

Exam administration is simply delivering the exam. ATD CI administers exams via a secure testing platform delivered by its test delivery partner, PearsonVue. PearsonVue is responsible for security on site at test centers, for scoring exams, and issuing score reports to candidates. Scores are reported to ATD CI several times a day.

ATD CI does monitor PearsonVue compliance with test security and reviews any incident reports filed at the testing center. Candidates with a concern during exam administration should make a report to the testing center proctor or administrator, and then to ATD CI.

Once items and exams have been pilots and cut scores established, exams are administered on an on-going basis. However, test development does not stop there. Exam items are in constant motion so that new items are available should there be a breach of exam security or it is time to rotate in new test forms. ATD CI releases new test forms on an on-going basis typically once a year, which requires on-going item drafting, review, and pre-testing. This ensures the exam reflects real-world current practice and ensures its reliability as a test instrument.

ATD CI works with its test development partner to monitor exam performance on an on-going basis. We review individual test question performance to make sure those items scored on an exam continue to fall within acceptable statistical performance. If something changes, we can replace the item or the entire form. We also review overall exam form performance to ensure its reliability in determining who has mastered the content covered and who has not. There are specific sets of statistics and guidelines for the acceptable range of those statistics that ATD CI monitors and follows.