Standard setting, in the context of criterion-referenced test development, describes the process used to determine the thresholds of minimally acceptable performance levels. When there are two categories of performance, it is used to differentiate between those who have mastery of a given subject matter or skill set versus those who do not. The desired outcome of most standard setting sessions is a fair and defensible minimum passing score, also called a cutoff score or cut score. If there are more than two categories of performance, such as less than proficient, proficient, and advanced, then standard setting involves establishing multiple cutoff scores.
The main process issues involved in standard setting revolve around establishing a standard setting team, facilitating the team in working together to create a vision of a minimally competent person, motivating team members to agree to personally take the assessment, managing standard setting activities so that closure is made on a defensible decision within the time constraints faced by the team, and managing team member absences and personnel changes within the team.
A standard setting team should be comprised of subject matter experts (SMEs), a planner, and a facilitator. All participants on the standard setting team need to sign confidentiality agreements, including the planner and facilitator. SMEs are the team members who make the standard setting decision, with the guidance of a facilitator; the planner handles logistics related to the standard setting activities.
Regardless of the specific method used to set standards, the credentials of the SMEs need to be documented as a part of the standard setting process. This provides evidence to support content validity. The credentials that need to be documented include relevant positions held; the amount of time served in those positions; relevant education, training, licenses, registrations, certifications and certificates held; general contact information; and references.
The planner should have some general understanding of the standard setting process but does not need to be an expert. He also should understand the organization’s procurement regulations. And although it may sound superficial, the planner should understand the feasibility and appropriate scale and style of the standard setting event being planned—in terms of details related to travel, accommodations, and refreshment options during the sessions; expectations and standards of appropriateness can vary greatly from organization to organization. He also should have the ability to communicate effectively with critical decision-making personnel within the organization.
The facilitator should have expertise in the standard setting process and be able to manage the group dynamics of a standard setting session in person or electronically. The planner and facilitator functions can be accomplished by the same person, but they do not have to be.
Sometimes it is optimal for the planner to be a member of the organization engaged in standard setting and the facilitator to be an outside consultant. Or both positions can be effectively hired out. If neither position is hired out, however, it is critical that the in-house employee who is responsible for facilitating the sessions has adequate expertise as well as the independence to do the job. In other words, the facilitator should not be put in the position of having to run a session that is being directed by SMEs who have an agenda in terms of the outcome, or who might be that facilitator’s supervisor now or in the near future. From an accountability perspective, this is a reason to consider hiring out the facilitation component of the work.
The first order of business in any standard setting session, after giving team members a general orientation regarding the purpose of the session and expectations, is to have the SMEs define a hypothetical minimally competent person. There is almost always a humorous backlash to this request. The SMEs perceive that the whole reason they are involved in exam development is to ensure that the highest standards are being met, and they often find it ironic that they are being asked to think in terms of minimal competence. The issue is that most SMEs have optimal or ideal competence foremost in their minds and they need to make a shift from the idea of optimal performance to envisioning the lowest performance level that a person can have and still be considered competent. Facilitating a productive discussion on the concept of minimal competence can enable the SMEs to come to a shared vision of what this means operationally, in terms of applying this to standard setting activities. Success of the standard setting session depends on some type of consensus, agreement, or at least a manageable degree of disagreement on this concept.
It is valuable to have the SMEs personally take the exam as if they were examinees. Even though items have been adequately reviewed prior to the standard setting session, additional item flaws or scoring problems can become startlingly obvious when the exam is actually administered to SMEs on the standard setting team. Some SMEs will not be happy about being asked to take the exam; they may suggest that they are the experts and should not be required to submit to this activity. Other might worry that they will do poorly and be embarrassed.
SMEs can be reassured. Scores should not be shared. Each SME will know her score, and this will inform her participation in the standard setting session. Team members can be reminded that they are members of the standard setting team because they are the experts. They are not being judged; they are the judges and their insight on the standard setting activities will be more fully informed if they experience the items from the perspective of an examinee.
Managing a standard setting session
Effective management of the standard setting session requires that the session planner be aware of who is participating, why they are participating (including potentially hidden agendas), and any constraints related to time, interest, and effort. In workplace contexts, SMEs are sometimes drafted by supervisors or colleagues to be part of a test development and standard setting team. The task is sometimes perceived as an add-on to an already full set of job responsibilities. In credentialing contexts, SMEs might be volunteers who also serve in some leadership capacity within the credentialing organization, or even serve in a position on its board of directors. These team members are generally people who are strongly committed to the organization and are volunteering their time.
Often, team members may be geographically dispersed, making in-person standard setting sessions a challenge to schedule. Because of these factors, it is particularly crucial for standard setting sessions to be well planned and orchestrated in a manner that recognizes and values the above-and-beyond time and effort investment of those involved. In addition to conventional courtesies and expressions of gratitude, team members often feel less put upon when the planner shares the rationale underlying the need for standard setting activities that consume significant amounts of effort and time.
Management of standard setting sessions will vary depending on the method used, the session format (for example, in-person versus electronic), and the assessment format. In-person standard setting sessions are the most desirable format for many reasons, one being that when teleconferencing, nonverbal communication can be missed in coming to a shared vision of minimal competency, which is a core theme of the session. However, web conference sessions, which provide screen sharing or video, are feasible in many contexts. The reality of the resources available for standard setting activities cannot be ignored, but lack of resources is not an excuse to cut corners. If an in-person standard setting session is not going well and team members find themselves spinning their wheels, the facilitator should urge them not to make a rushed decision for the sake of having closure, but should get agreement on the feasibility of having some type of follow-up session, either in person or via a teleconference or web conference.
Ideally, members of the standard setting team will be present for all sessions and none will drop out and need to be replaced. Realistically, people will miss sessions and a member or even several members might need to be replaced. To minimize this occurrence, the planner should recruit team members who understand the time commitment and can be counted on to fulfill their responsibilities. When attrition happens (in spite of optimal recruitment strategies), the planner will need to organize a process to ensure that those who have missed meetings get caught up on decisions as well as the rationale supporting those decisions. This might take the form of providing the absentee member with standard setting session minutes, along with a one-on-one training summary of what transpired. When members are replaced, it is particularly important to ensure that new members are trained to understand the process and rationale that went into the shared vision of the hypothetical minimally competent person.
Note: This article is excerpted from Test Development: Fundamentals for Certification and Evaluation by Melissa Fein.
© 2012 ASTD, Alexandria, VA. All rights reserved.