With different questions guiding evaluation and without formal objectives and criterion-referenced tests, how do training and development professionals evaluate informal learning? Effective evaluation of learning programs starts with clear questions. Methods merely provide a means for collecting the data to answers those questions. The questions guiding the evaluation of informal learning include:
- What did workers learn informally?
- How do workers learn informally?
- How can workers receive recognition for their informal learning?
- What is the extent of participation in various informal learning activities?
- What is the extent of satisfaction with various resources used for informal learning?
- In what ways does the organization benefit from informal learning by workers?
- Which informal learning efforts that the organization formally supports are providing tangible benefits to the organization?
- How can organizations better support informal learning efforts?
Some of these questions focus on learning at the individual level, while others help evaluate learning across groups of workers.
Evaluating individual learning
Learning that happens as a result of an individual initiative or serendipitous activity is difficult to identify, much less label and report. Getting at this information involves a number of things.
Identify what workers have learned. Because workers establish their own objectives for some informal learning and other informal learning occurs unconsciously—and without objectives—often neither the workers themselves nor training and development professionals are aware of what individual workers have learned. This type of evaluation helps unearth what learning occurred.
Identify how workers have learned. Training and development professionals can use this type of evaluation to find out whether the resources provided for informal learning were actually used, and if not, which resources were used instead and why.
Recognize acquired competencies. This type of evaluation lets employers label the skills, knowledge, and abilities learned by workers and formally acknowledge it in employees’ education records. Such formal acknowledgment is useful when assigning job tasks or deciding on raises and promotions. This recognition also motivates workers to continue their informal learning efforts because it provides them with a tangible “credit” for individual activities often undertaken without anyone’s involvement or awareness.
Training and development professionals use the following evaluation methods to gather information about individual learning that occurs informally.
Self-assessments are instruments that workers use to assess the extent to which they have learned material in a particular domain. In many ways, self-assessments are like quizzes, in that they involve answering a series of questions or demonstrating a skill and then receiving feedback on performance. Self-assessments work best for identifying what workers have learned in domains that are well defined, or for content that has been externally defined.
With self-assessments, however, workers generally administer the “test” on their own, and the results are intended to reflect what workers have learned and where they need to continue their efforts. The results may or may not be recorded. Effective formal self-assessments usually assess objectives that would be covered in a formal training program and focus on how workers might respond to various work-related situations. Provide correct answers to questions and then explain why the answer was correct, so workers can learn from the experience.
For broader skills that cannot be addressed in a simple written assessment, responses tend to focus on distinctions in ways of thinking—such as a worker who has fully integrated the learning, a worker who has partially integrated the learning, and a worker who has not yet integrated the learning.
Process portfolios are collections of work accompanied by reflections and commentary about how the worker created the work samples, what the worker learned in the process of doing so, and how she felt about the learning process. Portfolios are usually created in collaboration with the worker’s manager, mentor, or another senior colleague.
The portfolio is called a process portfolio because its primary focus is on the means by which workers created the work samples. Portfolios grow with time and provide workers with opportunities to reflect not only on individual assignments but also on learning and development that occurred over time. They are often used in career development and counseling discussions.
A process portfolio contrasts with the better-known showcase portfolio, which highlights workers’ best work and merely states how they contributed to a project, rather than what they learned from it. Workers use showcase portfolios to provide evidence of qualifications for a position.
There are two components to a process portfolio, in which the worker provides commentary on his or her work samples. Workers use this part of the commentary in both process and showcase portfolios. It provides general information about the project and their contribution to it. The second part of the commentary is for specific information about workers’ professional development in the context of a project. Workers use this part of the commentary only in process portfolios; it is separated so that workers can easily create showcase portfolios from the same materials used to create the process portfolios.
Process portfolios can help training and development professionals determine how workers learned in addition to helping identify what workers learned. In particular, when responding to the second part of the commentary, some workers might identify parts of the process that provided learning opportunities. This commentary also asks workers to share their feelings about the learning process so that those reviewing the portfolio gain insights into the feelings of support and frustration the worker felt.
Coaching interviews with managers, mentors, senior workers, and similarly experienced and trained co-workers provide learners with the opportunity to get a second opinion on their learning processes. An effective coach helps workers gain a realistic perspective on the extent of their learning, not only asking workers to state what they have learned, but also asking them to provide concrete evidence that demonstrates that learning actually occurred.
In some instances, workers might not believe that they learned anything at all. By actively listening to workers, however, coaches might gain an alternate perspective. For example, a worker might not feel that he or she learned anything on the job or through a developmental experience, but then display knowledge that would not have been intuitive—and could only have resulted from the learning experiences. The coach uses these clues to suggest that learning has indeed occurred.
Coaching also provides an opportunity to explore how learning occurred after identifying what workers learned. To find out how workers learned, coaches should specifically ask workers about critical incidents in the work process, identifying what occurred at each of these junctures and then exploring how workers felt about these learning opportunities.
Employee education records track training completed by employees. Organizations use this information to acknowledge skills maintained and added. Nearly all large organizations and many medium and small organizations keep these records. Most organizations have established ways of automatically crediting workers for completing classroom, self-study, and virtual e-learning programs by tracking these through learning management systems, which transfer the information to employee education records.
Other information must be entered manually into the employee education record, such as completions of training offered by third-party providers, completion of academic courses, and completions of informal learning efforts. Organizational policies vary on who can update records and the evidence that must accompany the updated records. Some organizations let workers update their own education records and operate on an honor system. Others only let managers or human resources specialists update records and require that workers provide concrete evidence of completions.
Skills assessments are another means of recognizing informal learning by crediting workers for developing skills partially or completely through informal means. Many learning management systems provide the capability for tracking skills, but few organizations actually take advantage of this capability. That’s because most skills assessments involve rating workers on a list of specific skills that often exceeds 100 items. Some of the skills on these lists are general ones that apply to all jobs, such as the ability to work cooperatively with others and the ability to use common software applications, such as word processing, spreadsheets, and email. Other skills are specific to a class of jobs, such as skills associated with litigation and managing intellectual property among legal advisers, and the skills associated with providing technical support and correcting software problems among information technology professionals.
Although some skills assessment schemes ask people to rate skills in general terms, such as excellent, very good, or poor, more effective skills assessment schemes use rating schemes based on concrete and observable measures, such as “I have never heard of this skill,” “I can describe this skill but have never demonstrated it,” “I have demonstrated this skill once or twice,” “I have demonstrated this skill three or more times,” and “I advise others on how to acquire this skill.” This prevents workers from under- and over-rating their skills.
Skills assessment efforts face these challenges:
- Identification of skills. Some organizations choose to identify and assess broad skills, which often fail to provide specific information. Others identify highly specific skills, but the lists become so long (some exceeding 150 skills) that the assessments are too time-consuming and intellectually exhausting to complete.
- Self-reported data, usually entered by the worker with little or no verification. A second evaluation from a manager helps, but it is still opinion-based data.
- Timeliness, because the skills assessment represents the list of skills identified in a particular worker at a particular point in time. Some skills may atrophy with disuse, and workers may have acquired other skills that will not be reflected unless they have updated their skills assessments.
Certifications are another valuable evaluation tool for recognizing competencies. Workers demonstrate their competency through one or more of these means:
- passing examinations that demonstrate familiarity with a body of knowledge central to the field
- demonstrating competencies according to criteria identified and assessed by external assessors
- preparing a portfolio for evaluation by external assessors, that validates the originality of the work and provides evidence of competency to perform the work.
Although to be eligible for some certification exams, workers must have completed required formal training, the certification exam itself focuses on competence, regardless of how workers actually acquired it. As a result, certifications recognize informal learning by providing external, third-party recognition of skills and knowledge developed partially or completely through informal means.
Certifications are offered by several groups: professional and trade associations, who usually certify competence in a profession; technology providers, who usually certify the ability to competently install, use, or repair a machine or software; and internally within organizations to certify the ability to perform a job that is vital to the organization.
Learning badges are another type of recognition offered by some websites and independent learning organizations (such as the open courses at MIT) that provide external verification that a participant has completed a particular learning activity, and identify the skill that the learner has acquired. Learning badges are a relatively new development, so their impact on the evaluation of informal learning is not yet known.
Additional ways to evaluate and recognize individual learning exist. For details, check the European guidelines for validating non-formal and informal learning published on the web by the European Centre for the Development of Vocational Training.
Note: This article is excerpted from Informal Learning Basics by Saul Carliner.
© 2012 ASTD, Alexandria, VA. All rights reserved.