I looked forward to Saul Carliner’s session on evaluating informal learning. My dissertation on informal learning among University of Wyoming Extension educators closely connects to the topic.
Kirkpatrick Levels of Evaluation
Carliner began with a quick review of Kirkpatrick Levels of Evaluation:
- Level 1 – Satisfaction
- Level 2 – Learning
- Level 3 – Behavior
- Level 4 – Impact
He introduced several informal learning scenarios and explained the challenges of measuring with Kirkpatrick Levels of Evaluation.
| Evaluation request | Challenge discussed |
|---|---|
| Gather feedback on on-the-job training | Small participant numbers can make candid feedback difficult. |
| Report enrollment for an LMS introduction | Clarify whether collecting attendance data serves a useful purpose. |
| Measure learning from software help | Identify who uses the help resources. |
| Assess a confidential health information website | Account for privacy concerns when collecting data. |
Carliner finds Kirkpatrick Levels of Evaluation do not work for informal learning. He thinks Kirkpatrick’s Levels of Evaluation are a useful tool, but not in this case.
Informal Learning
Carliner described informal learning in terms of learner control. Who determines the process, location, purpose, and content? Sometimes, the worker controls, and in other cases, the employer does.
Carliner explained employees use informal learning to fill gaps between the classroom and the workplace. It is the practical application of concepts. These are tricks that help novices become journeymen. According to some studies, informal learning is the source of 70 percent of all work-related knowledge. Evaluating informal learning “fosters a culture of reflection and analysis to improve performance” (Laiken et al.).
Measuring informal learning
Carliner encouraged us to learn from and adopt methods used by other industries to measure informal learning. These industries included museums, marketing, and Web communications.
Museums and informal learning
According to Carliner, museums coined the term informal learning… now free-choice learning. Museums discovered that sometimes we learn because we want to, and sometimes we learn because others dragged us along. Museums use several techniques for measuring learning. For example, museums review visitor comments and measure how long people stay. Museums control space so that you can read labels and information boards. They use layout to guide the participant along. They also use self-selected survey methods.
Marketing
Marketing agencies are very skilled at measuring direct marketing. We should learn from their methods. Marketing agencies also rely on recognition studies. Are learners tied to your brand?
Web communications
Carliner also offered several techniques used in Web communications. For example, examine usage statistics and analytics. Supplement these with surveys and usability tests.
Questions to be answered
- What is the satisfaction with various resources used for informal learning?
- What did workers learn?
- In what ways, does the organization benefit from informal learning by workers?
- What is the extent of participation in various informal learning activities?
- Which informal learning efforts that the organization formally supports are providing tangible benefits to the organization?
- How can workers receive recognition for their learning?
Basically, learning and development is interested in what resources are being used by employees to learn. We also need to understand how people are learning so that we can recognize and promote them in their jobs for the skills they develop informally.
Self-assessments are less painful than exams, e.g., PowerPoint self-assessment. There are other ways to identify how workers learned informally, such as portfolios and coaching interviews.
Process portfolio – individual reflects on each item to identify strengths and weaknesses. This can be a challenge.
Coaching interview – Simply ask employees what they know and take an inventory.
Recognizing prior learning
How can you formally recognize the competencies workers have developed through their informal learning efforts?
- recognized acquired competencies
- employee education records
- skills assessments
- certifications
Certifications are demonstrations of competency to a third party. Typically, they include a written exam and a portfolio or work product. A certificate is for attendance, and certification is for demonstrated performance. With certification, an evaluation at the individual level, it is a way to assess individuals for new opportunities.
Make people want to learn. Let’s create the right environment. Let them tell us what they learned.
On an organizational or group level, we can measure the success of informal learning by determining the use of individual resources. We achieve this by analyzing data and compiling information from individual learning efforts.
Blogs are great learning tools.
Assessing resources
Carliner noted it is possible to assess satisfaction with resources. It can be done with surveys that focus on individual resources and informal learning processes. Focus groups can also assess resources.
We can assess the impact of individual resources using rating systems, specialized reports, and long-term studies. Follow new employees over time to understand how their learning develops.
Other ways to evaluate informal learning include embedding questions in an employee survey and establishing “panels”.
Carliner asked an important question about Level 1 evaluations. Why do them? If you are in a steady state, make the survey optional to only gather warnings about a problem.
Training is about the long term, not next quarter.
