I personally was looking forward to this session on evaluating informal learning by Saul Carliner, Director of the Education Doctoral Program and an Associate Professor at Concordia University. The reason I was excited about this presentation is because it very closely related to my recent dissertation, A study of informal learning among University of Wyoming Extension educators, soon to be posted on ProQuest.
Kirkpatrick Levels of Evaluation
Carliner began with a quick review of Kirkpatrick Levels of Evaluation:
- Level 1 – Satisfaction
- Level 2 – Learning
- Level 3 – Behavior
- Level 4 – Impact
He introduced a number of informal learning scenarios and explained the challenges of measuring with Kirkpatrick Levels of Evaluation.
Scenario – A manager wants you to collect Level 1 evaluations for on-the-job training.
- Hard to do a satisfaction survey because only onesy and twosy, hard to be candid.
Scenario – A training administrator wants enrollment information for the quick tour of the new Learning Management System, which the department uses to “sell” the system to users.
- No importance to collect the data.
Scenario – A product developmenager asks you to report on the extent of learning from the online help provided with the software her team develops.
- How to identify who is using the system.
Scenario – A board member of your nonprofit wants data on the impact of your confidential health information site that your agency publishes.
- HIPAA issues
Carliner finds that Kirkpatrick Levels of Evaluation does not work for informal learning. He thinks Kirkpatrick Levels of Evaluation is a useful tool, but not in this case.
Informal Learning
Carliner began exploring a definition of informal learning. Basically, he explained that it is how much the learner has control over process, location, purpose, content, and consciousness. In some cases, worker controls and in other cases employer controls.
Carliner explained that employees use informal learning to fill gaps between the classroom and the workplace. It is the practical application of concepts. These are tricks that help novices become a journeyman. According to some studies, informal learning is the source of 70 percent of all work-related knowledge. Evaluating informal learning “fosters a culture of reflection and analysis to improve performance” (Laiken et al.)
Measuring informal learning
Carliner encouraged us to learn from and adopt methods used by other industries to measure informal learning. These industries included museums, marketing, and Web communications.
Museums and informal learning
According to Carliner, museums coined the term informal learning… now free-choice learning. Museums discovered that sometimes we learn because we want to, and sometimes we learn because others dragged us along. Museums use a number of techniques for measuring learning. They read the museum comment book. They measure how long people stay. Museums control space so you can read labels and information board. They use layout to guide the participant along. They also use self-selected survey methods.
Marketing
Marketing agencies are very skilled at measuring direct marketing, and we should learn from their methods. Marketing agencies also rely on recognition studies. Are learners tied to your brand?
Web communications
Carliner also offered a number of techniques used by Web communications such as counting usage stats, running analytic reports, placing satisfaction surveys on every screen, conducting period surveys, and conducting usability tests.
Questions to be answered
- What is the satisfaction with various resources used for informal learning?
- What did workers learn?
- In what ways does the organization benefit from informal learning by workers?
- What is the extent of participation in various informal learning activities?
- Which informal learning efforts that the organization formally supports are providing tangible benefits to the organization?
- How can workers receive recognition for their learning?
Basically, learning and development is interested in what resources are being used by employees to learn. We also need to know how people are learning so they can be recognized and advanced in their employment for skills developed informally. Self-assessments are less painful than exams, e.g., PowerPoint self-assessment. There are other ways to identify how workers learned informally such as portfolios and coaching interviews.
Process portfolio – individual reflects on each item to identify strengths and weaknesses. This can be a challenge to do. Here are instructions for creating a portfolio.
Coaching interview – Simply, ask employees what they know, and take an inventory.
Recognizing prior learning
How can you formally recognize the competencies workers have developed through their informal learning efforts?
- recognized acquired competencies
- employee education records
- skills assessments
- certifications
Certifications are demonstrations of competency to a third party. Typically, they include a written exam and portfolio or work product. A certificate is for attendance and certification is for demonstrated performance. With certification, an evaluation at the individual level, it is a way to assess individuals for new opportunities.
Make people want to learn. Let’s create the right environment. Let them tell us what they learned. Tappestry is one way they can do that. Tappestry is powered by the Tin Can API.
On an organizational or group level, we can measure the success of informal learning by determining the use of individual resources. This is done through analytics and compiling data from individual learning efforts.
Blogs are great learning tools.
Assessing resources
Carliner noted that it is possible to assess satisfaction with resources. It can be done with surveys that focus on individual resources as well as focus on informal learning processes. Focus groups can also assess resources.
The impact of individual resources can be assessed with rater systems, specialized reports, and long-term studies. In terms of long-term studies, new employees followed and be evaluated how they are learning over time.
Other ways to evaluate informal learning include embedding questions in an employee survey and establishing “panels”.
Carliner asked an important question Level 1 evaluations. Why do them? If you are a steady state, make the survey optional to only gather warnings about a problem.
Training is about the long term not next quarter.