#astd2013, Evaluating Informal Learning with @saulcarliner

I personally was looking forward to this session on evaluating informal learning by Saul Carliner, Director of the Education Doctoral Program and an Associate Professor at Concordia University. The reason I was excited about this presentation is because it very closely related to my recent dissertation, A study of informal learning among University of Wyoming Extension educators, soon to be posted on ProQuest.

Kirkpatrick Levels of Evaluation

Carliner began with a quick review of Kirkpatrick Levels of Evaluation:

  • Level 1 – Satisfaction
  • Level 2 – Learning
  • Level 3 – Behavior
  • Level 4 – Impact

He introduced a number of informal learning scenarios and explained the challenges of measuring with Kirkpatrick Levels of Evaluation.

Scenario – A manager wants you to collect Level 1 evaluations for on-the-job training.

  • Hard to do a satisfaction survey because only onesy and twosy, hard to be candid.

Scenario – A training administrator wants enrollment information for the quick tour of the new Learning Management System, which the department uses to “sell” the system to users.

  • No importance to collect the data.

Scenario – A product developmenager asks you to report on the extent of learning from the online help provided with the software her team develops.

  • How to identify who is using the system.

Scenario – A board member of your nonprofit wants data on the impact of your confidential health information site that your agency publishes.

  • HIPAA issues

Carliner finds that Kirkpatrick Levels of Evaluation does not work for informal learning. He thinks Kirkpatrick Levels of Evaluation is a useful tool, but not in this case.

Informal Learning

Carliner began exploring a definition of informal learning. Basically, he explained that it is how much the learner has control over process, location, purpose, content, and consciousness. In some cases, worker controls and in other cases employer controls.

Learner Control in Informal Learning

Carliner explained that employees use informal learning to fill gaps between the classroom and the workplace. It is the practical application of concepts. These are tricks that help novices become a journeyman. According to some studies, informal learning is the source of 70 percent of all work-related knowledge. Evaluating informal learning “fosters a culture of reflection and analysis to improve performance” (Laiken et al.)

Measuring informal learning

Carliner encouraged us to learn from and adopt methods used by other industries to measure informal learning. These industries included museums, marketing, and Web communications.

Measuring Informal Learning

Museums and informal learning

According to Carliner, museums coined the term informal learning… now free-choice learning. Museums discovered that sometimes we learn because we want to, and sometimes we learn because others dragged us along. Museums use a number of techniques for measuring learning. They read the museum comment book. They measure how long people stay. Museums control space so you can read labels and information board. They use layout to guide the participant along. They also use self-selected survey methods.


Marketing agencies are very skilled at measuring direct marketing, and we should learn from their methods. Marketing agencies also rely on recognition studies. Are learners tied to your brand?

Web communications

Carliner also offered a number of techniques used by Web communications such as counting usage stats, running analytic reports, placing satisfaction surveys on every screen, conducting period surveys, and conducting usability tests.

Questions to be answered

  • What is the satisfaction with various resources used for informal learning?
  • What did workers learn?
  • In what ways does the organization benefit from informal learning by workers?
  • What is the extent of participation in various informal learning activities?
  • Which informal learning efforts that the organization formally supports are providing tangible benefits to the organization?
  • How can workers receive recognition for their learning?

Basically, learning and development is interested in what resources are being used by employees to learn. We also need to know how people are learning so they can be recognized and advanced in their employment for skills developed informally. Self-assessments are less painful than exams, e.g., PowerPoint self-assessment. There are other ways to identify how workers learned informally such as portfolios and coaching interviews.

Process portfolio – individual reflects on each item to identify strengths and weaknesses. This can be a challenge to do. Here are instructions for creating a portfolio.

Coaching interview – Simply, ask employees what they know, and take an inventory.

Recognizing prior learning

How can you formally recognize the competencies workers have developed through their informal learning efforts?

  • recognized acquired competencies
  • employee education records
  • skills assessments
  • certifications

Certifications are demonstrations of competency to a third party. Typically, they include a written exam and portfolio or work product. A certificate is for attendance and certification is for demonstrated performance. With certification, an evaluation at the individual level, it is a way to assess individuals for new opportunities.

Make people want to learn. Let’s create the right environment.  Let them tell us what they learned. Tappestry is one way they can do that. Tappestry is powered by the Tin Can API.

On an organizational or group level, we can measure the success of informal learning by determining the use of individual resources. This is done through analytics and compiling data from individual learning efforts.

Blogs are great learning tools.

Assessing resources

Carliner noted that it is possible to assess satisfaction with resources. It can be done with surveys that focus on individual resources as well as focus on informal learning processes. Focus groups can also assess resources.

The impact of individual resources can be assessed with rater systems, specialized reports, and long-term studies. In terms of long-term studies, new employees followed and be evaluated how they are learning over time.

Other ways to evaluate informal learning include embedding questions in an employee survey and establishing “panels”.

Carliner asked an important question Level 1 evaluations. Why do them? If you are a steady state, make the survey optional to only gather warnings about a problem.

Training is about the long term not next quarter.

Preparing for my next informal learning Webinar

Right now, I am working on supporting content for the third Webinar of a six part series. This installment is on creating content to support informal learning.

The upcoming Webinar will focus on four tools that I believe support informal learning in different ways. These tools are blogs, wikis, Flickr, and YouTube.

Presently, I have finished working on three of them, and I am starting to work on the fourth.

Here is are the pages completed:

If you have time, please take a look and let me know what you would like to see. The idea of these pages is to have additional support content for viewers of the Webinar.

Leveraging the university library’s RSS feeds

Throughout the summer, I will be giving Webinars to our Extension educators on strategies for improving their informal learning. This month, I am presenting on information collection strategies like RSS feeds and Twitter feeds. One of the ideas presented is on how to create a search query in the university’s digital journals, and save it as a RSS feed in Google Reader.

Rather than repeatedly search the university’s digital journals using the same search queries, you can set a search query and wait for updates to appear in your RSS aggregator. Here is a short video showing that process:

Session notes: The Yin and Yang of Formal and Informal Learning

The second afternoon session I attended The Yin and Yang of Formal and Informal Learning by Allison Rossett and Frank Nguygen. They started their session by explaining that formal training was the primary training method used by the enterprise; however, personal preference is informal learning. Rossetti cited Bozarth when reporting that 83% of companies saw value in informal learning, but only 36% employed it.

In the enterprise, courses are not dead and very much alive ,but there is an interest in employing informal learning strategies. Individual informal learning makes great sense, but for the enterprise there are more concerns. One of the challenges is that informal learning lacks certification, thus does not meet the promise of the enterprise. Enterprise or companies have a promise of service they must meet.

Here are some important questions to ask.
Is success defined?
Who chooses ends and means?
Must we prove that we can do what we say we do?
Are we in a position to go towards more choice and freedom? Is the culture forgiving?

Rossetti and Nguygen created a study to examine the disparity between what is wanted (informal learning) and what actually happens (formal learning). Their study resulted in a tool of 15 questions and two table of strategies. This tool is called the YinYang Tool, and it can be found at Http://yinyang.frankn.net. This tool helps to determine if a program or company is a good fit for informal learning strategies.

The Coast Guard used the YinYang tool to see if informal learning was good for them and new boat program. They stuck with a formal approach because of dangers of getting it wrong.

When you are talking about informal learning, you are talking about choice. The tool helps you determine the choices you have for delivering training. It will help you identify opportunities how to present training. Use the tool as a conversation starter to discuss training options and possibilities. Remember, you can evaluate a whole program or just sections of a program with the tool.

The goal is not to be informal or formal. The goal is to be better at what we do.

Informal learning – how to satisfy your thirst for more knowledge

Building skillsLast week, I had an amazingly powerful informal learning week. In short, I have been working on a Web project that uses a database to query data. When planning the project, the group I was working with wanted to present this information on a Google map. Well, I finally got to that part of the project, and I was not as smart on the subject as I should have been. After spending a number of hours on this learning project, and a lot of time on Google and Youtube, I was able to sort out my problems. The greatest majority of learning projects start with a problem to solve.

In 1968, Allen Tough conducted research on learning projects. What was a learning project? Why did people take on learning projects? I like many other people take on a learning project because we need to learn and apply a new skill (Tough, 1968). As Tough explains, a learning project lasts at least seven hours and can be made up of one or more learning episodes typically 10 minutes to 4 hours in length. In one-third of the cases, people wanted to learn so they could help others. The greatest majority of participants indicated they had prior skill with a task, they but they wanted to improve and become more efficient. Tough also noted that learners tend to focus on the practical aspects for learning rather than academic needs. As I reflect on my recent learning project, I would have to concur with his findings.

Earlier this week, I was working on a learning guide for learning objectives. During the class I am taking, we learned about learning objectives; however, the session only skimmed over the material and lasted about 15 minutes. Because I was not totally satisfied with the amount of information I received in class, I started a self-directed learning project. In this project, I explored resources I had on hand such as Mager’s book, Preparing Instructional Objectives: A Critical Tool in the Development of Effective Instruction and the Air Force Manual 36-2236, Guidebook for Air Force Instructors. I also conducted a solid search on Google and Youtube for useful resources relating to learning objectives and Bloom’s Taxonomy. Finally, I revisited resources I had squirreled away in Diigo on learning objectives and Bloom’s Taxonomy. Although, a formal learning class reminded me of the importance of learning objectives, my self-exploration helped my understanding of the topic. The results of my learning resulted in not only a new learning guide but also a lesson on learning objectives.

Also this week, on LinkedIn‘s The eLearning Guild discussion group, Ed Lines asked the question “Have you ever used YouTube to teach yourself something?” While writing this post, I had a moment to reflect on that question. I have to be honest, I often use Youtube to help move my learning forward. I use it for ideas for my martial arts training, when I am stuck on SPSS problems, learning how to use RPGMaker, getting inspired by others who have broken the code, and most recently, I used Youtube to learn more about Google maps, learning objectives, and Bloom’s Taxonomy.

A 1996 report from the US Department of Labor indicates 70% of all learning is informal (Gilmore, 2008). Examining my learning behavior, I would have to concur with this finding. I therefore believe it is important to learn how to learn well. I believe my strategies are pretty solid, I am curious how you go about satisfying your thirst for more knowledge?


Gilmore, A. (2008). Hands off: Facilitating informal learning. Certification Magazine, 10(10), 46 – 49. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=bch&AN=34615347&site=ehost-live

Tough, A. M. (1968). Why adults learn: A study of the major reasons for beginning and continuing a learning project. Toronto: Ontario Inst. for Studies in Education. Retrieved from http://search.ebscohost.com.proxy.uwlib.uwyo.edu/login.aspx?direct=true&db=eric&AN=ED025688&site=ehost-live