Click here to download the PDF version of the transcript.
Stan Skrabut: Well, thanks for taking time to listen to this podcast. It certainly means a lot. As I always note, you could be doing other things and maybe are doing other things, but you’re still hanging out with me and I really do appreciate it. Back in episode 18, I talked about the power of test pools, being able to draw from a lot of questions and putting out these quizzes that students could use to improve their performance. In episode 38, which we just recently did, I talked about adaptive release, which is one way of putting quizzes to work. This week we are going to dig a little deeper by talking about mastery quizzes.
Mastery quizzes are a powerful learning strategy that you can weave into your class to improve performance. We’ll talk a little bit about that as we go on. This idea of mastery quizzes is based on mastery learning. I’m going to spend a few moments talking about that. Mastery learning is based on the idea that given the right conditions, given enough time, given enough resources, every student can achieve mastery. That is the overall goal. If you focus on mastery learning that student performance will improve and it will also improve on course exams. If they are given multiple times to recall information that helps with information retention and also transfers to new areas.
Probably my first experience with this idea of mastery quizzes was when I was at the Airforce Academy Prep School. They used this principle pretty effective that every other day we had a quiz whether we wanted it or not, and then they would have intermittently these major exams. The quizzes were every other day. They were quick quizzes. They’d take 10, 15 minutes. You’d pound out the answers. They were always timed, and you always got feedback on how to improve. This, for me, was a powerful method of studying. I also used this idea of quizzing for my promotion tests in the Airforce. We had to take these exams on our specialty but also on leadership.
We had guides that we used and what we’d do is build these test banks. We’d go through these test banks over and over and over again until we got really comfortable in the material. By using a lot of questions and really comprehensively going through the book and finding every single area and turning it into a question, it was an active way compared to just passively reading the book. I am a huge fan of this idea of using these quizzes. Now these mastery quizzes, the idea is that you’re using a lot of low-stake quizzes and students are giving multiple attempts to go through these quizzes, but in return, they will increase their performance.
Part of the quizzes, you have meaningful feedback that points them where to go. We’ll talk about specific strategies on how to put this into practice. By offering these low-stake tests, what you do is you get them also acclimated to the pressure of taking tests. You actually, by all these low-stake tests that you can remove that high pressure. This allows them to focus more on the content and the material and what they’ve gotten wrong and how to improve that as opposed to focusing on the grade. That’s what we really want in the end is for them to shift their focus to the content as opposed to focus to the grade. How does this mastery learning work?
This guy Benjamin Bloom back in the ’70s, he outlined the strategy that if you provided a content and these formative assessments, along with feedback and corrective procedures, this would lead to mastery learning. He was really the individual that put this onto the map. The idea is that instructors would go out and they would teach these important concepts and skills that they wanted students to learn. This was typically done in small blocks and then after the instruction, there was an assessment that would identify what students had to work on. The instructors would take that information, give it back to the students, give that feedback and corrective measures.
Students would work on that, then they would get another formative assessment to continue further on that process, to allow them to keep honing into those learning goals. There were always these modifications that you’re trying to present it, not just the same way, as they say, louder but you were doing it from a different perspective, a different angle to see if you could find out where they could best learn. I think this ties in really strongly with universal design for learning because that’s also what you’re doing. You’re looking for these pinch points.
You’re looking for these areas where students are not getting it right the first time and providing additional supports and feedback and different options that will help students learn this material. This is a powerful strategy. A couple of things about these formative assessments is really comes into the feedback piece and how to do that. Benefits for using these low-stake quizzes. Number one, this is tied to mastery. It gives them an opportunity to maul over the content, actively try to find the right answers. When they engage the content this way, it will stay with them definitely longer.
It also helps these students identify gaps in their knowledge and also provides them with feedback on where they can go to improve those, fill in those learning gaps. The idea if you do this correctly and use these mastery quizzes that you’re basically encouraging them to engage with the content, to interact with the content because if they get something wrong, you’re pointing them where to go and telling them how to use this material or where they can find that material. They are low-stakes, they get opportunities to do this over and over and over again, it puts learning pretty much in their control.
They can choose not to do it, but there’s things we’re going to talk about how to make this more effective where this should be part of their grade. They have an opportunity to always improve their grade, which I’m a fan of. This ties back to this idea of gain-based learning, where your score is always going up. That’s the overall goal, so anything I can do to weave into my course where students are gaining points as opposed to learning points, I’m a fan of that. One of the other benefits from these low-stake quizzes is to reduce the anxiety about taking quizzes and exams, that the more that you practice this activity, the less stress that you’ll have in the more high-stakes exam.
By making these things online and self-grading, it also frees up the instructor to focus on areas where they are really needed. This opportunity to work with learners more one on one or improve the quality of the other instruction, this frees up time for the instructor. Now, with everything, there’s also a cost to doing this. One of the first costs is it does take some time to do it, but there’s strategies. As I talked about in the test pools, you don’t have as an educator to be the one creating the exam. You have to be the one vetting the exam, but you don’t have to be the one creating it.
You could use publisher content, so if you’re tied to a textbook and the publishers have put test banks out, you can take these test banks, go through it, make sure that the questions are what you want, and put those questions into your test pool that you can then draw on for these mastery quizzes. You can also have students design the exam by having them create test questions. Once again, it’s just a matter of you vetting those, using old exams as long as the content is appropriate. Those are all different strategies that help with the time, but there is time involved in actually building these quizzes.
The nice thing is once you have a quiz built, then it’s a matter of increasing the size, increasing the test pool. I really do think that the more effective quizzes are tied to a test pool and that you’re using all the tools, identifying which objectives questions are tied to, this will help give all kinds of wonderful measurements after you’ve administered this test. Those are things that are important. Certainly, when it comes to doing this, if you have only done this on paper and pencil, then moving to an online strategy, there’s learning costs associated with learning how to use the technology and do you have the adequate support for this.
This applies to both the instructor but also the students. There is instruction that has to go on to teach the students how to use these test banks, how to use a learning strategy that’s associated with them. Those are some of the costs that come into play. Let me talk about a couple of different ways that you can weave this into your classrooms. One is preparation for face-to-face class. This comes from a 2017 article, “The Use of Mastery Quizzes to Enhance Student Preparation,” by Richard Egan, Jerry Fjermestad, and Davida Scharf.
The article, “The Use of Mastery Quizzes to Enhance student preparation.” This was a class on ethics and impact on technology and society, part of a computer science program. The quizzes were set up associated with the chapters that the students had to read. These quizzes would be available for seven days before the class was actually given and they would be available to the students so the students could read, take the quiz. They could do it as many times as they wanted. Each of these quizzes was typically five randomly selected multiple-choice or true-false questions and this would go until the quiz closed.
The quiz would close just before the class started and students could do it as many times as they want. There was no delay between the time they took one test to the time they took the quiz again. As a result, students were better prepared for these lectures because they had gone through the course content and they were ready to go. The quiz stopped just before class started and became part of their grade and their grade was that the top score would be part of the grade. The results of this survey they did on the students says over 90% thought the quizzes and taking them multiple times helped them better learn the material and they also thought that this is something that should continue as part of the course.
That’s one way, preparing for face to face classes. I recommend the same thing for discussion boards, that discussion board should be encouraged, online discussion boards before the class and graded. That way the instructors know where the strengths and weaknesses are and these exams do the same thing. They help the instructors know what the strengths and weaknesses are before going into that class. Another way that you can use these mastery quizzes is to actually prepare for a more high stakes exam. This is a 2020 article, “The Educational Value of Online Mastery Quizzes in a Human Anatomy Course for First-year Dental Students,” by Lisa Lee, Rollin Nagel, and Douglas Gould.
Basically what they did is they put these mastery quizzes together and students had them available five days before the examination. There were 10 multiple-choice questions and the answers were randomized, the quizzes were built from materials from previous classes. They built the test bank actually pretty rapidly from what I can gather and the students could take these tests as many times as they want. The high score was the one that was added to the course grade. The results of this, the students definitely took advantage of this to reinforce content, clarify concepts, and as a way of preparing for the exam.
One of the interesting things, although that the mastery quizzes did not improve or reduce their exam scores, but multiple regression analysis showed that initial mastery quiz scores had a predictive value for their examination performance. This was kind of a positive trend. As they were administering these quizzes, one of the things that they did not do is, they did not tell them what the correct answer was. However, they did provide feedback where the correct answer could be found and this encouraged this diving in and going through the content.
Additionally, as the students were using this material there was a significant high number, 90% that they preferred taking these online quizzes over doing them on paper. That’s heartening if you’re thinking about going to automated quizzes which I would recommend. Students are a fan of this, that it allows them to take it whenever they want and I would keep them open. I would make them available. They’re closing them at a certain time but they leave it open for a large window. I would do the same thing. I would create a reasonable size window where students could go in and take the quizzes over and over again to improve their score. It just provides the opportunity to keep diving into the content.
I also noted in this particular study that the students delayed taking the mastery quizzes until immediately before the exam. They were basically figuring out if they were ready for the exam or not. Definitely a strategy. We looked so far at preparing for it in face to face class using mastery quizzes. We also looked at preparing for exam. Another study was that quizzes could be used instead of homework and this was in an advanced placement chemistry course. Instead of assigning homework they had these quizzes that students would take. The study that students had daily quizzes.
The way the course was set up every single day there was one of these mastery quizzes that was being administrative in the course and it took 10 15 minutes for each of these quizzes. They weren’t very long but they kept focusing on the material. US Department of Education found that having students take a test is almost always a more potent learning device than having students spend additional time studying the target material. This is definitely what I found that you’re basically making this more active learning than passive learning. If they’re just reading the book over and over and over it’s very passive but when you have them working on a quiz it’s very active.
They’re trying to get the right answer. They don’t want to do the wrong answer so they’ll do this. When these students in this chemistry course, they had to always be ready for these quizzes. They were always engaged with the material to be ready for the quiz. When these instructors set this up, they made sure that whatever was being taught the past two days was not in the current quiz. Basically, the quizzes kept focusing on previous materials and like I said, the quizzes were very short. Once all the students had taken the quiz then the answers were unlocked so they could see what the feedback and the correct answers were.
Students initially frustrated that they had all these quizzes but after a while came to realize how much they were learning because of these quizzes. Because they kept seeing a lot of the same content over and over again, it helped to reinforce things. When students who were constantly getting the information wrong realized that this material wasn’t going away so they started to engage with their instructors and ask for additional assistance in order to overcome that obstacle. Definitely, a lot of benefits doing this and once again students kept improving their performance.
Speaking of improving performance, SUNY Cortland psychology course they used this specifically to improve performance and they used rather large test pools or test banks. They had 200 questions for each chapter and so from this pool, they would create these mastery quizzes and randomly draw questions from the test bank. Students had to earn 80% or higher in order to get credit for that particular bank. If they got below 80, it would not count and therefore they had to keep doing the quizzes over until it finally counted as part of their grade.
The overall quizzes that they contributed up to 25% to the students’ grade but in some cases, they restrict the number of attempts that they offered for each quiz. Different ways that you can weave that in if you want. Talking about using this to improve performance, when you offer these short quizzes at the start of classes with feedback, this is an effective strategy to help them improve their performance. It helps them prevent this cramming getting ready for an exam and the procrastination that goes with that. By offering these quizzes at very regular short intervals, I learned at the Air Force Academy Prep School, we had one every other day, you were constantly brushing up on your information and building on it as opposed to we give a typical mid-term and a final exam.
That’s a lot of time, a lot of material to be preparing for and these little mastery quizzes along the way would help them stay on target. This could be done for many different disciplines. Another way that you can use these mastery quizzes is through adaptive release. Perhaps that you give them a series of these short mastery quizzes to work on that they have to get to a certain level and once they’ve gotten to that level then it opens up that major quiz. Another way that you can look at this is use it as gates, that once they have successfully gone through these gates then they can have access to that major quiz or major exam. That’s definitely a strategy that I’m a fan of.
Before they can go on to the next module depending on how your class is set up, that you can do the same thing. Strategies for success. The National Center for Academic Transformation pulled together lots of great information on making these mastery quizzes more successful. They realize that it is an effective tool when used appropriately for student learning. Here are some of the tips that I took from this and also a number of articles and pulled them together with these strategies of success for mastery quizzes. First, if you can automate your quizzes make them computerized that will help as far as scoring, providing feedback, effective and consistent feedback. It will save in the time of grading. Lots of benefits.
In your learning management systems and most places have them, that you can build these quizzes and do that. There’s also other products, but Math Department uses a quizzing tool to help students practice. The nice thing is with these tools is students can do this over and over and over again. The computer will never get tired of them and it allows them to just practice the material over and over. Another important strategy, these quizzes should be low stakes. It should not be a significant chunk of the grade, but enough to make it hurt. Anywhere 10 to 25% of the grade is important. Therefore these quizzes should be required rather than voluntary, that you shouldn’t give them options.
The idea is you’re going to use the quizzes as a learning tool to help them improve their skills. It should be a mandatory part of your class. As far as getting a score, using the highest score is always I think, a good approach. That’s the one that seems to be recommended often, but you can also use it in adaptive release where you need a minimum score in order to release additional content, so different ways that the scores can be used. The idea is that you really want this to be an interactive exercise, where they go through the material, go back and engage the content, come back, repeat the quiz and go back and forth.
They’re not just seeing the same quiz over and over, ideally that you want to go ahead and have a test pool where you’re drawing this information, that’ll help it out quite a bit. This should not be a one and done. That’s more summative type of assessment. This is formative. Having the ability to repeat these quizzes over and over again, where they can keep looking at how well they know the content, you should make sure that you allow that capability. The idea is it’s supposed to be a learning tool. As far as feedback goes, students should have the ability to see what they got wrong.
You may not necessarily want to give them the correct answer right away, but at least let them know what they got wrong. Also provide quality feedback to tell them where to go find the right answer. This feedback should be pointing back to notes that you’ve provided to the textbook, to whatever content that you have, and let them know where they can go find this. One thing that showed up there that was interesting, so we have these textbooks that have the answers in the back of the book. There was something that was written in one of the articles that spoke about that.
Let me see if I can find it. “When giving students access to practice exercise answers, not contingent on their answering the question is worse further learning than giving them no study aid, thus the typical study guide that has quiz answer keys at the back may be attractive to students, but counterproductive to learning. Giving it too easy does not help the learning.” I thought that was interesting. Making them work with that feedback, making them go back to the resources, looking it up for themselves will help learning more than just simply giving them the correct answers.
Really the first time I saw that but, as I think about it, it makes a lot of sense to me. When providing this feedback, some of the elements that you should have is letting them know if they answered correctly or not. Having them, point them to the resources where they can find the correct answer. Definitely useful stuff. In order to make it easy for grading for the computer, definitely recommend sticking to multiple-choice, matching, true and false. Basically, the things that do not require a typed in response. The reason for that is, if someone misspells a word, they’re off by a letter that indicates that it is incorrect. If spelling happens to be the important piece on there, yes, definitely include those.
For essays, they would have to wait for somebody to grade the essays and that would hold things up. Try to just keep it simple in terms of how the computer can grade and provide feedback makes it– That’s the route that you should go. Use the power of these test banks. One of the things that a good test bank will do typically in the learning management system is it will randomize questions. If you’re working for a test pool, it will randomize pulling questions out of the test pool, which is nice. Then randomize the presentation of those questions, but it can also randomize answers.
When you write questions that randomize answers, you may want to stay away from responses that say, A and B or all of the above, because if they randomize, then that just would not make sense. The number of questions that you use should be relatively small, 15 to 25 is probably optimal if you’re drawing it from a large test pool that has 100, 200 questions. My time in the Air Force when we were helping with qualification examinations, we had thousands of questions that we could draw from and we often did. That is the idea of mastery quizzes.
One, really you need to consider it and consider a lot of them that at a minimum, you should probably try to aim for one a week based on the content that you’re presenting. You can have multiple mastery quizzes as you’re going in there. The mastery quizzes can be over an entire chapter, for example, or it can be on a specific concept. You have the ability to control that. Consider using them, certainly grade them and make sure that students are getting credit for that. You may want to also consider them as gates to other content that they have to get a certain level. Maybe to the midterm exam, before you give the midterm exam that they have to pass these five little mastery quizzes at a certain level before they get the midterm exam.
I think you’re going to see what your students are able to do and what they can do improve compared to other classes. I would love to hear your experiences with this and if you’ve tried this. I’ve had a lot of success with using these quizzes. That’s why I wanted to make sure I talked about this again, but that’s what I have for you today and before I let you go, here’s a plug for my book, Read to Succeed.