Click here to download the PDF version of the transcript.
Stan Skrabut: Well, thanks for taking time to listen to my podcast. It certainly means a lot. I really, really do mean that. As I always note, you could be doing other things, but you’re hanging out with me. I really appreciate it. Well, my class is over, so now I have to start exploring some other tools and strategies that I think you’ll benefit from. In this episode, we’re going to take a look at test pools.
Specifically, we’re going to look at their benefits, how to use them and some easy ways to develop questions for these test pools. Why am I talking about this topic? Well, at the place I’m working, we’re making a steady push to open education resources. I talked about this in Episode 3. I’ll put a link back to that episode in my show notes. As a result, they- when I say they, the instructors have gotten a little reliant on the publisher’s materials.
The publisher will typically put out test banks and then the faculty will use them to support their instruction. Nothing wrong with that, but if you go to Open Education Resources, you may lose control of those test banks. You won’t have access to them anymore unless you pay additional funds to do that. Basically, they have to go old school and create their own tests again. That’s one of the reasons that we’re doing that.
There’s also other reasons. These are reasons that I find test pools very beneficial because I think they really do help learning if you use them correctly and not just a one-and-done type of exercise, but giving them opportunities to use them as a learning resource where they can actively engage their minds and start diving into those tests or those quizzes and just go over them over and over and over again.
Eventually, they will learn the material just as well as if they’re studying, reading the textbook line for line and trying to memorize it so– Because it’s an active learning tool, it seems to do quite well. I’ll talk a little bit more about why I’ve come to believe that. First of all, some reasons why you should even consider online quizzing, what are some of the benefits?
One of the things is that you can over time build up a huge library, a massive library of questions and keep working on finding the right questions that students will use. With online testing, these can be a multiple choice, true and false, short answer, matching essay, all kinds of things. Students, they get immediate feedback. If you set up everything correctly and provided feedback in the questions, this can be a very powerful tool where they could go get remedial help that you are pointing them to resources that will help them.
I’ve seen a lot of instructors, they just put the basic question in with the answers and that’s it. I think that more can be done by using the feedback feature of online tests. Students like it because they can also take these tests a lot faster, that taking a quiz online is faster than paper-based methods. With this, with this time-saving in grading, the time-saving and taking the test, you can offer more tests.
I’ll talk a little bit about some strategies that you can apply with test pools that I think are really powerful. The fact is, is you could give more quizzes, as you build up your test pool, it makes it easier that you can do this. Students definitely favor the online quizzes more than in-class quizzing. They have more control of their time, when to take the quiz, so they’ll– Maybe they are really great at taking quizzes in the morning. Then they can set up their time to do exactly that, or they can set up their time late in the evening if they wanted to, but they could take it during their optimal time.
By providing online quizzes, they can practice repeatedly. Therefore, they have less stress during the actual exam. If you use a test pool, you can generate quizzes over and over and over again, that students can practice on. That way when they actually take the real quiz or the real exam, they’re a little more comfortable. They should do better on the actual exam.
Students use this method for actually preparing for exams that if you provide the opportunity with a good test pool, that they will go out and repeatedly use these– I can’t say all students, we know that it’s not true, but definitely, there are students that will go out and use these test pools to help improve their results on a final exam. The research is showing that students who took online quizzes yielded a 10% score increase, that they were doing better because of doing online quizzing.
This results into a lower dropout rate. That showed up in the research. There was also a study done with dental students who self-developed exam questions. I’ll certainly talk about that, student develop questions. The results show that the students generating their own assessments developed higher cognitive level exam questions than the instructor generated assessments. They performed as well or better on tests compared to the control group. So, definitely, certain benefits for doing this.
The reasons I would definitely use online quizzing, immediate feedback, that’s definitely a given. The other thing is, you can randomize the quizzes. You can set up if your pool is large enough that you can generate multiple exams that no student would take the same exam. There’s ways to control it, right? You certainly want your subcategories that maybe you’re pulling three from this and three from that. You may have 20 in each one of them, but it allows you to help to measure the objectives that you want to measure.
As I said, with the randomizing, students will tend to practice more until they think they got it. Then they’ll stop practicing. Another benefit for online exams that you don’t find in paper-base is the multimedia aspect, that you can insert different multimedia assets in there. Over the past set of episodes since the beginning of this podcast, I talked about multimedia and the power that these different multimedia assets have, but this is something that you can weave into these online exams or online quizzes to get a lot of benefit from them.
The time and savings for you as an instructor. Now, naturally, there’s upfront time, but there’s upfront time in face-to-face exam, so that’s no different. Once you have this online exam prepared and it’s good to go, then you can deploy it over and over and over again without a lot of overhead to you. It will automatically grade, true and false, even short answer, multiple guests, but you’re certainly going to have to do the longer answers and essays.
That should be a time savings because then you’re just focusing on one set of questions and you should be able to roll through that. When you’re doing some online quizzing, some of the strategies that you want to consider with the program is when you deploy it, deploy it so it only shows one question at a time. The research has shown that if you give multiple questions at one time, it’s not as effective as just one question per screen.
Also, the researchers indicating that you should reserve feedback until really the end of the quiz. Otherwise, that you actually jack up stress and you don’t want to do that. Having this huge test pool, and I’ll talk about this a little further here, helps reduce the possibility of cheating. The larger your test pool is, the more that you can randomize the quizzes and controlling for time decreases the ability to cheat during those exams.
Definitely reasons why I would go for online quizzing, use technology so you can focus your energies otherwise. Why am I a fan of test pools? Well, this actually starts back in my time in the Air Force. In the Air Force, there’s really two things that I want to talk about. One was our qualifying examination. I worked in security forces, for example. For every duty position I had, whether it was a team leader or entry controller, alarm monitor, security controller, what have you, I had to go through an annual qualifying test, so every year I had to sit down with the quality control team and be evaluated.
Part of this evaluation was a written test, an oral test and a practical test. Typically, on a written test, you would get 25 to 50 questions that you would answer, they would grade and you will get your score. Oral question, they would pull a bank of questions out of the test pool and they would ask you those questions. In preparation for this, everybody that worked in security forces was given access to this huge test pool and you were able to basically generate these quizzes based on your qualification, so you could practice, so you could look at, review all the questions.
Now, having worked in training, training was responsible for developing the test bank. Quality control was responsible for administering the test bank. In training, when we built questions, naturally, we were working from objectives, we had criteria and objectives that we were working from, but those criteria and objectives mapped very closely to our operating instructions. We had these rules and regulations, where all this material really came from. We pretty much tapped every single word in those operating instructions as we built exams. We use Bloom’s Taxonomy to build the test questions. We had hundreds to thousands of different questions. Depending on the qualification, there was hundreds to thousands of questions. These were extremely large test pools and we just gave them out. We gave them out to the individuals who were doing the job.
During their downtime, they would work their way through these questions. It was okay if they memorize them, because it was the same thing that they were supposed to learn and memorize is part of their job. It did make a difference. Quality control would take a random sampling from these test banks in order to conduct their assessment. Basically, the goal was to learn the material. These tests banks were extremely useful.
Another case was preparing for our professional fitness examination. As I moved up in rank, I had to take these tests. One test dealt with leadership. The other test dealt with our specialty, our overall job. My job was security forces. In order to prepare for that, we would buy these guides that mapped to the- there were certain guides that we work at certain books.
One was dealt with professional fitness exam, so that book, I forget how many chapters were in it, I think something like, 28, 30 different chapters. We would get this other book that just basically had questions. Those questions mapped to that book. I think one of them had 2,800 questions. Every night, as I was preparing for that test, I would sit down and take hundreds of these questions just to see where my strengths and weaknesses were. Then I go back to my book, identifying the weaknesses I had and highlight and make notes and figure out strategies that I could do this. I basically worked through that 2,800 questions at least three or four times in preparation for my test.
It helped. Every time that I did that with a thorough mission in mind, I got promoted. I became a big fan of test pools. A test pool, what is it? Basically, it’s a large bank of test questions that you can use repeatedly in multiple instances. When you set up an exam in a learning management system like Blackboard or Canvas, you can basically say, “I want to pull from this test pool.” Your test pools can be organized into categories. If you happen to be reliant on a textbook and you’re going through chapters 1 through 15, you can have Test pool 1, focus on Chapter 1 and have all those questions built around Test 1 and then Test 2 and Test 3.
In the end, you’ll have hundreds of questions to thousands of questions prepared that deal with those 15 chapters. You can incrementally expose your students to those questions. Perhaps that you have a random test generator set up for Chapter 1. Students go in there and they could take questions from the first chapter because they haven’t been exposed to the other material, so that’d be unfair. They go through that. Now, when you get to Chapter 2, you can have one random generator that produces questions only from Chapter 2. You can set up an entirely different one that pulls questions from Chapters 1 and 2.
Then when you get to Chapter 3, you could have one focus on Chapter 3 and then another one focus on Chapter 1, 2 and 3. You can continuously do this. Then when you want to do say, a midterm exam, you could have the generator pool from the first eight chapters. You can indicate very specifically, “I want them to meet these criteria,” depending on how you set up your test pool and it will produce a test like that.
One of the benefits, it’ll randomize it so every student gets a different test based on the ability. In most test pools, you can indicate specifically that I want this question. It has to be on all the tests that are created. You have that ability to do that also. Ways that you can use this is you can say, set up extra credit where students who complete a random exam and achieve 100 points or 100%. Then they would get credit as part of extra credit. Maybe you give four extra credit points for Chapter 1.
What you could do is, you could have a random test generator generate one test. They get the first set of points. They get credit for that. Through adaptive release, then you expose the second quiz and then the third quiz and then the fourth quiz, based on the fact that they got 100%. You can also use these tests to test knowledge before they are exposed to content, or before they can get to the next section or chapter that you can use these random quizzes to do that.
Lots of different ways that you could do this. How do you fill this test pool? Well, lots of different ways. One of the first ways that you may want to consider are any old tests that you’ve already created. If you have old tests, that’s a great starting point for questions. Now it’s just a matter of adding to those questions. What you can do is, you can use some previous tests to add to the test pool and then build your own set of another quiz for that chapter and add to it. One of the ways that I think is quite clever and a great way of using or getting your students involved is to actually get them involved and have them create and submit test questions.
You can do this as part of a group activity. You could do this as part of just a solo class activity or out-of-class activity that they generate test questions. You can then present them in class, maybe have them deliver these questions on a PowerPoint deck, and then have students write these questions based on whatever criteria that you want as to whether or not they’re good questions. Now, naturally, your vote always wins, whether or not that could be added. This is a great way for them to review to also think about their instruction at a deeper level that if they were going to assess someone, what questions would they use?
Naturally, one of the things that you want to do is provide some guidance on how to create good question. I’ll include some of those things in the show notes and put those in there. With this, you can take a typical class maybe of 25 students in your class. Each one has to develop five questions on a chapter. That’s 125 questions just for one chapter. If you have three sections, you now have close to 400 questions for your test bank. Great way to just prime the pump and get that going, but you can do this year after year, so literally there’s thousands of questions that they can pull from.
That is kind of down and dirty on test pools. It’s hard to show you exactly how to create one. Most learning management systems have a good set of tutorials. Test pools are typically part of that. You should be able to find them with no problem. Based on the research that I’ve done on this topic, that if you use these test pools to help, for example, learn vocabulary, for a discipline and you give this as extra credit, that students who go through these quizzes will very much pick up the language of that discipline, and more importantly, start using it in other disciplines that it will become part of them and- really just powerful stuff, something that you should consider.
With that, yes, I’m just going to move on. Before I leave you, here’s a quick plug for my book, Read to Succeed.