Craig Barton: Hello. It's me, Craig Barton. I hope you're having a wonderful start to your summer holidays. Now, series 1 of Inside Exams might be over, but before we launch into series 2, I want to share some extra behind the scenes chats with you. You might have guessed from enthusiasm at the time, but talking to Zeek, one of AQA's senior researchers for our multiple choice questions episode, it was quite honestly one of the best days of my life. Now, I hear the same criticisms being levelled at multiple choice questions all the time. So I couldn't let Zeek leave without getting him to bust a few of the biggest MCQ myths.
What I want to do now Zeek, we're going to put multiple choice questions on trial, right. You can be the witness if you like. And I am going to propose big arguments I've heard against multiple choice questions. And I want to know what your response is.
Zeek Sweiry: Sure.
Craig Barton: Right, ok. So the first one that I often hear is they only test factual knowledge. What's your response to that?
Zeek Sweiry: That's not true. It's really important to state that, but it is true to say that there is always a danger that most questions written, unless there's really careful thought about what it is that you're doing when you're writing them, will end up assessing factual knowledge only. It requires more thought to assess higher order skills, in some way the difference is simple. If you take a subject like science, if you have no source material whatsoever, no data, no table, no context, nothing and you just ask a question, it's almost certainly the case that all you can then do is assess factual information and nothing more.
But it's actually quite simple to go above that and assessed, for example, analysis by including novel information, a table, some source material, a context of an experiment and so on. And as soon as you do that then it's eminently possible to start writing questions that are assessing analysis and their understanding of that context and, you know, it might be calculations within the context or other aspects of it. To go higher than that level, higher than analysis, obviously there are some skills that simply cannot be assessed using multiple choice.
We can't assess meaningfully a candidate’s ability to build structured, longer arguments. We can't meaningfully assess creativity, original argument and so on because by definition, we are giving them the answer and they have to find it. So there are limitations, but we can assess... if we take for example, GCSEs and A-levels in the UK, we can probably assess quite a lot in many subjects. We can assess quite large proportions if you like of any specification with just multiple choice.
Craig Barton: OK. The jury is convinced on that one I think, Zeek. Allegation number two, my kids just guess them. They don't do any thinking, they just guess any answer. What do you say to that?
Zeek Sweiry: The evidence suggests that there's almost never complete guessing. Unless there such levels of disinterest that there, you know –
Craig Barton: Yes, that's true.
Zeek Sweiry: – interestingly, with computer-based assessments that can be detected. Because the giveaway is that if less time is spent on a question than it sensibly takes to read it, then you know that the motivation is low, for example, and the candidates are likely to be guessing in those situations. But generally, it's not true. And for anyone that doubts that, the best way of testing it is to put an assessment in front of you, a multiple choice one, that relates to a subject that you know absolutely nothing about. Let's pretend dentistry if you're not a dentist. You will probably find yourself still reading the questions despite the fact that it's very unlikely that you're going to know what the correct answer is, and still trying to determine what the correct answer is from the options in front of you, or at least trying to eliminate one or two of them so that you've got a greater chance of being able to then guess the answer. So in practice, it doesn't work that way.
Craig Barton: Yes, and I think a key point there is, and certainly some of the kids I've taught who don't put any effort in and just guess, they are the same one ones who just leave out a non-multiple choice question anyway. It's not for me the specific nature of the question that lends itself to not putting the effort in, it's more the students if that makes sense.
Zeek Sweiry: Yes. I think that's absolutely right. There is always a some level of issue with the fact but you can guess. Because some marks are going to be picked up, but in the end the assessment has to be designed, particularly if there's a large proportion of multiple choice questions, it's designed so that even for candidates that did guess, in theory, across the test, they are still not going to pick up a grade from doing that. The test has to be designed that way. For example, you're not going to get a 100% multiple choice test with four options per question where if you get 25% as your result, you're going to pick up a grade in that assessment. And if you did, then I would suggest this probably a problem somewhere with that test.
Craig Barton: This next one Zeek, I've got to admit, I do feel this a bit and I'm interested in your take on this. What do you say to people who say it's a hell of a lot of effort for just one mark in an exam to answer one of these multiple choice questions?
Zeek Sweiry: It can be. I think that's absolutely right, and I think that's one of the reasons why we don't see as much multiple choice questioning in GCSEs and A-levels in many subjects. It's because there is consideration given to the amount of work required to answer a question or the amount of knowledge or skill if you like required to answer a question, and so often, it's not appropriate to ask questions where there is lots to do, comparable to for example a constructive response question in the test that's worth three or four marks and then only give it one mark. It also can lead to strategy amongst the students which we don't want to see.
You know, if that was the case that the test work like that, then I would probably advise my students to do all the longer questions first and then come to the multiple choice questions in the end, because actually the amount of time you spend on each one given that they're only worth one mark means that logically you are better off leaving them until the end. That's one of the reasons I think why we use multiple choice in maths, but not necessarily that extensively, is for that reason but we want to assess larger scale problem solving, showing working and so on, and if you ask question that requires multiple steps and then you only give it one mark at the end, then it is in the sense I've just described, it's unfair on the students and it doesn't represent what they've just done in relation to the whole test.
Craig Barton: Yes, I think that's fair. Let me go controversial on you now Zeek. I know that AQA's Maths GCSE papers they have, I think it's about six multiple choice questions per paper, something like that, and I think all papers I've seen start off with at least three or four of these multiple choice questions. What do you say to the argument that actually it's not a good way to settle students into a paper, to confront them immediately with a challenging question and really plausible distractors that really get them kind of panicky a little bit. Is it not better to ease them in with a non-multiple choice question?
Zeek Sweiry: I think that's a very fair question. Maybe the counter argument to what you've just said –
Craig Barton: Yes, go for it.
Zeek Sweiry: – is that you often feel as a student answering multiple choice questions that you've got a good chance of answering... that you've answered it correctly, because one of the answers is there. So where as if it's a constructive response question, and this is all largely psychological admittedly, then it's more likely to be blank. It's very unlikely that you're going to leave it blank because you least have a go and even with some limited understanding, you can often eliminate some of the distractors and feel like you've got a good chance of answering.
Craig Barton: Yes.
Zeek Sweiry: One other thing that's were saying with maths, is that what you'll find with maths, is that what you'll find with the maths questions that will ask, or all awarding bodies really I think we'll ask at GCSE and A-level, is they're not necessarily just a straight calculation type questions, but they're often ones that require you to make comparisons between the options and I think those are quite interesting and they can't be asked at this constructive response, so for example, ordering fractions or saying which one of these fractions is the largest or the smallest, these are things that... they don't work in the same way as other multiple choice questions because you are asked to directly compare one option with the other ones and therefore say which one is the biggest. So in the end, often many of the questions we use in maths are of quite a different style and type. And they're things that you couldn't actually ask as a constructed response question. Whereas in science, or english, most questions that you asked as multiple choice can also be asked as constructed response as well.
Craig Barton: That's interesting. One more for you to try and defend this, what do you say to the argument that curriculum time shouldn't be spent learning a strategy to answer a particular question type, multiple choice questions. It should just be spent on teaching students knowledge. What would you say to that argument?
Zeek Sweiry: I think that is true, I think some strategy a modicum of strategy is always going to be inavoidable. You know, one of the biggest points to make is that it's absolutely crucial with other question types, constructed response questions and so on, that students are made familiar with the mark scheme for the particular assessment and how it works and the sorts of responses that are awarded in constructed response questions, because there are different ways of answering questions and unfortunately they are often... particularly for longer questions where there are command words used that don't have a clear meaning that's easy to define, explain, describe and discuss and so on.
Understanding what's required by looking at past mark schemes and so on is an inherent part, and there is some level of strategy in that. So, based on that, I think the strategy with multiple choices actually in one sense and the teaching behind the question type is probably less of an issue actually, than it is with constructed response questions where you've got to understand exactly how a mark scheme works. And you know as the candidate, one of those answers is right, so other than those keys that we talked about before, which may be taught to some extent here, and are certainly taught in America, they don't necessarily have the same strategy implications if you like, you know, teaching strategy implications that other questions have.
Craig Barton: I have had one of the best times I've ever had here. This is my kind of day this, chatting about multiple choice questions. So thank you so much for your time today, I've loved every second.
Zeek Sweiry: Thank you, I've really enjoyed it too.
Craig Barton: Thanks.
