London, Paris, Rome and Asmara

Series one
Episode four | 28 May 2019

In this episode of Inside Exams, Craig Barton speaks to AQA’s resident multiple choice questions guru Zeek Sweiry, who reveals the secrets to writing brilliant multiple choice questions, and science teacher Jinny Bell explains why she finds them so useful in understanding what topics students are struggling to grasp.

Featured in this podcast

Craig Barton – Maths teacher, podcaster and author

Zeek Sweiry – Senior Researcher at AQA

Jinny Bell – Science teacher at Guilsborough Academy

Episode resources

• Interactive quiz [PDF]
• Annotated multiple choice questions [PDF]

Transcript

Craig Barton: Hello and welcome to Inside Exams. I’m Craig Barton, I’m a maths teacher with 15 years of classroom experience. I like to think I can capture the attention of a rowdy class and explain long division, but when it comes to exams I have to admit there are some things I still don’t quite understand. And I’m sure you might feel similarly perplexed at times too. So across this series I’m asking what you’re keen to know from Exam Boards. Armed with your questions I’m meeting the people behind the question papers to get some answers.

First though I’ve got a question for you. Outside the classroom, how often do you come across multiple choice questions, a) never, b) occasionally or c) almost every day? I’d argue the correct answer is c), think about it. At the doctors you might be asked to tick the box that best describes how active you are.

Maybe you were having a drink and doing a local pub quiz last night. We’re constantly having to choose between multiple options but what I want to know is in an exam context are multiple choice questions a) a valuable confidence boost, b) only useful for basic recall or c) unnecessarily demanding? I’ll keep my own questions in mind but I also want to know what you want to know.

Caroline Wilkes: I’m Caroline Wilkes, I’ve been teaching for approximately 20 years. As a mathematician one of the things that we’ve had introduced at GCSE level is multiple choice questions, these didn’t really exist for us before on previous syllabuses. One thing we are finding with them is they’re fantastic revision questions because the answers aren’t just random, they will give three misconceptions as well as the correct answer. I’m interested to know how you would come up with a multiple choice question for the purposes of assessment though, you know, are you looking for questions where the other answers are typical misconceptions? Is that something that you’re doing on purpose?

Craig Barton: So is the best way to approach writing a multiple choice question a) to write the question first, b) start with your correct answer or c) come up with distracters first. If you haven’t heard of a distracter before, it might sound a bit Machiavellian, but it’s just the technical term for the incorrect options that every multiple choice question needs. Like Caroline, I honestly have no idea where to begin so I’m going to meet Zeek Sweiry, a senior researcher at AQA, for a little help.

Zeek, absolutely fantastic to meet you. I cannot wait for this conversation because I’m just going to put it out there from the start, I absolutely love a multiple question.

Zeek Sweiry: Excellent.

Craig Barton: And I love, nothing makes me happier than speaking to somebody else who loves them as well but I’m very much aware that there’s some criticisms of multiple choice questions, perhaps some misconceptions, and I want to dive into all of that in this conversation. My first question for you Zeek is there’s a perception that multiple choice questions are easier to answer than non-multiple questions and we’ll dig into that later. But are they easier to write than non-multiple choice questions?

Zeek Sweiry: That’s a good question and I think generally the answer that you’ll find in the research and from test developers is absolutely not, they’re not easier to write. The fundamental difficulty, and I’m sure we’ll talk about this more as we go along, comes down to coming up with good plausible distracters to the questions. And the whole validity of a multiple choice question will ultimately depend on having plausible distracters, those can often be the challenging part of multiple choice questions to write.

The other huge challenge is that it’s often said that multiple choice questions cannot be written to assess higher order skills. They can, but again there’s a challenge in doing that.

Craig Barton: Let’s imagine that together we’re going to write a multiple choice question, I mean nothing would make me happier. Where would the process start for you? Do you start with the question or do you start with the answers?

Zeek Sweiry: Perhaps the way I would go about things and I’ll explain a few reasons why. Would be to start with an idea of what it is that you want to assess. For example, if you’re working with a particular curriculum or specification, what statement within that specification or curriculum it is that you want to assess? But without actually writing the question. The next point is really about thinking about what it is you want to assess and at the same time, simultaneously, whether there are misconceptions or other types of plausible distracters that would go along with the correct answer to that point that you want to assess.

Otherwise the danger is, and probably the most common cause of having invalid questions, is to write the question, then write your answer as one of the four options and then just think “Right, I need to come up with three other alternative options”, that’s, it’s fair to say, the wrong way of writing questions. Because if the distracters become an after thought in that way, they’re almost always susceptible to a number of problems that will generally mean that the question will be invalid and almost redundant within a test.

Craig Barton: So you’re writing the question with the misconceptions in mind and if you leave those distracters to the end, that’s when we run into problems, is that right?

Zeek Swiery: I think that’s absolutely right. It’s not always fair to call them misconceptions.

Craig Barton: Oh okay.

Zeek Sweiry: So in some subjects it’s easier to think of alternative answers as misconceptions. So, for example, in mathematics if I asked a question that required the candidates to add fractions, then there are some obvious misconceptions there about adding the denominators together and so on. In other subjects we might not necessarily think of them as misconceptions. So for example, if this was an English reading assessment then really there are no misconceptions that exist before that text has been created.

Craig Barton: Ah that’s interesting.

Zeek Sweiry: That the test is based on. So ultimately rather than thinking about them as misconceptions, it’s often the case that you’re thinking about what the competing information is. So, for example, if the question was what year was a certain person born? Then if there are other numbers or years within that text, then those become the distracters. They’re not really misconceptions in the same way as those maths examples are.

Craig Barton: That’s really interesting. I’ve only got experience of writing maths multiple choice questions. My bias suggests that maths lends itself particularly well to multiple choice questions but I mean do you agree with that or are they equally suitable to all subjects?

Zeek Sweiry: I think that maths perhaps lends itself well to multiple choice questions in relation to other subjects when your primary purpose for writing is for formative diagnostic purposes. Because, going back to that point about misconceptions, there usually are for any piece of mathematics you want to assess, there usually are common misconceptions which the question writers, the teachers are well aware of, and by including those within questions, there’s a really powerful diagnostic aspect to it. Because you can then start to look across, for example, a class as to what the common misconceptions are, what the things you really want to then teach to ensure that your class really understands the concepts.

I think one of the things is that the biggest advantages of multiple choice questions are that they are more reliable to mark and you can assess more of the curriculum in a shorter space of time if you like. But that doesn’t necessarily work in the same way for mathematics because if the answer is a single number, it is essentially as reliable to mark. If it was open, as if it was a multiple choice question, so you don’t... and it’s the same really with the content coverage point. So you don’t necessarily get the same advantages in a summative assessment where there is no diagnostic purpose to writing questions as multiple choice.

Craig Barton: What are some of the key things that your question writers need to bear in mind when they’re putting together a multiple choice question?

Zeek Sweiry: The aspects of the question are split up into four key areas. The biggest one by far are distracters and as we said before, the validity of the question really rests on the quality of the distracters. And the two key points about the properties of the distracters, they must be plausible and they must be wrong, which actually sounds in many subjects, is more difficult than I’ve just made it sound, just simply to say they must be wrong.

In a subject like, let’s take English reading again, where there is going to be more subjectivity, there is going to be individual responses to the text that’s just been read, then there’s always the danger that two different students could legitimately have read the same bit of text and understood it in a slightly different way. And so what we might think of as an objectively correct answer to a question isn’t necessarily and there may be other ways of interpreting that question.

Craig Barton: Just on distracters Zeek, because obviously this is a massive part of multiple choice questions, so plausibility is in there and is that essentially saying “There’s no point having a redundant answer there”. If you can’t think of three plausible distracters, you can only think of two, well then just have it out of three options, the correct answer and two plausible ones as oppose to just shoehorning in one that’s pointless. Is that the point?

Zeek Sweiry: That touches on a lot of important points and yes, fundamentally it is. So to give you a simpler and, you know, in a sense more trivial example, if I asked you the capital of a country that you may be less familiar with, the capital of say, if I picked Eritrea, as a capital then you, at this point, you don’t know the answer.

Craig Barton: I’m in trouble, yeah, yeah, yeah, absolutely.

Zeek Sweiry: In my opinion, yeah. But I give you three options that are London, Paris and Rome -

Craig Barton: Oh now, okay, yes.

Zeek Sweiry: - now you know the answer.

Craig Barton: Things are looking up.

Zeek Sweiry: So that question has failed to ascertain whether you know the capital of Eritrea. You know the distracters have fundamentally given the answer away to you.

Craig Barton: That’s interesting.

Zeek Sweiry: And even though that’s in a sense a trivial example, it works exactly the same way within most assessments. If that’s easy to discard the distracters for a candidate that doesn’t have the required subject knowledge to answer, than fundamentally it’s not assessing, the question isn’t ‘assessing what we intended it to assess.

Craig Barton: Yes.

Zeek Sweiry: So I haven’t assessed whether you knew the capital of Eritrea in that. The difficulty here is with the tension between trying to make the distracters plausible and wrong and so the danger is that as soon as you make them wrong entirely, you know, entirely and objectively wrong in every sense, then in many subjects there’s a danger that they become easy to treat as implausible and see they’re wrong.

Craig Barton: Yes, yes. I mean you mentioned one of the big advantages here is the reliability in terms of the marking. But it sounds to me there’s a hell of a lot of effort needs to be put in to make sure that we get that advantage whilst not getting the disadvantages of implausibility and so on. Is it worth it Zeek, are we getting enough benefit from the reliability to outweigh all these potential minefields that we could stumble upon to getting that right question?

Zeek Sweiry: I think that’s a good question and different people will give you different answers. So in America, just based on the fact on that they use multiple choice so extensively, they would absolutely argue yes. We have, in this country, we’ve always been slightly multiple choice averse, therefore we might come up with a different answer to that question. I think the best way to approach it, is to ask what the best way of asking a particular question is. Because you do have different advantages and disadvantages than by thinking about the piece of content you want to assess, it’s often possible to start distinguishing between those advantages and disadvantages and thinking about, for that particular question, whether it is going to work better in one format or another. So as an example, if I asked the question, “Why is”, this is science question, “Why is ammonia sometimes added to drinking water”?

Craig Barton: Okay.

Zeek Sweiry: Then that’s a constructed response question and as a constructed response question that might work perfectly well. Then I start to think about whether I can come up with plausible distracters to this question and it’s not easy to do so. And this is actually based on a past question from decades ago that I came across, and the distracters in this question, one of them was to make the water undrinkable and one of them is to make the water green. And so we all know that, you know, those two cannot be sensible responses to that question and therefore it hasn’t worked as a multiple choice question in that case and you might argue that as a constructed response question it would have been a more valid question, a more valid assessment of that candidate’s understanding of that particular point.

Probably the next most important are about the language. There is one not often discussed advantage of multiple choice questions which is that constructed response questions are much prone to ambiguity than multiple choice questions. If I asked you a constructed response question then there may often be more than one way of interpreting it. If I ask that question as a multiple choice question then quite often the very fact that I’ve given you four options and one of those must be correct, has reduced the possible number of interpretations you could apply to that question.

There are though still a number of language issues that do need to be thought of. Most of them are the same issues you would consider with constructed response questions. There are some that need to be considered specifically to multiple choice or they’re more important perhaps within multiple choice. So one that often comes up is the use of negative words like ‘not’. Now for a number of reasons the use of negative words in questions tends to cause problems for test takers. There’s two dimensions I think to this. One of them is that it can add to the language complexity of a sentence and the other one is that it’s quite easy to overlook the word and the danger is that if you overlook the word, then you’re answering the wrong question, the entirely wrong question.

Craig Barton: Yes.

Zeek Sweiry: And so there’s been examples of research where this issue has been trialled and the effect of simply emboldening the word ‘not’ in that question is drastic in terms of the performance of the students, because it’s a short word so you’re more likely to miss it. But also because most questions in tests are written in the positive. So that’s our default, that’s our expectation. So if the question goes against that, then we’re going to miss it.

But the point to stress really in relation to multiple choice with negative words is that I think that the challenge in dealing with negatives in multiple choice is actually greater. And so to give you an example, there’s nothing wrong and sometimes it’s necessary to use not in a very simply constructed sentence, “Which of these is not X?” You know, whatever that happens to be and that sort of thing could be used in science or anything else. The other one point that’s maybe specific to multiple choice when it comes to language is that you’ve often got a choice when you write a multiple choice question about two types of format.

The first one is writing the stem as a question, so it ends in a question mark and then you’ve got your four options or however many is below that. The other method is called sentence completion or stem completion and what happens there is that the stem is the start of a sentence and one of the options correctly completes the stem.

Craig Barton: Got it.

Zeek Sweiry: Now both of those that I think the literature would suggest and in my experience would also say are fine, except for the one danger which is that when the word count if you like for the question is high, then there is a greater danger with the sentence completion one. Imagine you had several lines of text in the stem and then each option was also several lines long, then you’ve suddenly got to hold in your head everything in that stem and then read it against each one of the options to decide which one is correct.

Craig Barton: Wow. I mean you’ve give us loads already. Is there anything else that springs to mind?

Zeek Sweiry: Well the next type of issue is really about cues that give away, so this is the... so we’ve talked about distracters and we’ve talked about language. The third is about unintended cues that give away the answers to the questions. Now in America students are much more familiar with this as an issue. They are taught that if you don’t know the answer to a question, that you really have no idea, then look out for certain cues that might allow you to eliminate some of the distracters and therefore get to the correct answer without actually possessing the required knowledge.

It’s still something at the end of the question writing process, it’s not something that, you know, I’d want to impede peoples’ thinking when they’re writing and even when they get into the heart of the distracters and the content. But at the end it’s a check, are there any cues? Are there any properties of the distracters and the stem that are giving away the answer? And so some of those might be that there is often considerably more detail in the correct option than in the three incorrect options to the question.

Now it’s often the nature of the option makes it require more words but to the student who has been taught this, it’s a giveaway that that’s likely to be the correct answer. You know, those are another set of things to think about but, as I say, I would always look at those right at the end of the process. They shouldn’t impede your thinking while you’re writing.

Craig Barton: I’ll tell you what I’m thinking here Zeek, because as a teacher I’ve used lots of multiple choice questions in my teaching so I think they’re great to provoke discussion between students, even if the student knows that option A is the right answer, well why is B wrong? Why is C wrong? What other one would you have chosen? What are the plausible distracters? I think they’re a great teaching tool. But I worry here Zeek, I’m certainly not putting enough thought into these compared to your question writers. So should I and other teachers be using these or is it too dangerous to use them if we’re not considering all these kind of guidelines?

Zeek Sweiry: I think there are lots of very good reasons to use multiple choice in the classroom. I think the one that you mentioned earlier about using misconceptions is absolutely fundamental. There isn’t always the time to look at, you know, long answers to questions that students might look across lots and lots of responses, you know, potentially a whole test of responses and try and think about where students are going wrong and where their misconceptions lie. That ability, particularly in mathematics, to base your options on misconceptions can tell you so much about students’ understanding of a particular topic that would otherwise take a considerable amount of time to gauge if you are asking the same questions in a constructed response format.

Craig Barton: Well Zeek, I have had one of the best times I’ve ever had here, this is my kind of day this, chatting about multiple choice questions, so thank you so much for your time today, I’ve loved every second.

Zeek Sweiry: Thank you. I’ve really enjoyed it too, thanks.

Craig Barton: It’s clear that exam boards are real fans of multiple choice questions, quite rightly I think. So how can we teach our students to answer them well and what’s the merit in using them throughout the year as a clever learning tool? Jinny Bell is a science teacher at Guilsborough Academy, I’m going to meet her to find out how she likes to use them. So hello Jinny.

Jinny Bell: Hello.

Craig Barton: First off, thank you so much for inviting us into your lovely classroom.

Jinny Bell: Absolutely wonderful to have you.

Craig Barton: And I wonder if we could start by just giving us a bit of context about your school if that’s alright.

Jinny Bell: Absolutely, yeah. So we are an academy and we are based in Northamptonshire. We are an 11 to 18 college so we go from Key Stage 3 right up to A-level.

Craig Barton: Now previously I’ve been speaking to AQA’s multiple choice guru, so he’s absolutely loving multiple choice questions but I want to get to the truth of the matter. So, as a science teacher, what do you think of them?

Jinny Bell: They are a fabulous way of being able to almost test students on a really broad area of the curriculum really. What I quite like about them, I think is that there’s lots of different ways to do the questions so it can, I mean especially for science, you can look at the structures of things and you can look at how well they understand definitions and things like that. On the other side of things, in terms of multiple choice questions, I sometimes think that often being worth just one mark can mean that students almost throw that away a little bit, kind of just treat it as a “Oh it’s a one mark, I’m not really sure”, and honestly you would not believe the amount of students who don’t even answer a multiple choice.

Craig Barton: You’re joking.

Jinny Bell: I’m not.

Craig Barton: Leaving them out completely?

Jinny Bell: Absolutely. “I didn’t know that one Miss”, “Yeah, okay, but it’s a multiple choice”.

Craig Barton: Might as well have a go.

Jinny Bell: Just have a guess at this guys, yeah.

Craig Barton: What’s your students’ view of them, do they like them?

Jinny Bell: I think they like doing them, I think they like the fact that it feels sometimes a fairly high challenge, sometimes a nice low challenge, but it also feels quite low risk. Because they always know that the right answer is there and it’s lovely for them to sit there and be able to say “Miss, I knew it wasn’t that one”, brilliant, so you’re narrowing down your options. And most of the time it’s out of no more than four, sometimes only three, the right answer is there, they can look at the language, the familiar key words, think about what they’ve learnt and try to distinguish which one the correct one is, I definitely think that’s a strength of multiple choice questions.

But then, at the same time, I think sometimes looking at the answers they get a little bit confused between the differences between them. So it’s not always easy for them to choose the right one because they’ve got so many options there, that if, I suppose if it was a one marker where they had to write down the answer, they would sit, think, come up with an answer and they would just have to go for it. Whereas when it’s a multiple choice they often read the answers before they’ve even read the question, you know, they’re just looking for something that looks the most familiar.

They’re looking for something sometimes which always, you know, it really gets me, “Miss, it sounded the most scientific”. Oh okay, that’s all very nice, yeah. So I mean my advice to students is always read the question, work out your answer and if it’s there go for it, gut instinct, put it down, go for it. If it’s not, go and have a rethink, you know, and get your highlighter out, have a little look at the demand word.

Craig Barton: Do you get the sense that students perceive them as easier than the non-multiple choice questions? Because I’ve found this, that kids go in and think oh, it’s almost like a, I’ll take a little mental breather here, answer a couple of multiple choice and then we’re getting into the trickier stuff or do you not find that?

Jinny Bell: I think they do a little bit. I think there’s always the element of “Oh it’s multiple choice, the answer is there, it’s nice and easy, just one mark”, definitely. But on the other side of things sometimes it’s not a straightforward question and they’re really frustrated, “Oh Miss, it’s one mark and it was so hard”. They get to the stage where it’s just almost not even guessing anymore, they’re just kind of, you know, looking for things that look familiar or things like that.

So I don’t think they always necessarily find it easy, I think it can completely depend on the question. Sometimes, especially if it’s a calculation, or if you’re looking at working something out in terms of interpreting the question, looking for an explanation, making the links and sometimes really focusing on what the command word is at the start of it. And sometimes the students will just “Oh, it was so much work to do Miss and it was so much thinking and oh, oh it just wasn’t worth it”. Oh.

Craig Barton: Let’s talk exam strategy.

Jinny Bell: Okay.

Craig Barton: Strategy for answering these questions. So aside from the kind of subject knowledge itself, are there any techniques you teach your students for specifically approaching multiple choice questions?

Jinny Bell: Oh completely. My first thing is definitely formulate the answer before you even read the options.

Craig Barton: And are talking cover up, cover up the options?

Jinny Bell: Yeah, why not if it works for you?

Craig Barton: It’s like not even look at them.

Jinny Bell: Yeah, absolutely.

Craig Barton: Oh.

Jinny Bell: Yeah, definitely. I think by doing that, if you get the correct answer and it’s there, you’ve got that gut instinct, you’re got that confidence and you can say “Yes, I’ve done this”. If, however, you then have an answer that is not an option, it gives you a really good indication of where you may have gone wrong. “Oh, this answer is too high, what have I read wrong? Let me go back to the question”. On the other side of things, sometimes if there are similarities between answers, it’s really key that students look at those similarities to work out the differences before they get there.

You know what, when I was, what 9, 10 years old, the last question on a maths paper was “Which of the following pairs of numbers adds together to make 1?” And they gave four sets of pairs of numbers and they’re all decimals, and I got it wrong. And I got it wrong because I just saw 9 and 1 and went for it. I saw zero point nine and zero point zero one, and I added them together and I got the wrong answer. But what I did notice, as a 9 or 10 year old, is that there was also zero point three five and zero point six five. And I looked at that and thought “Oh, there must be two correct answers”.

Craig Barton: Oh wow.

Jinny Bell: And I was really annoyed that I got that wrong. That has stuck with me. Do you know what, it probably makes me more likely to say to the students, “If you think there’s two right, two that are correct, go back and look at the difference between them, be very clear to yourself what it is you’re reading, try it again, just recap it, do it again”. But if we’re going to back to what we said earlier, this is then worth so much more than one mark and I think this is why students don’t do it. I think that’s probably why I didn’t do it. I mean I was 9 years old, you can forgive me, right?

Craig Barton: I’ll tell you what though, that is a beautiful sounding question, that straightaway, I like that. One I think makes a good multiple choice question is that if I’m your teacher and I see, let’s say that was option A, you’ve answered that, I know the mistake you’ve made.

Jinny Bell: Absolutely, yes.

Craig Barton: I know the misconception and I... whereas if it wasn’t a multiple choice question, I’ve got to go through your working out, I’ve got to go through the other 30 kids in the class, to try and figure out what these misconceptions are.

Jinny Bell: Absolutely.

Craig Barton: But I can identify, if there’s 20 kids in the class who have all gone for A, I know it’s a place value thing, I know it’s –

Jinny Bell: Completely, yeah.

Craig Barton: – so that’s a big strength isn’t it of multiple choice?

Jinny Bell: Absolutely, yeah. And I think this is one of the reasons why used in the classroom by teachers, they can be absolutely brilliant. Realistically I think that’s a great way of looking at it because it says to students, mistakes are common, mistakes are easy to make, you know, let’s spot the common errors, let’s go and practice where people make these mistakes and really eradicate those, like you say, misconceptions. So I think it can be really good for that.

Craig Barton: So we’ve talked about exams, let’s talk about the use of multiple choice questions in class throughout the year. Now would you be a regular user of them outside of this kind of build up to exams? Would you use them throughout the year and, if so, how?

Jinny Bell: Yeah, absolutely, absolutely. They are so good for, and this is my favourite line, low risk, high challenge, just to get kids thinking about their facts, thinking about their knowledge, practising the recall. Because I think practising the recall is something that we probably take for granted, especially when you’ve been teaching a few years. You’re so used to doing the topics, you’re so used to the key words, the knowledge, the facts that there are, that it comes back to us really easily.

For students, it’s probably the first and only time they’ll ever be taught it and they’ll ever learn it, especially the ones that don’t do a lot of revision at home. And another reason that I absolutely love multiple choice is because you really can make them as simple or as difficult as you want. So you can give some students some really simple knowledge recall, like you say, to build confidence. And I think that’s fantastic in lessons, especially in physics, everybody hates physics, especially in subjects that they struggle with.

What I also really like though is that you can build in misconceptions and common errors and if you build in those common errors and get kids to almost, I don’t want to say practice making errors, but I think that’s exactly what it is. And it’s okay to make mistakes and it’s okay to learn from those mistakes so that we look back in the future and go “Oh, I’ve made this mistake before”, like let’s not make it again.

Craig Barton: It’s really interesting because there’s two things there. So the first is what Doug Lemov calls planning for error and it’s what we talked about before whereas if you’re going to go into a lesson and you know you’re going to use this question, and you know the correct answer is D, and you know that answer A, if a child answers that it’s because of this reason. If they answer B, it’s because of this reason. It means you can go in with explanations, resources, help sheets, lined up ready to go for the kid to answer that.

Jinny Bell: Absolutely, yeah.

Craig Barton: As oppose to trying to think on your feet, if it’s a non-multiple choice question, a kid has answered this and I’ve got to think what the hell, where the hell has that answer come from and what am I going to do about it? Whereas I can do all my planning before the lesson, so that’s one thing I like about them. And the second thing is we can use the wrong answers, so even if a child gets a question right, it’s not the end of the questions, so the correct answer was C, so you can then say to a child –

Jinny Bell: Why was it not B?

Craig Barton: – why was not B? Why is the question right to put A in there? What misconception would that reveal?

Jinny Bell: Absolutely, yeah, and really dissect the answers and the thinking behind the design of the question and things like that, yeah, wonderful.

Craig Barton: They’re such a useful thing and even for differentiation I find it useful so I’ll ask a multiple choice question, let’s say the correct answer is C, a group of kids gets C right, a load of kids have got the other answers. So I can say “Right, the kids who have C right, I’ve just got to sort the rest of these kids now, so you lot, what I want you to do is I want you to write me an explanation of where each of those wrong answers came from, why are those wrong answers there? And then I want you to create your own question”.

Jinny Bell: I would then definitely think about building in some answers where they have to show their working out, you know, whether it’s a long answer questions or whether it’s a calculation, just to see if they’re making any errors that maybe I haven’t anticipated yet or maybe the Exam Board hasn’t quite anticipated yet. So I think I’d add a bit of that in as well but, you know, it doesn’t take anything away from the use of multiple choice questions.

Craig Barton: And again, this comes back to classroom culture, that I don’t want the kids to see this as I’m assessing them in the moment, this has to be a learning thing. I need to know what you know and what you don’t know.

Jinny Bell: Oh and so do they, they need to know what they know and don’t. I always say to my students, the most powerful thing you can know is what you don’t know and as soon as –

Craig Barton: Oh I like that, say that again, that’s flipping good, I like that, go for that one again.

Jinny Bell: – thanks. The most powerful thing you can know is what you don’t know and as soon as you realise what you don’t know, oh you can, you know, you can accelerate towards success really can’t you? Because you can fill those gaps as quickly as you can.

Craig Barton: The last thing I want to speak to you about is the process of writing multiple choice questions and is this something that you do, do you write your own and, if so, do you find it a useful thing to do?

Jinny Bell: Absolutely. Recently I wrote multiple choice questions for definitions in physics. This is a recent one.

Craig Barton: Can you give me one live? Can we do one?

Jinny Bell: Oh.

Craig Barton: Now I am horrendous at physics.

Jinny Bell: Let me have a think back.

Craig Barton: So give me a nice easy one maybe to get me in.

Jinny Bell: One of the ones would be something along the lines of define the farad.

Craig Barton: The what? The farad?

Jinny Bell: Farad.

Craig Barton: Jeannie, what are you giving me here, the farad?

Jinny Bell: Yeah, F-A-R-A-D.

Craig Barton: Define the farad. And I’ll tell you now, if this was a non-multiple choice I’m leaving this one out. So go on, give me a couple of options.

Jinny Bell: Okay. It is the unit used to measure capacitance, it is where let’s say one farad is equal to one [coulon] stored per unit potential difference. It is charge over potential difference.

Craig Barton: God.

Jinny Bell: Now technically all three of them are completely inter-related, okay. So the thing about the farad is it is the unit for capacitance.

Craig Barton: Obviously, I mean I was just about to say that until you interrupted.

Jinny Bell: Well obviously. But it wouldn’t be the right answer would it obviously?

Craig Barton: As I was going to go on to say, yeah, correct.

Jinny Bell: Yeah, so the idea behind this question is that when you are defining quantities you should define them with quantities. But if you’re defining a unit, you should define it with other units, if that, you know.

Craig Barton: Okay, yes.

Jinny Bell: So being a unit, the farad, you’d be looking for the answer that involves other units. Whereas the charge stored per potential, oh whatever I said, charge over potential difference, actually has listed the quantities, which is the correct equation for capacitance but doesn’t answer the question. And then the top one actually just says what it is and it isn’t a definition of a unit.

Craig Barton: Again that goes back to what we said before where you’re learning something from the kid’s answers. Let me try and sell you on this dream, right. So this is something we do in our department, tell me if this is the worst idea you’ve ever heard. So picture this, so me and you are here, right. Let’s say we’re teaching and I’m going to get you back here, so none of this physics nonsense, let’s say we’re teaching ratio, a proper topic here, so we’re teaching ratio. So what we would do is I would write a question on ratio and you would write a question on ratio.

Jinny Bell: Okay.

Craig Barton: And on the back of the piece of paper you would write your choice for, let’s say, the four answers of the multiple choice, so one right answer and your choice for the three wrong answers, and I would do the same on the back of my piece of paper.

Jinny Bell: Okay.

Craig Barton: And then we would swap questions and I would look at your question and I would write what I think would be four good answers. And you would look at my question and write what you think would be four, and then we’d compare.

Jinny Bell: Lovely.

Craig Barton: And what’s nice about that is when I look at my answers and then I look at your answers to the same question, sometimes we’ll have the same, so like a common misconception, but more often than not, you’ll have an answer that I won’t have thought of and I think actually I should have included that one. And then we can have a really good conversation about why did you pick that one? Well I’ve seen my kids do this last year.

Jinny Bell: Yes, definitely, yeah.

Craig Barton: So I find like a collaborative way of writing these is actually a really good thing to stimulate discussion within the department between colleagues, does that make sense?

Jinny Bell: Absolutely, yeah. And I think when we look at the fact that our colleagues are teaching different cohorts of kids, you know, different cohorts of kids, sometimes they’re setted and you might find completely different levels of misconception and misunderstanding and just plain wrong ideas that they’ve managed to pick up from somewhere. And when you compare those actually across the board, you’ve got a whole host of things that you can turn round to your students and just sort of say, “Hey, let’s look at why this is wrong”, you know. So yeah, 100% collaborative planning for that would be definitely up my street.

Craig Barton: Well Jinny, I could literally talk to you all night about multiple choice questions but I think they’re going to kick us out of your classroom so we’d best end it there. Jeannie, thank you so much, this has been an absolutely pleasure.

Jinny Bell: Thank you very much.

Craig Barton: Well I feel like I’ve found a kindred spirit in fellow MCQ fan Jeannie. I love this idea that the most powerful thing you can know is what you don’t know. And that’s where multiple choice questions really come into their own in the classroom. We should all be using them, not just for simple recall, but to really probe our students’ understanding. If you’ve been convinced of the merits of MCQs and want to use them more in your classroom, head to the podcast show notes where you’ll find some examples of MCQs that have been annotated to highlight their strengths.

Throughout this series I’ll be seeking out more exam writers, markers and pioneering teachers to ask them all of our burning questions. So if you want to swot up ahead of exam season make sure you rate, review and subscribe to the podcast. Until next time, goodbye.