The boundaries of electing a new pope

Series one
Episode six | 24 June 2019

In this Inside Exams episode, Craig Barton asks Lesley Meyer, one of AQA’s senior researchers, what factors are considered in setting grade boundaries, and modern foreign language (MFL) teacher Jane Wood explains how her school managed to get their tiering decisions right.

Featured in this podcast

Craig Barton – Maths teacher, podcaster and author

Lesley Meyer – Senior Researcher at AQA

Jane Wood – MFL teacher at Cardinal Hulme Catholic School

Episode resources

Myth-busting grade boundaries

See the common misconceptions about grade boundaries and the truths behind them.

Transcript

Craig Barton: Hello and welcome to Inside Exams. I’m Craig Barton. I’m a maths teacher with 15 years of classroom experience. I can reel off the definitions of indices and numerators, but some elements of exams are still shrouded in mystery.

So, across this series I’m going behind the scenes to find out what goes into each element of exam season. Today I’m tackling a big one; how grade boundaries are awarded. Watching sports like cricket, football and gymnastics, I’ve always noticed the way objective technology and data are combined with subjective human judgement and experience to award victories.

Which side is given more credence in cases of conflict? What criteria are they looking for when they make their judgements? Is a less experienced referee or judge expected to rely more heavily on statistics than their veteran colleague? I wonder if there are parallels with examiners. How heavily do they rely on past exam SATs? Are our students’ grades being decided on the whim of a single individual or is there much more to it than that?

Before I look for answers to my questions, I want to know what else you want to know.

Johnny Mitchell: I’m Johnny Mitchell. I’m a history teacher, and I’m wondering what you can tell me to help me understand why the grade boundaries change every year.

Craig Barton: It’d be really useful to know what factors actually affect grade boundaries each year, and I’m definitely interested to learn how cold, hard stats are combined with human judgement to ensure the fairest grade boundaries are set each year. So I’m going to meet Lesley Meyer, a senior researcher at AQA, to find out how that relationship works.

Lesley, it is fantastic to meet you today. If I’m right in saying, you are a senior researcher who specialises in standards in awarding. Is that right?

Lesley Meyer: Yes that’s correct.

Craig Barton: I’ll tell you why I’m dead excited about speaking to you today, because we’re going to talk about one of the most controversial things an awarding body gets involved in, and that’s grade boundaries. Tell me this, how do you set the grade boundaries?

Lesley Meyer: Well, we sit down with the senior examiners who have written the papers, the chief for the specification and the chair for the specification.

Craig Barton: All the big names are involved.

Lesley Meyer: All the big names are involved. That’s right. And prior to that sitting down, we come up with some grade boundaries that we think are likely to be roughly the right marks for the grade, and we do that by looking at where the grade boundaries were last year, considering what percentage of students have got each mark this year, looking at the distribution of marks –

Craig Barton: Oh so the data, so this is after the kids have taken the paper.

Lesley Meyer: This is after the kids have taken –

Craig Barton: So no decisions are being made whatsoever before the data is in.

Lesley Meyer: No, absolutely no decisions are made before the data is in. So once we get all the data in and all the papers have been marked, we consider the mean mark from last year, the mean mark from this year, how the spread of marks has changed, what the ability of the students is and we –

Craig Barton: Sorry to interrupt, can I just ask on that; where are you getting the sense of the ability of the students from?

Lesley Meyer: Well, if we’re working at A-level then we get a sense of their ability from their GCSE grades that they took a couple of years before. If we’re working at GCSE level then we look at what they got at Key Stage 2.

Craig Barton: Wow, five years ago.

Lesley Meyer: So it’s five years ago, and it’s the strength of the relationship between Key Stage 2 and GCSE, and then GCSE to A-level. It doesn’t matter from the point of view of individual subjects so much, because what we’re looking at is the continuity of that relationship from one year to the next, and as long as that continuity is there, that overall relationship between Key Stage 2 and GCSE doesn’t actually change very much.

Then that means that what we can gauge from the students’ ability will be a really helpful tool in predicting where we think those grade boundaries should be this year. So, we get the mark distribution, we look at where we were last year, and we take everything into account from a statistical perspective to come up with some suggested grade boundaries that we think will be right for this year. Then we get the senior examiners in the room with the chief and the chair, and we select scripts around those proxy grade boundary marks that we’ve come up with beforehand –

Craig Barton: Ah, so on the boundary, are we looking at?

Lesley Meyer: On the boundary and in a five mark range around it, so the mark above, two marks above, the mark below and two marks below, and enough scripts to share between those examiners who sit and scrutinise these scripts, and then they tell us whether they think the mark that we’ve come up with is about right for the grade that we’re talking about. So we do that for every single paper.

Craig Barton: Wow. Now I’ll tell you what Lesley, you were saying things that are music to my ears there, because I’m a mathematician, so I’m hearing mean, I’m hearing spread, I’m hearing sample size. What I’m also hearing is it’s not just the stats. There’s an element of human judgement in there. Is that right?

Lesley Meyer: That’s absolutely right. We could do it statistically a lot of the time, but you have to admit that sometimes statistics are wrong. They can give you the wrong indication. I’m a statistician myself so I know that you can, if you like, get the wrong impression from statistics sometimes.

However, equally it wouldn't be right simply just to rely on examiner judgement, because we’re all human beings and we all make mistakes. So the key thing here is to bring those two really important pieces of information together, the statistical side and the judgmental side, and pool them to make sure that we get the right results for candidates.

Craig Barton: One of the things I love to do on this series, Lesley, is pretend I’m a kind of fly on the wall and get a bit of an exclusive as to what goes on behind the scenes. So, if I was this fly on the wall watching one of these meetings, where all the big names are together, making these decisions about the grade boundaries, what would that look like? I mean what would be on the table? What kind of discussions would be going on?

Lesley Meyer: Ha, right, well, what we would have in the awarding meeting would be the specifications themselves, the question papers and marks schemes, so that all the senior examiners that have gathered together can look at those papers and remind themselves what they’re all about.

So the senior examiners will be sitting round the table, looking at a candidate’s script from this year, get the equivalent from last year, see what sort of standard the student achieved last year, and then compare the two and go through the whole paper comparing like this. So, if you like, they have to imbibe the standard from the archive script and get that into their head for that particular grade, and then translate that into this year, and think, okay, well has this candidate, as a whole, bearing in mind if it’s a different question paper, they’re being asked different things, different context; has this candidate done a bit better or about the same, slightly worse. Do I think this candidate therefore deserves the grade compared to the archive script?

When you think about that, they’re there doing this for every single paper of every subject, they might be there for whole days, two days. A-level history AS takes a week, and then we’ve got A-level to do after that. So they’re there for an awful long time, which makes you probably understand that possibly human judgement would be fallible if we were only relying on that, and that’s why they also need the help that we give them with the statistical evidence as well, so we bring it all together, and it’s a little bit of a helping hand, because there’s an awful lot going on, on this table.

Craig Barton: I’ll tell you what I’m picturing here, like when they elect a new pope, they lock everybody in the room and you’re not allowed out until you’ve made that decision. Is it like this, like the doors are locked, you’re not coming out until we get these grade boundaries. This is absolutely incredible stuff.

Now, a recurring theme throughout my conversations with AQA experts for this podcast series has been the constraints AQA are under from Ofqual or the Department for Education (DfE) for certain things, whether it’s the wording of questions or certain assessment objectives and so on and so forth. I assume you’re under similar constraints when it comes to the proportion of students you can award a certain grade to. How does that come into play in terms of the setting of grade boundaries?

Lesley Meyer: Yeah, Ofqual keeps a very close watch on all of the awarding organisations when we’re awarding, particularly the AS and A-levels and GCSEs, because the key thing here for us all, and definitely from Ofqual’s perspective, is to make sure that we carry forward standards in each subject, from year to year, not just within each awarding organisation, but across awarding organisations.

So, when you pool the results from all the five main JCQ awarding bodies, our overall results in each subject will stay constant every single year, so we’re maintaining those standards. So, what we do throughout the time when we’re setting these grade boundaries, which is doing awarding as we call it, we have weekly telephone conferences between all the JCQ awarding organisations and Ofqual, where we talk about what's happened during the week, if you like, and raise any issues that have come up that we think might be particularly important for all awarding bodies.

So, if we are having a particular problem with setting a grade boundary, it might be across various subjects or it might be in one particular subject, at those teleconferences we will raise it and say “is anyone else having this problem?”, and we talk about it with Ofqual and sort out what we’re going to do. Because it might be, for example, that one awarding organisation is awarding a particular subject, maybe a few days before someone else. Well, in which case when that someone else comes to do the award, they need to start being prepared for it as well.  So we’re all working together on this through the whole process.

Craig Barton: So I’ve been a maths teacher for 15 years now. The first 12 years I was dead happy, because it was A*, A, B, I was following everything. Then it all kicked off and all of a sudden we’ve got numbers now, grade nine, grade eight. Were you annoyed at it?

Because you’ve got your system, you can compare it to other data from previous years, and all of a sudden it’s all up in the air, because I know it was a nightmare for teachers, but it must have been a nightmare for you and your team for setting these grade boundaries. How did you do it in that first run-through, let’s say from the maths GCSE when it had changed from letters to numbers?

Lesley Meyer: What helped was that at least we knew that the aim was to link the standard from the legacy GCSEs to the new ones at particular grades. So where we used to be thinking about the standards, particularly at A, C and F, we are now bringing that forward to new GCSE seven, four and one.

Craig Barton: Did it mean that there was more scope or more need for human judgement with the new numbering grade system because you didn’t have as many statistics to fall back on, or did it play out in roughly the same way?

Lesley Meyer: Throughout the process of transition between legacy and new, when we were setting the new standards, the key thing from Ofqual’s perspective was to maintain the standards, comparable standards between the legacy spec and the new spec. So, from the point of view of standards of the overall subject, we could be very sure that the statistical evidence would be the way to go, because when you think about it, there was so much change in the system.

We’ve got legacy grades going to new grades, we’ve got old papers being completely re-written into new papers, completely different specifications, new marks schemes, and to some extent a completely new candidature, because the teachers were no longer used to teaching the spec.

We didn’t want that first cohort of candidates to suffer, because it was a new specification, by dint of being the first cohort through. That wouldn't have been at all fair.

Craig Barton: Or, I guess the flipside, you don’t want them to benefit unfairly either.

Lesley Meyer: No, absolutely. That’s true. So, to make sure that neither of those things happened, we were very keen on taking forward comparable standards from the legacy spec to the new spec, which had to really be done statistically, because there was so much else mushing around in the system that human judgement wouldn’t have been able to deal with that.

However, what we did need is, for a start, the senior examiners, chiefs and chairs to look at the scripts on those marks that we were coming up with and say “yeah, that feels about right in comparison to where we used to be. I’m looking at what this student has done now at this grade, yeah that feels pretty much right”.

Craig Barton: Final question for you Lesley. So I’ve spoken to other assessment design managers. They tell me they’ve got the hardest job. I’ve spoken to the social media team. They’ve got the hardest job. Or, is the person with the hardest job you and your team, who’s got to set these grade boundaries?

Lesley Meyer: I suppose you could say that the hardest job for all is quite possibly for the teachers, because they’re the ones that have got to get the knowledge into the kids in the first place. We just deal with the results of it.

Craig Barton: You’re endearing yourself to our listeners there Lesley. That is a cracking answer. Lesley, you have taken a subject that I’ve been fascinated about for years and I have learnt loads, so thank you so much for inviting me in today.

Lesley Meyer: Thank you Craig.

Craig Barton: Okay, I feel much more clued up on how exam boards set grade boundaries. But in some subjects, we as teachers have a big role to play in choosing whether our students will sit foundation or higher tier papers. I know I’m not the only one who can struggle with the weight of that decision, especially since the introduction of the new GCSE grades. So I’m going to meet Jane Wood, an MFL teacher at Cardinal Hulme Catholic School, because I’ve heard she gets her tiering decisions spot on.

Jane, first off; thank you so much for inviting us into your school today.

Jane Wood: You’re very welcome.

Craig Barton: Now, I’ve been looking forward to speaking to you for ages, because we went through the new grading system for the maths GCSE, and this threw up loads of questions about tiering decisions, and I think as a school, for some of our kids we got that decision wrong.  So I’m hoping I can learn from you how to make those decisions a bit better in future.

So, my first question is that I want to ask you, Jane, is; before the grading system changed for MFL, how did you make tiering decisions?

Jane Wood: Well we could mix and match, so, often, if we had pupils getting a seven, say in writing, but listening and reading were their weaker subjects, we were able to enter them for foundation. So we were very, very anxious about moving over to the new tiering system, because we no longer had that flexibility and we had to put them in for either foundation or higher. So that was a massive change for us. So we played it safe last year. We’re very lucky at Cardinal Hulme because we teach in sets and they’re set according to their ability, and also all of the lessons are on at the same time, so throughout their GCSE career, we can move them from class to class without it really impacting on anything else.

Generally it’s the sort of top two, three sets who do higher and the lower sets that do foundation, but we’ve just recently had movement from kids in the bottom set coming up to my set one, because they’re going to do higher, and literally we made that decision in February based on their second round of mocks that they did.

Craig Barton: So talk to me about some of the things that would cause you to change the tier the student may sit.

Jane Wood: Their performance in exams. Their resilience is quite important, because in languages, on the higher paper especially, they have to be quite resilient in their listening and reading papers, because it’s in peaks and troughs, whereas in maths it’s very much a gradual climb to the grade nine, whereas it’s not for us.

So, if you don’t have a particularly resilient student, and I’m thinking of my set five last year where there was one student who had a grade seven target, that was never going to happen, because of many factors, but mainly this resilience thing, so she was suddenly faced with a grade nine question, question two, she would just give up and say “I can’t do it”, and so that’s quite a challenge I think for language teachers.

Craig Barton: That’s fascinating that, because it’s a whole different ballgame going into that exam.  It’s almost a test of psychology above and beyond the test of ability.

Jane Wood: So you would think that the overlap questions would be the first questions on the paper, but in fact they’re not, and then the translation that they have to do at the end as well has really difficult elements in that, and so it is really difficult for the students, and it’s hard for teachers, I think, to teach that resilience.

Craig Barton: Now, one of the things we found difficult as a school, well, as a maths department, whenever the new grading system came in, is we had no data to go off, whereas in the past you had years and years and years of mock exams. You had years of data on those, so you could say if a child got this in this mock exam, they’re probably going to go ahead to get this grade, and they’re probably most suited to this tier, but of course the new grading system comes in and all that’s disappeared.

So what did you do from a data perspective? Where are you getting evidence from?

Jane Wood: We spoke to the maths department, but it was really hard, because we knew we couldn't compare grade boundaries in maths to grade boundaries in MFL, because we don’t have whole cohorts of schools who do it.

So the proportion, I suppose, of the grade boundaries are different. You have to have a certain amount grade nine, grade eights, so we kind of looked at what we thought, we thought it’s going to be more challenging, and we came up with our own grade boundaries, which were very high now in hindsight, and we got a very pleasant surprise obviously on results day, because we had under-predicted what the students would get.

And you know that could have gone against us, because we could have demotivated students, and we’ve learnt from it, because we got far too many fives.  Had we known what the grade boundaries were going to be, we would have put more students in for higher.

Craig Barton: It’s better to be that side of it though, isn't it?

Jane Wood: I think for the first time round, yeah.

Craig Barton: And it’s an interesting one, so we did something similar where we invented our worst case scenario grade boundaries, but there is that fine line, isn't there, between being on the safe side but, as you say, not demotivating students.  How do you cope with that?

Jane Wood: Well what we also did, obviously for the listening and reading, because you can’t say to them “this is what you need to be able to get a good grade nine”, in maths you can say “this is a grade nine style question”, but what we did do is, for our writing and speaking papers, we had grade criteria.  So we had a checklist of things that we needed to do to get a grade nine, a grade eight.

We went loosely off what was provided by AQA, so we kind of thought about the old GCSE and the types of language structures and verbs, and we came up with that, so when we marked pupils’ writing work, that was quite motivating for them, because we’d say “well, this is what mark you got according to the marks scheme, according to us, but you had all the elements of those grades in”.

Craig Barton: One thing we got obsessed about and I don’t know if this is true in MFL, these crossover questions or the common questions, the questions that are going to be on foundation and are going to be on higher, and we used that to inform quite a few of our decisions for tier entry.  Was that something that you did, and if so, how?

Jane Wood: Yeah we do use the crossover questions, but for us, we’d look at the holistic picture, because we kind of said as a department, we’ve got most control over the writing and the speaking papers, and if a student on the foundation paper, for example, is smashing a five on all of those papers, well then we know they pretty much need to have a go at high.

Similarly on higher, if they’re getting between fours and fives, I’d rather them do the paper that they’re going to feel they’ve succeeded in than struggle on the higher to get a five, and that was kind of part of our reason for choosing tiers last year as well.

Craig Barton: Were there any particularly difficult decisions you had to make?

Jane Wood: We had a few students perhaps who had joined us later than year seven and who perhaps had done different languages at different schools, and they were high ability students in other subjects, but because they didn’t have the foundations in Spanish, so that was hmm, what do we do, what do we do?

They were taught, obviously, extremely well for the foundation, and then in January time, yeah, January, we moved them up into the higher class, and I think because they had those strong foundations, they did go on to do extremely well, and I think they got a seven and an eight maybe. And also I had a very difficult decision to make, because I had an ab initio French class as well and –

Craig Barton: You’re going to have to tell me what that is.

Jane Wood: So we’d alternate a system in our school whereby we offer both languages, but on alternate years. So last year was a Spanish year for us, but the students can still take a second language. So I had a class of only five and they’d not done it before. Obviously they were really able linguists. They were in top set for Spanish. But my challenge was do I put them in for foundation or do I put them in for higher, because they’ve got targets of sevens, eights and nines, but they’ve never done French before.

Needless to say there was lots of blood, sweat and tears, and there was one girl where I 'umm'-ed and 'ahh'-ed and have I made the right decision, but she still came out with a five –

Craig Barton: You put in for higher?

Jane Wood: Yeah, in for higher, and then the others got sixes, sevens and two nines actually, so they did incredibly well, but I think that was because they had the Spanish as well and so they knew the format of the exam, they knew the exam strategies and techniques so -

Craig Barton: But you must still have had a bit of nerves going there right?

Jane Wood: Well I did, because I had no idea what the grade boundaries were going to be and -

Craig Barton: Did you find the same as we did the first year of the new grading system, I mean I’m always on edge over summer in the build-up to results, I don’t think I slept for the whole of August, because anything could have happened that first year.  Did you have the same kind of nerves?

Jane Wood: Oh absolutely.

Craig Barton: It was horrible weren’t it?

Jane Wood: Yeah, and I remember sending a member of SLT a message on the Thursday morning saying “how did we do?” and he sent me back “ah yeah, languages are looking canny”. That was what he said to me, and so I was like, oh that doesn’t sound too good. And then he sent me through the breakdown of results, and we’d massively, massively under-predicted by 20%, and it was because of our grade boundaries.

If we hadn't played it safe, if we’d had the grade boundaries, but we’ve learnt from that and we’ve 20% more kids doing higher this year than there were last year.

Craig Barton: Let’s go a little bit more controversial. So I’m going to quote something here from Ofqual that I find fascinating, and I’ll give you my opinion on it and I’ll be fascinated to hear yours. So Ofqual say this. “All students should be taught so they learn subjects and skills logically and can develop confidence.  It is better to do a foundation paper from real knowledge and learning than tackle a higher paper just armed with exam tricks”. Now, that sounds fair enough. Where could be the possible argument with that?  But I question sometimes; is that actually better for the students because, as I’ve said, sometimes I’ve taught kids in the past who I know, alright, they’re going to have a worse experience doing a higher tier paper because they can’t access as much of it, but they’ll probably come out with a better grade if they did the foundation paper.

Is that something that you could see in MFL or am I talking a load of rubbish?

Jane Wood: So, in an ideal world, yes that would happen, so yes you would like your student to do that paper, but we’re governed by results and being measured on results, so of course you’re going to put them in for the higher paper, because you get more points for them if they get -

Craig Barton: Absolutely.

Jane Wood: But I totally get it, and to be honest it ties in with, we’ve just become a lead school as part of the new DfE initiative for MFL hubs, and this all ties in with the work that we’re doing at the moment, which is leading down to Key Stage 3 and about improving languages teaching from them, because we’re not equipping students with the right skills to succeed from lower down their education. And it’s been a real eye-opener for me, because it’s made me think, you know what?  If I taught phonics properly from year seven, perhaps their performance would be better in the listening exam. So I think that statement does tie in with the sort of wishes of the MFL hub if you like, because that’s what we want to do ultimately, is equip our students with the skills to be able to tackle those, the knowledge rather, not the skills, to be able to tackle those papers.

So when they get faced with a higher reading paper, because of their knowledge of vocabulary and the fact that we’ve done word families from year seven, they can deal with the difficult words because they can see the links with them.

Craig Barton: The other thing I wanted to talk to you about was; it’s one thing us as teachers being uncertain about grade boundaries with the tier changes and so on, but there’s also the uncertainty that the students face.

And like I remember kids, year 11s of three, four, five, six years ago, they had access to all the grade boundaries so that when they did past papers under exam conditions, they could say with certainty “I got an A in this paper”, “I got a B in this paper”, and the more they did, the more confident they were as to where they were at. But obviously, again, with the tier change and the new grading system, all that’s disappeared now.

So my kids, they were saying “I’ve got 72, what does that mean?” and I was like, well I’ve no idea, it could mean this, it could mean that.  How did you, if you did, stop your kids getting obsessed with grading when you couldn’t say with any degree of confidence what grade they would get at any point?

Jane Wood: It’s a really good question.  As I say, we did the success criteria, but inevitably they want to know.  We developed another system in school.  One of our science teachers is an absolute whizz with an Excel sheet, and he developed this data sheet whereby it worked out the averages of classes, and it came up with pie charts and –

Craig Barton: Oh now we’re talking.

Jane Wood: – grades and stuff, so we could say like, well, these are what our results have always been in the languages department.  Does this tie in?  And you know, this is like generally the percentage of kids who get a nine to a seven, we generally hit that target, is that happening with that set of data?

And so we did it that way and I think that was really helpful to the science department as well certainly in using that Excel spreadsheet. I didn’t understand how to use it.  I just put figures in and it works everything out for me.

Craig Barton: I love a bit of Excel. The answer to many a problem is an Excel spreadsheet.

Jane Wood: But that, because the students do want to know, it’s not good enough just saying “you got 85%”.  It was easier with the writing and the speaking, because we could say “that was worthy of”.

Craig Barton: And just finally from me, if we’ve got any teachers listening who are faced with these decisions, faced with these tiering decisions, is it foundation, is it higher, have you any words or wisdom for them, any advice that you’ve learnt over these last couple of years to pass on to them.

Jane Wood: I think you have to think about what’s right for the student. Ultimately yes, they might have a target grade of an eight, but are they realistically going to come anywhere near that if you put them in for higher, or are they going to leave that exam thinking, I did it and I got a five?

And yeah, they’re three grades below, but you’ve got to think of the bigger picture. It’s compared to the rest of the country, not their Key Stage 2 SATs results, when we work out our data. But it is hard. Speak to the students. Speak to your head of department. Speak to your SLT link. Speak to the other teachers. Are they underperforming in other subjects? Is it a cross-curricular thing?  Because a lot of these students will be feeling immense pressure. If there’s a chance they’re going to get a six...

This is what we did, I suppose, last year. If they were consistently getting sixes, we weren’t going to put them in for a five, and actually we didn’t get any fives off the higher paper, because we played it safe and we put them all in for foundation and they all got fives.  We only got a smattering of threes and we didn’t get anything below that. And we don’t have just the crème de la crème doing it here. We have 65% do the EBacc, so we have strong numbers at GCSE.

Yeah, I think you’ve just got to do that. If they’re just scraping fives on the higher, it’s not worth it.

Craig Barton: Not worth the risk.

Jane Wood: No.

Craig Barton: Well I’ll tell you what. In a world of uncertainty you’ve given me plenty of clarity there.  I just wish I’d spoken to you before we made some of our decisions ourselves. So, thank you so much for your time today Jane.

Jane Wood: You’re welcome Craig. Thank you.

Craig Barton: It’s interesting to hear that. Like exam boards, Jane actually relies on a combination of data and opinion to make the right tiering decisions for her students. Whole year timetabling in a way that makes it easy to switch students between sets seems like a really clever idea to me too.

If you feel as though you’re only just getting started learning about the awarding process, head to the podcast show notes where you’ll find some common misconceptions about grade boundaries myth busted.

Over the course of this series I’ll be talking to more exam writers, markers and pioneering teachers. So if you want to swot up throughout exam season, make you rate, review and subscribe to the podcast.

Until next time, goodbye.