Exam marking: planting seeds to hit the mark

Craig Barton: Hello, and welcome to Inside Exams. I'm Craig Barton, and I have 15 years of experience as a maths teacher. I'm quite confident about preparing my kids for exams, but I’d much less of an idea about how those exams are created.

In this series I got access to one of the countries biggest exam boards, and I’ve been finding out what you’d like to better understand from them.

Katie: Hi, I'm Katie, I've been an English teacher now for 19 years, and it’s just to consider how papers are actually marked, are they marked by the same person? And how we ensure quality control across all questions.

Craig Barton: We heard about the standardisation process earlier on in the series. But there are still questions to be answered about quality control and consistency in marking.

I would suspect technology plays an ever-increasing role. I remember when you used to go and get your car fixed, and it would take the guy a load of time just to work out what the problem was. Now, they can just plug in a laptop and it does the diagnosis for them, straight to the heart of the issue, every time.

So, does tech benefit exam boards in the same way? To save time and make marking more accurate and consistent? If anyone can answer your questions Katie, it's AQAs Head of Marking, Pete Hawcroft.

Okay Pete, so what I want to talk to you today about is, I want you to imagine that I've just sat an exam, maybe a maths exam, English exam. I want to know the journey my paper goes on, and crucially, I want to be sure that as a student my paper is going to be marked as fairly as possible.

So, let’s go right back to square one. I've just finished me paper, what happens, where does it go?

Pete Hawcroft: So, obviously once your exam has finished, one of the first things that’s going to happen is you're exams officer, so someone who has a really critical role at the schools, who look after all the administration of the exams, are going to take those exam papers and using the stationary and the labels that AQA have provided to them.  They're going to package them all up, make sure that their attendance register has been filled in correctly, because it's really key that we understand which candidates were present in the exam, and which candidates were absent.

So, it's really important that the register is accurate. They will parcel the register and the scripts up in, basically it's like a grey envelope sack we provide, they put a label on it and all those packages in England, certainly will be collected by Parcelforce. So, the vast majority of our exams now are marked on the screen, and to get on the screen they need to be sent to what we call a, scanning bureau.

Craig Barton: I tell you what, I'm glad you asked this year. Mind you, I was saying this because we put this out on Twitter, and Miss Radders, who I know well, Hannah, that was a big question she wanted to ask out of everything on the series. How do those papers get on the screen?

So, what is the practicalities of that, Pete?

Pete Hawcroft: So, the papers get on the screen, basically we have a couple of different of marking systems. But in essence, it's the same process. So, they will be delivered, on mass, by Parcelforce, to what we refer to as a scanning bureau. So, some data gets keyed and essentially a very narrow margin down the side will be sliced off, to turn a booklet into individual sheets. Those individual sheets are then scanned and then they will appear in one of our electronic marking platforms. And the scripts have certain information keyed from them into the system, because obviously we’re talking about millions of exam papers.

Craig Barton: Yeah.

Pete Hawcroft: So, it's really important that my exam paper ends up against the right electronic record for my entry.

Craig Barton: Yes.

Pete Hawcroft: The vast majority are marked electronically; we still have got about 300,000 exam papers that will go to examiners home addresses. So, when those exams officers are parcelling up those exam papers and putting them in those envelopes, they’ll be going off to Mr Smith who lives in Birmingham, or a Mrs Johnson who lives in London. That’s a really small percentage of what we do now, the vast majority are marked electronically and go off to these scanning bureau addresses.

We have some qualifications where the candidates are perhaps providing a different kind of content, that may not be conducive for our e-marking technology. So, for example, candidates who are doing oral examinations.

Craig Barton: Ah, yes.

Pete Hawcroft: Or, where they have recorded a video performance, we don’t have a marking system that will enable those kind of responses to be marked on screen. So, it's that kind of stuff that still goes to those examiners at their home addresses.

Craig Barton: Okay, so let’s imagine my paper has been scanned, it's on the system, what’s it's next stop along the journey? Who’s seen it first?

Pete Hawcroft: For the vast majority of candidates, the first person who will see their responses is the examiner who’s going to be marking them. So, a small number of the exam papers will be seen by the senior examiners as part of that standardisation process, and the seeding process which I think we’ll perhaps talk about shortly. And once that candidate’s response has been marked, it will start a journey back to AQA where it will become a mark in our internal processing systems.

Craig Barton: We’ve touched a little bit upon this earlier, but it’s, I think it’s worth re-emphasising, what are some of the benefits of this marking on screen versus the paper-based marking?

For, in terms of examiners and also in terms of the benefits to me as a student, having my paper marked more fairly, I guess.

Pete Hawcroft: Yeah, so there's a few. So, we’ve already touched on the logistics element. As an examiner, because you are in effect drawing down on an electronic pool of scripts. As soon as the live marking date starts and you're cleared to mark, you have content there you can start to mark. So, that makes the delivery of those 8.8 million exam papers, it really helps AQA with getting the examiners up and running.

Also, there's a key element from a quality perspective. Electronic marking allows you to use a quality assurance method, which we refer to as seeding. The candidate’s responses that have been pre-marked by senior associates at meetings before live marking commences. And these are essentially dropped into your marking at intervals, as a bit of a check to see are you applying the marking standard correctly.

Craig Barton: As a marker, you don’t know which is a, one of these sample ones or not?

Pete Hawcroft: No, so you don’t know, you don’t get any warning what the seeded items are going to be. So, you'll be sat at home, you'll be marking merrily, a seeded item will get dropped in, and you have to mark that within an acceptable tolerance. We have a set of business rules that we configure in the background about how much somebody could be away from that predetermined mark, or how many times somebody could be away from those marks before we’d have to take corrective action.

The way we see seeding is that it's not about trying to catch examiners out, it is about trying to ensure that we are marking accurately and consistently and fairly across all those examiners, for all those candidates. And it's a way to assure ourselves that our training and standardisation has worked. It's a way to assure ourselves that we are picking up any, what we would perhaps refer to as errant examiners, so perhaps they're being too generous, or perhaps they're being too severe.

And it's a way for us to try and help those examiners understand, perhaps where they’ve gone wrong and get them back onto the standard. And it is also a mechanism where we may have to, in some scenarios, stop an examiner from marking, which can be really difficult. But that is something that we need to do, because we are committed to getting the candidates the right marks.  And if you think about something like GCSE English Language, in excess of half a million candidates trying to get about 2000 examiners all to mark consistently and fairly to the same standard.

Electronic marking is really powerful for that. So, we have a defined pot of seeds, and all those 2000 examiners will be presented with seeds from that common pot.

Craig Barton: I see, yeah.

Pete Hawcroft: So, electronic marking really helps us to assure better quality of marking for the candidates.

Craig Barton: I want to ask about that tolerance within the seeding process. I can only speak as a maths teacher, but I know, like, I can imagine if it's like a one-mark question. You’ve got to be bang on, like, if you’ve got people who are dishing out zero marks and the seed says it's a one mark, that’s going to get flagged up. But I know from our departmental meetings, when we try and mark, multi-mark maths questions, say a four-mark question, we will be arguing left, right and centre. I think it's two marks, I think it's three marks and so on.

Where does the tolerance come in, like a multi-mark question within maths? Let’s say, it's a four-mark question, do you have to be bang on with the seed before it gets flagged up?

And then how does that transpire for, say a, twenty-mark question within something like English? Where are the tolerances within these seeded questions?

Pete Hawcroft: So, it very much depends on the nature of the question. We work with our lead examiners, we work with our, I guess I’d describe them as our more analytical teams at AQA, to understand how we set those levels.

So, for the scientific and the maths ones, predominantly smaller mark tariffs, we do have mechanisms and, you know, processes in place to set that appropriately. We’re not out to try and trip up the examiners, but we are out, as I said before, to try and ensure that we do have a high quality of marking.

To talk about some of the longer responses and some of the tolerances, and it's interesting. So, a lot of the longer form, what we refer to as longer form answer questions, are marked holistically as a whole script.

Craig Barton: Right.

Pete Hawcroft: English Language as a case in point, is marked as what we would refer to as segmented marking. So, the five questions within that paper are marked separately within a segmented marking system.

But I think probably the point to make is about the longer, more subjective SAs, there is a balance about where you and how you strike the tolerances. Because if you were to set a completely, black and white, you have to match this mark that’s been predetermined. The likelihood in a subjective subject, of all examiners hitting that exact mark, in all reality that’s probably not going to happen.

Craig Barton: Yeah, absolutely.

Pete Hawcroft: So, we kind of refer to it as, you know, there are other acceptable values for a response, than perhaps the seeded item that has been awarded.  And these alternative acceptable values is something that we are moving towards with our latest marking platform. So, moving away from a seeded response, and you could be two marks higher or two marks lower.  It maybe that for a particular response, the alternative acceptable values, it could be you’d accept just two or three marks beneath it.  But you wouldn’t accept a higher mark for that particular item.

Craig Barton: I see. Oh, so it's not a symmetrical tolerance, that’s interesting.

Pete Hawcroft: It's more symmetrical at the moment, and to move away from the segmented marking. So, subjects such as English Literature, some A-level history subjects where you’d expect to see quite long responses from candidates. They are marked as a whole script, so we don’t segment or separate the individual questions that examiners are going to see.  An examiner will mark the whole exam paper.

And in some of those cases now, they will have these alternative acceptable values, and in some cases, they will have still tolerances.

Craig Barton: If we go back to my paper, so let’s imagine I've just sat my GCSE maths exam. Is this one of these segmented ones, are multiple people going to essentially get little bits and bobs of my paper? Multiple examiners, sorry, I should say.

Or is it going to be one person marking my entire, let’s say, paper one?

Pete Hawcroft: So, an examiner will log onto that particular marking software and they will see options to mark question ones, question twos, question threes. And what an examiner will do in this marking method is, they’ll probably log on, they have to qualify when they log on.  So, they have to do a couple of seeded items before they can start to mark.

And they will then start to mark all those question ones.  So, they’ll see question one, question one and question one, but for lots of different candidates. When they’ve done those, they may do some question twos or whilst they start marking question ones, an examiner somewhere else in the country may be starting to mark the question threes.

So, that approach does mean that it is in all likelihood it would be different examiners who have marked different items for individual candidates.

Craig Barton: And do we know with that segmented method, do we know it's more reliable in terms of marking?  I mean, my intuition is, it is, because again, if I'm marking… Well, I just think to how I mark by hand, if I've taken in a load of tests, what I used to do in my early days of teaching is I would mark one script all the way through, then another script all the way through, and another script. But what I did learn fairly quickly, as I, it was far quicker for me and my marking was more accurate if I just did it essentially page by page.

So, I did, opened all these scripts at the same page and I did all the question threes and fours, then the next three and four, next three and four, and so on and so forth.  So, I assume that segmented marking is not only quicker, but it is more accurate. Would that be right?

Pete Hawcroft: Yes, and I think all the research supports that segmented marking, as well as online marking in general, is more accurate. And you're quite right, so, examiners will choose to start to mark an item, will qualify and I think they get in a bit of a rhythm about how they mark it. They get used to applying that section of the mark scheme. So, we do see in our data that some examiners will start with quite a low tariff item, and they may mark their full quota. So, 300 clips, they may sit down one Saturday afternoon and do all those items.

And some items are quicker to mark than others. So, it's more conceivable that you perhaps mark your 300 questions, that may be a two mark or three-mark tariff. You're not going to sit and mark 300 20-mark essays in one go.

The other way that is supports the quality and supports the accuracy of the marking is that where we do have an examiner who perhaps is lenient or severe, the impact of that examiners marking is in effect diluted.

Craig Barton: Of course.

Pete Hawcroft: Because they are only marking one item amongst maybe 20 or 30 items. Where we spot that, that examiner is perhaps too lenient or too severe and they would continue to fail those seeded items, and ultimately, we have to stop them. The other good thing about the quality of marking on electronic marking, is that we can delete those items, we can put them back in the pot and if, in effect, we can get somebody else to mark them.

Craig Barton: We’ve spoken about technology just on a practical level, how it can help get these papers in front of examiners. The questions in front of examiners more easily.

Is there a role for technology to, essentially direct questions towards examiners who’ve got a certain area of expertise? Who maybe better at marking a certain question than other examiners? Does that ever happen in any subjects?

Pete Hawcroft: I think what you're referring to is something that we would call targeted marking.

Craig Barton: Okay.

Pete Hawcroft: We’re not quite there yet. But it is something that we are focusing on. What we can do though, without having targeted marking where we can assign set questions to an examiner. What we can do is prevent examiners from marking certain questions. So, in the segmented marking it may be that an examiner has really struggled to grasp one or more questions within an exam paper.  Perhaps they’ve not passed the seeded items, and we've decided to stop them from marking anymore on some items.

But they can still carry on and mark other items.

Craig Barton: Makes perfect sense. We heard in the standardisation episode about the different hierarchies that are involved in the examination process.  So, I wonder what support is there available? If I'm an examiner myself, marking some scripts or items.

Pete Hawcroft: They have a team leader, so every examiner has a team leader. Every team leader has either an assistant principal examiner, or a principal examiner.  We call them lead examiners now.

So, everybody has someone to support them. The span of control is really key, so a team leader will probably have between five and eight examiners that they support. The odd occasion we may go over that. But they have a manageable number because they are reviewing their quality, and they're also there on the end of an email, at the end of a phone. So, if an examiner is perhaps struggling with the marking, or for a first time examiner, perhaps if they're finding it a bit overwhelming, and they need a bit of advice or a bit of reassurance, we know that our senior examiners, our team leaders play a really key role in supporting examiners, in keeping them committed to the marking process. They do help us.

Where an examiner is maybe thinking of dropping out, they’ll give them some guidance and support. And we know these team leaders will often spend quite a long period of time, perhaps on the telephone or lengthy email exchanges, to try and give these examiners the support.

So, it's really key to know that as an examiner coming to mark, you're not on your own. You’ve got people at AQA, you have your senior examiner, so it may feel like you're sat at home on your laptop marking your 300 papers. But you aren’t alone in support and help is there to get you any support that you need during that process.

Craig Barton: Well, this has been fascinating. I’d never even considered how many, how my exam paper gets in front of the eyes of the examiners, let alone the amount of standardisation and the segmentation and all of that, that goes into ensuring that my exam paper as a student is marked as fairly as it can be.

So, I found that absolutely fascinating. So, thank you so much for your time.

Pete Hawcroft: You're welcome.

Craig Barton:We’ve just heard how much work marking is. Although I'm pretty sure you don’t need telling that, that’s the case.

Workload, so marking, is the single biggest reason teachers leave the profession.  But I've heard of teachers who are finding other great ways of explaining their student’s performance to them, instead of hours of writing endless ticks, crosses and written feedback.

Juliet Park, Director of Languages for a West Yorkshire trust, is one of those teachers.  I'm going to meet her to see if I can nick some of her smart ideas.

Craig Barton:So, Juliet, I'm going to be honest from the start, I'm not a big fan of marking, it does my head in, it takes over my life, I'm sick of it, I want a change.  So, before we talk about what you do now, what did your marking strategy used to look like?

Juliet Park:Well, I came into teaching after industry, and always been a passionate linguist, was very excited about going into teaching until I sat for the marking policy in my first school.  Which shocked me, because I was not used to working on Sunday night for seven hours.

Craig Barton:Yes.

Juliet Park:Big change from industry.  And I just thought, this isn't how I'm going to go about marking, and so I probably was classed as a bit of a maverick in my previous school.  But, for me getting the results made the fact that, well, it was the fact it didn’t matter to my head at the time because she saw the results were coming in.

Craig Barton:What was the previous, what was taking up your time in these many hours you were putting in?  What were you doing?

Juliet Park:Well, I think, you know historically the idea was lots of written feedback in books.  And the presumption that students will spend hours reading it, and try to act on it.  But, obviously after several hours of marking, your handwriting becomes illegible and trying to write what you want to explain to kids in written form is not always very effective.  And obviously over a time I thought, this is ridiculous.  And mine reduced more and more, and I used different techniques which are working really well.

Craig Barton:Well, just before we dig into that, it's really interesting because there was two things that frustrated me about how I used to mark.  The first was the time, and exactly like you, it was Sundays, it was evenings, it was just non-stop.  But also, and I'm interested on your take on this, I don’t mind things taking time if I think they're having a positive impact, but I don’t even think it was.  I was certainly putting more time and effort in than the kids were.  So, those hours, they seem to be just wasted hours, did you find that as well?  That even though you're putting in this time, it wasn’t actually affective?

Juliet Park:Absolutely, absolutely.  I realised children and students weren’t taking it on board, while they claim they’d read it, but then I didn’t see any benefit or improvement from it.  But I didn’t do it for very long anyway, I am not an avid marker.  Or rather, not in written feedback where I do my marking in a very different way, whether it's in the classroom, formatively, quick verbal feedback, or whether it's for homework reasons, or after a mock exam for example.

So, my feedback approach is very different to what it used to be, or what I was asked to do.

Craig Barton:Well, you're going to have to us what it is then.  So, talk me through, you're frustrated with the hours you're putting in, the lack of impact, so, what changes did you make?

Juliet Park:So, the first thing I did was, I decided in the classroom a lot of verbal feedback can be quite useful.  But, equally, students hear it and you can’t pause for five minutes while they just take that on board, you’ve got to keep the class moving.  So, for example, I use a lot of correction coding, which many teachers will use, but I think you have to use it in an effective way.

Craig Barton:Just talk us through what correction coding does.

Juliet Park:So, you're using abbreviations for particular errors, whether it's grammatical, whether it's spelling, whether it's word order, whether it's punctuation.  So, at one time, I, well, looking back in the past I, like many teachers, would’ve written those codes and expected them to just go away and apply them.

But they need some training, they need and element of training.  Now I am massive fan of mini whiteboards, I could not survive as a teacher without them.  For example, if students are doing a translation activity, there’ll be a sentence on the board that students have to write it on the mini whiteboard and then what I will do, I will actually use correction coding verbally, as I'm looking at their answer, before putting up the correct translation on the board.

So, I’ll be going around going, Sophie, right, SP, Daniel AE, Dylan WO, so they have to write the code.  Then I put the answer on the board, and then their job is to apply that code to the problem.  They compare their answer on their mini whiteboard to what’s on the PowerPoint, etcetera.  And they should be able to tell which word has a problem.  So, they're actually learning what those codes mean.

Craig Barton:So, these correction codes, this is fascinating this, because the way I’ve used them in the past is, I've just, I've done my marking and when it gets to the point where I'm about to do this written feedback, that I can’t be bothered doing.  I think, well I know, I'll save myself loads of time, I'll do these correction codes.  So, I'll put things like, DM, daft mistake, or little, you know, my own ones.  And sure, it saves me time, but I'm not convinced that even then when I give it the kids back, it's any more effective than if I’d have hand written it.

So, a big tick for me in terms of saving time.  But I’m not sure it ‘s adding to my students learning.  So, you’ve tweaked this though, haven’t you?  So, it's taking it out of the kids’ book and almost putting the ownership on them to put the effort in.  I mean, would that be right?

Juliet Park:Absolutely, yes.  I think you drip feed maybe the codes, so, I have rejigged the order the codes that appear on a little sheet that they stick in, and we start to drip feed them.  So, with year sevens for example, we might just work on spelling and word order and punctuation.

So, we focus on two or three at the time and they start to apply it.  As time goes on, we might give them some work that has deliberate errors in on the board, and we say okay.  Step two is actually correcting it, step one is you pretending to be the teacher and applying these codes.  So, it's a gradual build up, they need training, the need to understand how to apply it.  And then ultimately what I have found has been highly effective, and I don’t do this on a weekly basis, it's at the end of say, a unit or after a mock, if they needed a sort of deeply marked piece of work, I will actually put those codes on, for example, a piece of written work.  I will number each error, so there's a number, like number one with a circle around it and there might be an SP or an AA.

And then, this is what works very well for me, is I just use the voice memo on my mobile phone, and I will record feedback, verbal feedback.  I can say, in say, two and half, three minutes, what might take me 20 minutes to write.

Craig Barton:Wow, okay.  Let me get my head around this, because this sounds absolutely fascinating.  So, the in-class stuff that we’re doing is a lot of mini whiteboards and you're telling them the codes, and then projecting the model answer on, and they're then correcting it themselves.

Juliet Park:Absolutely.

Craig Barton:But then whenever we come to these, whether it's an extended piece of work, or whatever it is that you're marking at home.  Whereas in the past you may have marked it and put these codes on and so on and so forth.  You're now numbering the errors, and then doing the verbal feedback.

So, talk me through the practicalities of this.  What are you using, is it like an app or?

Juliet Park:Okay, yeah.  There are apps around there, but I found just going into the utilities section of my smartphone and using voice memo is just as easy as anything.  And I tend to do them in batches of six, otherwise you can, you can try more, but I do find it has worked really well for me.  Because what it is really, is that I can say, okay, Dylan, lets have a look at number one.  Think about your grammar here, you know, and I can explain it.  And then, what I can do then is say to them, you need to now act on this feedback.  So, if there aren’t many errors, they’ll just purple pen, they’ll just, you know, correct the errors.  Or if I think there's more than that, I will say, now you need to redraft that piece of work.

Now not only then do they act on that feedback; they have to actually summarise it.  So, I will give a summary at the end and say, okay, if you look back over those correction codes, you can see adjectival endings are your biggest weakness.  So, the next time you do a piece of work, this is what you really need to focus on.

The nice thing is, then they have written it in their own words, they can play back that, you know, those little comments time and time again.  They can keep pausing and recapping anything they didn’t understand.  They’ve got a recorded audio of me next time, say, they do a piece of work.  But they’ve also got something in writing, so, let’s say, six weeks further down the line, they're going to do another piece of work.  I'll say to them, right, look back at the last bit of audio feedback, what were your biggest weaknesses?  Think about that.  Be mindful of where you went wrong last time and try not to let that happen this time, if that makes sense?

Craig Barton:This is, yeah, it certainly does make sense.  So, what's the practicalities for the students?  So, you're giving this written piece of work back, they’ve got these codes, how are they accessing this?

Juliet Park:That’s a good question.  So, this is the bit that takes a bit of time really, I have to make sure I rename each sound file, and then just forward it to my school email, and then forward it onto the student.  I think listening sometimes can help them, rather than trying to read something, and I just think, when you can speak a little bit more informally to students, and explain a point that, for example, if you try and cram and contract your comment into say, a line.

If you're speaking, it might be the equivalent of four lines of writing, but obviously, it's quicker to do.  And I think sometimes you just need to be able to explain it in a bit more of a wordy fashion than just trying to write it.

Craig Barton:Oh absolutely, I think I'm almost sold on this.  I'm imagining, so I do my written feedback and it's, like I say, the kids have handed their work in on Tuesday, and Thursday I give it back.  So, in lesson I give out the books, and then the kids work through, making the corrections.

How does that work with listening?  What are the practicalities of them listening to this?

Juliet Park:That’s a very good point you’ve made, because we don’t actually do the correcting in lessons.  As we know as teachers, we are so short on time and the idea is that some students will have not that much to correct, and others have like, you know, quite a bit.

So, therefore they do it independently.  And I push a lot of independent learning, recapping, revisiting in their own time.  And it takes time I think, to build students up to the point where they do feel confident and independent.  And trust, you know, whether you can trust them to get on with it really.  But we do it, in that they do it in their own time.  I think this is very much the new way of thinking as well, is flooding students with lots of repetition, giving them loads of practice and then starting to do bits of self-assessment and peer assessment.  Not just through mini whiteboards, not just through using correction coding.  But just the activities you use in the classroom, where you're saying, look, you own your learning, you are accountable, you need to become mature and dependent, independent, sorry, mature and independent with how you're dealing with language learning.

And what, three weeks ago we did an assessment.  We were bowled over how amazing those pieces of work were.

Craig Barton:Just give us a bit of perspective, how much time do you reckon this is saving?  Let’s say you had a class of, I don’t know, 25 students.  How long would it take you to do a piece of work in the traditional way versus this?

Juliet Park: Well, if you think, if I've got, say, a class of 25 and each recording, let’s say, is around two minutes.  So, that’s what, 50 minutes of recording.  And I've often sat there on a Saturday thinking, crikey, if I was writing everything I was saying, it would not be two minutes of students.  You're looking at probably a good 10 minutes per student.

And then when you start to do the maths, you can then make a comparison.  But it's not just the saving in time, it's not just the efficiency, it's the sheer effectiveness.

Craig Barton: Exactly, wow, I mean, I've done a quick calculation now, I think you're saving eight, this will be embarrassing if I've got this wrong, eight minutes per student, 25 sheets, 200 minutes, that’s, you know, over three hours you're saving.

Juliet Park: And you wouldn’t probably keep writing, would you?  You know, you’d have to space it out again.

Craig Barton: Oh no, correct, my hand would be aching like mad, yeah, absolutely.

Juliet Park: But I don’t think if it was in written form, even if it was typed up, they would learn from it and act on it in a way they can do verbally.

Craig Barton: For your colleagues, what are they saying about this?  Because I know, and often it's mixed messages that come down from Ofsted, get misinterpreted and so on and so forth.  But there's definitely a feeling that’s been out there over the last few years, that Ofsted and SLT need to see written feedback in books.  So, how does, how do you get around that issue?

Juliet Park: So, it's a good point and it is a point I raised in my last school, and, you know, I said, so, as long as you can see feedback, does it matter who’s done the feedback.  And they said, well, it's more a case of the fact that there is feedback, but also importantly, that they’ve acted on it.  And if you think about it, the main change is not that there isn't feedback there, it's the colour of the pen.

So, at one time there’d be lots of green from me, now there’s lots of purple.  And what I also say to students is, you must at the top write audio feedback.  And then it's absolutely clear to anybody looking at it, that you have actually heard that feedback.  It's not been in written form; you’ve heard it somewhere.

Craig Barton: Yes.  It's interesting isn't it, because it's almost a bit of game playing in that sense, but we’ve got, we’ve all got to do it.  But you're absolutely right, that’s the critical thing, it's what the kids do in response to the feedback.  But that’s the only thing that matters, isn't it, you could write reems and reems and reems of feedback, well who cares if the kids aren’t acting upon it?

If we’ve got teachers listening who think, I really like the sounds of this, I want to give this a go.  What would you say are the critical things that make this approach work?

Juliet Park: So, verbal feedback, I think don’t do it too often, because it can lose its effect, it can probably, in either case if they were having to listen to something every week.  But I mean, as teachers, we don’t deep mark every week.  This to me is a, say, once a half term, once a half term.

I think then, in and amongst that the verbal feedback is the quick verbal feedback in class.  And also encourage self-marking and peer marking, so take the onus off the teacher.  But the audio approach, I would only do once a half term.

Craig Barton: Do you still give actual marks?  And does that come alongside the verbal feedback, or is it separate?

Juliet Park: So, again, it would depend, obviously at certain times of the year you’ve got to obviously have data available. So, if it is a mock, I tend to give it back and ask the students to redraft it.  And then I might actually give a grade on the redrafted piece of work, rather than do it too soon.  Because I want them to act on it first.

Because I was on an embedding formative assessment initiative with Dylan William a couple of years back, and I seem to remember somewhere reading that it was very much, well if you give a mark too soon it can, obviously, kill the confidence of the weaker learners.  And the higher ability might sit back and think, well, I'm near my target, good, I'll sit back and I don’t need to do much more.

So, I tend to get them to do the redrafting, and then I say to them, once you’ve redrafted it, then I'll grade it.

Craig Barton: Just to finish on this, I'm always interested in teachers marking strategies generally. Now, whether you're doing verbal feedback or written feedback, do you have any order that you approach, let’s say, a set of 25 pieces of work.  And the reason I say this is, I have a definite technique. So, I always open up with a good one, because if I go in with a kid who I know has probably produced, let’s say for want of a better expression, a piece of work that’s going to annoy me and put me in a bad mood.  Then I'm in trouble for the next few hours.

But I also, I always save a good one for the end, because I want to end on a high. So, I want to think, at least I've got that one coming. Do you do anything similar to that, how do you choose the order you mark?

Juliet Park: Are you talking about marking and giving a grade? Or are you talking about just giving feedback generally?

Craig Barton: Yeah, just general, just general, yeah.

Juliet Park: It depends what mood I'm in. It depends what class, and it depends if there's a particular student that is not one of my favourites, I might think, he's going to the bottom of the pile.

Craig Barton: Right, okay. But does that not annoy you that it's hanging over you, you think, that one’s coming up?

Juliet Park: Yeah, I think sometimes I probably shuffle the books around a bit, or the pieces of paper, I might think, I can’t even look at that. I'm doing that later today, that will put me in a bad mood.

Craig Barton: It’s the psychology of marking, it's fascinating. Well listen, this has been a game changer for me. I'll have to think hard about how this applies to maths, I think. But then I think, well I write reems of written feedback anyway in maths, so, there's no reason it can’t. But this is certainly something I'm going to consider, because as you say, it's not just a time saver, it’s potentially a lot more effective than what I've been doing anyway.

So, thank you so much for your time today.

Juliet Park: Thank you.

Craig Barton: I actually don’t think it's dramatic to say ideas like Juliet’s could revolutionise our working lives. Maybe verbal feedback isn't the method you’d choose, but I think it's pretty exciting to be freed from the idea that there's only one way to mark.

You can find the resources Juliet mentioned in the podcast show notes, alongside a resource about the myths of marking.

I'll be back in two weeks with the final episode of series two. But in the meantime, make sure you rate, review and subscribe to the podcast.

You can also join the conversation and ask your own question on Twitter, using #InsideExams.

Until next time, goodbye.