Engineering

This video looks at how the general assessment principles are applied in GCSE Engineering.  Specifically, it explores the different question types used in the subject, use of language, context and imagery in questions and aspects of mark scheme design.

Transcript

Hello, my name is Brinda and I’m a teacher of GCSE Engineering and an AQA associate and I’ve been asked to speak to you about the rationale behind assessment in our subject.

Hopefully, you'll have already seen our what makes good assessment videos about the principles of assessment, which cover concepts such as validity and reliability and what those terms mean. In this video, I'm going to talk about how those principles apply in engineering.

This presentation will focus upon question types, the use of language, contexts and imagery and the mark scheme, however, before we start, let’s have a quick recap of the structure of our exam which is set to address Sections 1 to 6 from the specification.

The examination paper accounts for 60% of a student’s overall mark for the qualification and starts with several multiple choice questions that allow us to test a student’s breadth of knowledge and the application of their practical engineering skills. As the examination paper progresses, there are a number of short answer questions. These assess in-depth knowledge including calculations and extended response questions that draw together elements of the specification including 8-mark questions which are typically the highest mark tariff on the paper.

Question types.

As GCSE Engineering makes use of a range of question types to fully test a student’s ability and allow them to showcase what they know and can do, it could be helpful to talk through each question type individually.

Here, three types of questions are shown, all of which are used in GCSE Engineering. Do you know where in the question paper they would most appropriately be used? A clue is to look at the marks available and the command words that are being used. These provide indications into the type, and sometimes depth, of response that is required from students. The three question types on display are a multiple choice question which are included at the outset of a question paper, a short answer question which is found after the multiple choice questions and others are throughout the paper and, finally, an extended response question which is usually found towards the end of a question paper.

Let’s talk through each of these, beginning with multiple choice questions. Multiple choice questions are notorious for being difficult to write, but when designed well, they can be used to cover a wide sample of a given topic area in a short amount of time (as in GCSE Engineering) and can validly test recall and application of knowledge.  We believe multiple choice questions are a great way to start the GCSE Engineering examination paper as they can feel more accessible to students, which can build their confidence as they progress through the examination.

On this example, taken from June 2023, it shows the separate parts of a multiple choice question. Firstly, the stem presents the problem, question or an incomplete statement to be completed. The stem is followed by possible and plausible answers, with the correct answer called the key and the incorrect answers called distractors. The stem should be written as succinctly as possible to allow all students to understand what is being asked of them. As with all questions, any unnecessary information in multiple choice questions should be removed and the wording should be unambiguous.

Using Multiple choice questions in the classroom.

When teaching, multiple choice questions are fantastic to use in the classroom, as imagine that in your previous lesson you taught ‘testing methods,’ from section 3.4.2 of the specification and you wanted to quickly check your students’ understanding or recall of this. You could use a multiple choice question, like this one from June 2023, to effectively assess your students’ knowledge and iron out any misconceptions, as well as preparing them for this style of question in the examination. Furthermore, a question like this can also be used as a stimulus for further discussions around this topic and as a quick revision exercise.

A good way to understand what makes a good multiple choice question, is to consider one which requires improvement. With this in mind, please have a look at this multiple choice question, on screen, and consider how it can be improved. Please pause the video here to review the question and start the video again when you have identified issues with it.

Firstly, as it stands, this question doesn’t assess what we want it to as accurately as it might, as the distractors aren’t all plausible as two of them are not types of engineering materials. This means a student can rule out 50% of the distractors straight away, reducing the level of demand and effectively making the question a 50/50 guess for each point.

Secondly, a very strong hint of what the answer might be is provided in the possible responses, as in answer C the answer includes the word ‘polymer’.

Thirdly, another error you may have spotted is that the answers are not in alphabetical order. When writing a multiple choice question, distracters are listed in alphabetical or numerical order to remove any sense of bias or pattern which can unintentionally be written into the question.

A final thing you may have noticed is that this question would be better suited to including a command word. For short answer and extended response questions this would pose more of a problem as students would not have a signpost as to the type (and depth) of response required, however, although in multiple choice questions command words are not strictly necessary, they can, at times, be considered helpful. You might legitimately use ‘which’ or ‘what’ in this example because the format of multiple choice questions already constrains the way in which a student can respond.

So, taking all that into account, this question could be improved to be more like the one shown here, from November 2020.

Use of language

Getting the language in a question correct is difficult. The upcoming slides explain how language can impact on all types of questions in an assessment. This includes a focus on the use of command words, the importance of avoiding ambiguity and negatively written questions and how including key terminology from the specification helps to ensure that questions are as accessible to students as they can be, therefore making the assessment increasingly reliable and valid. I will endeavour now to explain each of these item areas in greater detail.

When considering the use of language, let’s start with command words as these play a key role in signposting to students the type, and sometimes depth, of response that is required for a given question. GCSE Engineering has a list of command words which are available on the AQA website, however some are also provided for you here.

As it’s important that both the student and teacher know what to expect from the assessment, when questions are written they are checked to ensure that only command words from the published list are used. This is to provide consistency within the examination papers and to avoid confusing students by using terminology they are unfamiliar with in a GCSE Engineering context.

Using the same command words on a consistent basis helps us to ensure validity within the assessment, as teachers are able to fully prepare students for the type of question, they will be encountering so students can therefore be fairly assessed on their knowledge and skills and not their ability to interpret the question.

If and when you write your own assessment material, it is essential that the correct command word is used. Students need to be familiar with them and to have the opportunity to practice answering questions that use the common command words in the GCSE Engineering examination paper. If the command word is wrong, it leads to significant problems such as the one outlined here which asks a student to ‘Describe a finish used on metals.’  Pause the video and consider what your answer would be and what other technically correct answers could appear based on the poor wording of this question.

The command word ‘describe’ suggests that detail is required in the answer but how do you describe a finish? A response could be ’paint comes in a tin and can be brightly coloured and quite thick in texture….’  Alternatively, you could describe how a finish is applied to metal, but that isn’t what the question requires given that only one mark is available. A superior manner in asking this question would be to use the command word ‘name’ as through changing this, the student could answer ‘painting, dip coating or electroplating for the one mark. This is much more straightforward, and the student isn’t left guessing whether they have ‘described’ the finish correctly for one mark.

Teachers should ensure that students are familiar with the list of command words linked to the GCSE Engineering specification. Students need to have a clear understanding of what is required based on the common command words, so that they are fully familiar with them.

When you write your own assessment materials to use with your students, it is essential that the correct command word is used. Students need to be familiar with them and to have the opportunity to practice answering questions that use the common command words in the GCSE Engineering examination paper.

Here we have a similar question from the June 2023 paper. In this, the command words is ‘name’ rather than ‘describe,’ thus reducing the ambiguity in the questions and allowing for a clear list of possible answers from the specification.

Good assessment material avoids ambiguity. The use of language is a key area that can be problematic for students. It can be a barrier to success despite their understanding of the subject content. A question needs to be explicit in what it is asking and what knowledge and skills it is aiming to assess. This is particularly true when it comes to writing a multiple choice question. It is important that there can be no other possible right answers. If more than one answer could be correct, then marking that question becomes problematic. Consider this question for example - what is the correct answer?

This question is ambiguous as any of the materials listed in the question could be used, although it seems like the intention is for the answer to be A.

Another example of ambiguity can be seen here.

Marking this question would be very difficult as the question asks how ‘you’ would manufacture the axle. Personalising this question adds ambiguity as we could all manufacture the axle in different ways and still be correct.

An additional issue is that it isn’t clear what size the axle is or what it is made from, or even what the stock form is, all of which would affect the way it would be manufactured. A better way to ask this question might be to include the dimensions on the diagram and ask: ‘use notes and sketches to describe how the steel bar would be turned to the correct dimensions for the axle using a centre lathe.’ Include the names of tools and processes, and safety issues, as seen on screen now.

Writing assessment material requires several checks to be made on the questions and these checks are carried out by numerous people. It is very easy for question writers to become focused on a single element of a question or view it only from the perspective representing the way that they want the question to be seen or read. Having several people review questions means other perspectives are considered and potential problems are caught before any papers are sat by students. Before I explain this point in greater detail, please have a go at answering this multiple choice question:

Re-read the question and check your answer - did you pick A? I imagine many of you did, however some may have fallen into the trap of not reading and comprehending the question fully and therefore focusing upon the words ‘stock form’ and ‘metal’ and assumed as to what the question is asking, without noticing the word ‘not.’ There’d certainly be more chance of this if it were one question amongst many, as when we read questions quickly and under time pressure, we tend to skim read the information.

For a question to be well received by the reader it needs to be easily understood. For this reason, we try to avoid using negatives in the question stem such as ‘not,’ however when we believe there’s no better way than this to ask a question, we will always embolden the word ‘not’, as on screen now.

Key terminology

Another way to avoid confusion and ambiguity is to use phrasing and vocabulary from the specification. Assessment materials tend to use the exact vocabulary from the specification to ensure all students attain fair access to the question. Have a look at the question - can you alter it so that it uses key terminology from the specification? You may choose to pause the video to give yourself time to consider your answer.

This question uses the term ‘margin of error’ which is not included in the specification, therefore students may not know what a margin of error is, making the question invalid. The relevant terminology taken directly from the specification is ‘tolerances’ and therefore using this terminology, as seen in the second box, will allow better access to the question.

The example on the right shows where key terminology has been lifted from the specification and used in the question. This highlights the need for students to be familiar with the wording in the specification as opposed to other words used to describe items that may be part of an individual teacher’s vocabulary. Another example to be aware of, is that although the term ‘thermoforming polymers’ is commonly referred to, the GCSE Engineering specification doesn’t use this term - instead, it uses the term ‘thermoplastics’ so this is the vocabulary we would include in our question papers.

Accessibility

When discussing the use of language in exam papers, accessibility is crucial. The classic example to illustrate accessibility might be the use of the words ‘utilise’ versus ‘use.’ Where there is a more easily understood alternative, we always try to use it. A student is more likely to understand what to do if we say, ‘use the data provided’ rather than ‘utilise the data provided.’

At AQA, we write questions succinctly by removing unnecessary language and limiting the number of ideas expressed. Words that are fundamental to the meaning of the question are sometimes emboldened.

With accessibility in mind, are you able to cut the number of words included in this question, whilst still retaining a clarity within the instructions? You may wish to pause the video to give yourself time to do this.

Perhaps the easiest way to simplify this question is to remove the entire question stem as it isn’t adding anything to the question.  If removed, the question “explain the function of cams in a car engine. ” remains clear and does not contain any superfluous information so is therefore increasingly valid as it assesses a student’s knowledge in engineering rather than their comprehension of a question.

It’s not just language that needs to be considered in GCSE Engineering to ensure assessment is accessible to all. Have a look at the question shown here. Pause the video to consider why might this question be difficult for students to access?

The answer to the question is bearing, with ball-bearing or bearing race as acceptable answers. The image is clearly required to answer the question; however, does it need to be so complex? The question refers to ‘a component,’ singular, so students would be expecting to see just one component. In the image there are variations of the same component. From looking at the image you could also understand why some students might think the answer could be ‘gear.’ In addition, the angle of the image could be changed to make the features of a bearing much clearer.

A simple image of one standard bearing at an angle that shows its features would better support accessibility and test what is intended given the question.

Context and images.

Assessment in GCSE Engineering sometimes makes use of context as a method of testing a student’s ability to apply their knowledge in different situations. Contexts can also make a question more accessible to students, as it gives them a real-world scenario through which to apply their understanding and may act as a cue to help them start their response.

One way of applying a context to a question may be through the use of imagery.  In this example, the image provides a clear indication as to what the system is and what it is for. There are many different forms of keypad secure entry systems and therefore it supports the question stem by providing specific information, so all students focus on the system as shown. This also supports consistency in the assessment. As is the case here, the image needs to be clear and not contain irrelevant information which could lead students down the wrong path.

Sometimes, component or product images are used so that students can visualise the types of materials they are made from or to see the form of the object so that they can decide how it is best manufactured for example. Here, students are not being tested on their ability to identify the actual mechanism that is being used for the sliding gate, as it can’t be seen - the sliding gate is a technical context that allows them to consider and choose a mechanism that would be suitable, which requires application. A rack and pinion or crank and slider may be suitable here and therefore the context and image in this example allows students to show their knowledge and understanding of how an appropriate mechanism operates.

This example shows how a diagram can be added into multiple choice questions. Here the student needs to refer to the image to consider the features of the toy brick. Through doing so, the student should then be able to make a decision about what process would be used to manufacture it. This is another example of where the student needs to apply their knowledge and understanding to a visual context. By reviewing the visual context, in this case the toy brick, the student can identify the key and discount the distracters. In this case, the image is again clearly required to answer the question.

Questions that use images, however, should be used with caution as they can introduce problems into examination questions. Information that is provided but is not required can mystify and mislead students. Students may forfeit time - and marks - by trying to decipher material that is not relevant to answering the question. It is also important not to use contexts that students would be unfamiliar with, because they may feel they don’t have the required knowledge to answer the question, when they actually do, thus leading to invalid assessment. An example in GCSE Engineering could be using engineered components or products that are bespoke or unusual and that any 14- to 16-year-old student is unlikely to have ever seen.

The use of imagery should be carefully considered. Sometimes it is tempting to add an image into the assessment to break up what can be a very text heavy examination paper - however, if they are not chosen correctly, images can both disconcert and distract students. Have a look at this adapted question from a GCSE Engineering examination paper. This question did not originally include a supporting image as it didn’t need it. By adding in an image of a ladder, nothing here is assisting the student to answer the question. In fact, it is likely to mislead them as the image is showing a metallic ladder, when the question response requires a non-metallic material.

This example is similar. Adding a generic image of a buzzer isn’t necessary as it doesn’t help the student to answer the question. The question is asking for the name of a different audible output device other than a buzzer, so an image only adds to confuse, rather than support the reader. The addition of an image to this question and the previous example including the ladder, shows how the use of images can influence and mislead students instead of helping them. If images are to be added into an assessment it must be necessary, helpful, or both.

Other examples where contexts and imagery are used in GCSE Engineering assessment can be seen in the Maths questions. Writing the Maths questions within an Engineering context is an Ofqual requirement so we can’t simply test the Maths in isolation. These questions can sometimes prove more challenging for students as applying their knowledge and understanding of Maths to a real-world engineering context adds to the overall demand of the question. We can however provide scaffolding for students, where we break the task down into smaller steps that are clearly signposted and follow a logical sequence.

This can mean splitting up the information in a question such as displayed here. In this example, the diagram supports the question as it includes visual information about the dimensions or size of an engineered product. Having this shown visually will support students to more readily access the information and understand what is being required from the question.

The mark scheme.

Alongside the examination paper, the mark schemes (MS) are crucial as they contribute to ensuring reliable assessments. A mark scheme is developed alongside the examination paper. Before the final wording of a question is decided, the mark scheme is structured to ensure that the focus of the assessment is rewarded correctly. It’s the same as planning a lesson. Teachers know and understand the objectives of their lesson before they design the specific content and activities as ways to deliver and achieve those objectives.

The wording of the mark scheme is critical and needs to ensure that it will allow students to be credited for their correct responses even if they are not the answers that were originally anticipated. It goes without saying that the mark scheme and examination paper should match up. It is important that it is clear in the question what the marks are being awarded for - a student should not be penalised if a question has not explicitly asked for something, and they should be credited for things it has asked for.

In GCSE Engineering we use both points-based mark schemes, as seen on the screen here, and levels of response mark schemes which we will come to next. Points-based mark schemes are where marks are given for correct points made by a student. Correct answers are clearly defined, and each correct point made directly corresponds to a mark. Points based mark schemes help students immediately understand why their answer achieved, or failed to achieve, marks. It is important to note to your students however, that the points listed on a points-based mark scheme are usually indicative content only, as seen in the example here. Any other valid answer to a question will always be accepted.

The other type of mark scheme we use is a Levels of Response mark scheme. These mark schemes, like the one on screen now, are broken down into levels, each of which has a descriptor. The descriptor for the level shows the average performance for that level. Like for points-based mark schemes, Indicative content is also provided beneath the Level of Response grid as supporting guidance only.

This type of mark scheme is better suited to unconstrained questions where students have more freedom in how they respond. In GCSE Engineering, we typically use them for Assessment Objective 3 extended response questions as it allows Examiners to mark students’ responses holistically and to credit a wider range of responses. When using levels of response mark schemes examiners first identify the appropriate level. This is done on a ‘best fit’ basis. They then decide on the mark within the level. The focus of level of response mark schemes is on assessing the quality of a student’s response rather than assigning a mark per correct point, as with points-based mark schemes.

The level descriptors can be difficult to word in a mark scheme as they need to be detailed enough to guide an Examiner and ensure that all suitable responses (including those that were predicted and those that were not) can be assessed fairly, but also not so detailed that they become unwieldy and difficult to apply. Examiners develop a common understanding of the differences between the levels by reviewing and discussing a range of student responses.

Teachers should use these levels of response grids with their students to show how their work will be assessed. It is useful to share mark schemes with students so they can judge the level of their own responses, meaning they can then understand how to improve their answers and examination technique. Self-assessment remains a highly useful tool within the classroom to prepare students for their GCSE Engineering examination.

I hope you have found the session useful and that it has given you some ideas for creating your own quality assessment materials for use with your students. If you have any further queries, please feel free to contact us at engineering@aqa.org.uk and remember that numerous useful GCSE Engineering resources are available on the website or via Centre Services.

Thank you.

Questions you may want to think about

• How can you use these insights to prepare your learners for exams?
• Do your internal assessments reflect the approach of the exam? To what extent do you want them to?
• What’s the most important or surprising thing that you’ve learned? How might it influence your teaching?