Computer Science

This video shows how we assess programming skills in Computer Science (GCSE/AS/A-level) and how to correctly apply the mark schemes. The subtle differences in how mark schemes are applied to handwritten program code (GCSE) and that produced in an IDE (AS/A-level) are also considered.

We use examples of student work to illustrate the application of the mark scheme and also give you the opportunity to write your own mark schemes in response to the answers we provide.

Transcript

Hello, and welcome to this presentation on demystifying assessments for Computer Science. I'm Paul Meakin, the Chief Examiner for A-level Computer Science. And in this video, we're going to take a look at how AQA assess computer programming skills at GCSE, AS and A-level.

Programming is a core skill in Computer Science, and at each of these levels of assessment, we have two question papers at the end of the course, one of which is mainly focused on assessing students’ programming skills.

We will start by looking at the GCSE Computer Science qualification. For GCSE, we chose to assess students primarily in the programming language that they learned to code in, rather than using pseudocode; but opted for a paper-based assessment rather than an on-screen one, as many centers were concerned about the practicalities of running on-screen assessments with large cohorts of students. Assessing programming in a real programming language, using a paper-based exam, brings some complexities with it – such as how to deal with incorrect syntax – which we will discuss later on.

Whilst GCSE programming assessments will primarily focus on the programming language that a student has learned code in, some questions may use pseudocode. If students use pseudocode in a response, they can use any version of pseudocode that they wish to, as long as the meaning and logic flow is clear. When we present pseudocode to students within a question, we will always use the standard form described in our resources so that it is familiar to them.

A question paper and mark scheme go together and are developed together. The mark scheme for a particular question shows what we expect to be the most common correct answer or answers to the question. When assessing programming questions, the mark scheme cannot hope to capture all the possible correct answers we'll see, as programming problems can be tackled in many different ways – although often, some methods of tackling the problem are more common than others, and these methods might appear as examples in a mark scheme.

When marking student responses, we look at their answers and compare them to the mark points in the mark scheme, rather than the example solutions. Any working solution is mark worthy. Once students have taken the exam, and the lead examiner and other senior examiners see the range of answers produced, they can adjust the mark scheme to reflect any common mark-worthy answers that were not anticipated when it was originally written. For programming questions, both the design and programming marks may be added to, and alternative approaches, such as the use of recursion rather than iteration, can be mapped to the existing mark points in examples, so that the examiners understand how to do this in a common way.

Training materials are produced for the examiners who will mark the paper so that they understand how to apply the mark scheme to likely solutions, and how to deal with solutions that are not commonplace. Examiners’ understanding of how the mark scheme should be applied is tested before they're allowed to start marking. The process of training examiners to mark consistently, and modifying the mark scheme where necessary, is known as standardisation.

At GCSE, AS and A-level, marks may be available for the design of code. These marks relate to the selection of appropriate techniques to solve a problem. They are obtainable even if the student has written code that would not work. Design marks like this function at a higher level than programming marks.

The programming marks at AS and A-level are only available for syntactically correct working code. At GCSE, some leniency is applied to this, as students are writing code on paper and don't have the benefit of using an Integrated Desktop Environment, or IDE, to check their syntax and execute their code, which AS and A-level students can do.

For GCSE Paper One, we've produced notes and guidance for teachers on marking programming questions, making explicit the approach that examiners will take when marking handwritten answers to questions. These can be downloaded from our website using the link shown on screen. AS and A-level Computer Science Paper One students answer questions using a computer and an IDE to write their code. After developing their code, students copy and paste it into an electronic answer document. Sometimes, students use screenshots, but copy and pasting works much better, as the text is usually bigger and clearer for an examiner to read. In addition to the languages available for GCSE, AS and A-level students can also use the Java language.

As the guidance indicates, at GCSE, we’re more flexible in terms of ignoring syntax errors when awarding programming marks as, unlike at AS and A-level, students have not had the opportunity to have their syntax checked in an IDE, or to try executing their programs. On mark schemes, mark points are given in programming language-neutral terms. Examples are provided in all the languages offered. The examples cannot possibly cover all the answers seen, but give a flavour of the most commonly seen responses.

The overriding principle for GCSE, AS and A-level papers is that if a student's answer works and answers the question, it should be able to get full marks, even if the answer does not map neatly onto the mark scheme. For example, suppose a question asks for some code to produce the average of all the numbers stored in an array of floating-point numbers called Readings. The mark scheme has three points. Mark Point One is for initialising a total variable to 0; Mark Point Two is for adding each element in the array to the total; and Mark Point Three is for displaying the result of dividing the total by the number of elements in the array, to arrive at the average.

Four fully working solutions, given by students, are shown on screen. The first two solutions should clearly get all three marks. They've answered the question using the method anticipated by the mark scheme, at iterating through the array to calculate a total, then dividing this by the number of elements in the array. The third student has made use of library functions in visual basic to answer the question. Does the third student get one mark, two marks, all three marks?

Obviously, they should get all three marks. Each student has produced a working solution, even if the third is not one that the marks scheme had anticipated.

The fourth student has made use of a library function that pretty much does the whole job for them. Clearly, they too should, and would, get all three marks, even though the solution does not map neatly to the mark scheme.

When writing questions, we try to anticipate the range of methods that students might use to solve a problem. If we wanted to assess a specific skill that a student could use in their response to this question, such as iterating through an array, and did not want students to be able to use the method shown in solutions three and four, then we might preclude such answers. This could be achieved by stating in the question that in their answer, a student should not use particular libraries, or that, in order to get full marks, they must use a particular technique, such as iteration. In this instance, we would likely choose to ask a slightly different question, which was not a standard task that a library function existed for.

Why not try to write your own mark scheme for the question, taken from a past paper? The question asks the student to write a program that inputs two email addresses and check if they are the same, displaying a message and the entered email address if they match, and a different message if they do not match. When writing the mark scheme, you'll need to consider how many of the five marks should be available for design, and how many should be available for programming. You should try to make your points language-neutral, so that they can be applied equally to each of the three available languages.

Mark schemes often incorporate accept, reject and ignore points. Do you need any of these?

Accept points represent points that should be accepted as alternatives to the main points in the mark scheme. For example, an accept point might denote a mark-worthy point outside of the scope of the specification. Ignore points give information about aspects of students’ answers that can be ignored when allocating marks, such as the case of identifiers, minor spelling mistakes in messages, and so on. Reject points identify commonly given answers that may look superficially mark-worthy but are not. Pause this video and have a go at writing a mark scheme, then we will look at our mark scheme for this question.

This was our mark scheme. We awarded two design marks and three programming marks. The design marks were for using meaningful variable names throughout, together with having two variables to store the email address inputs and for using one or more selection constructs. The programming marks were for storing the user input in two variables correctly for the email addresses, having a correct expression to check if the email addresses matched and for outputting the correct messages and email address.

Here are some example solutions produced by students. Solution One, written in C#, is fully working, so received all five marks. Solution Two, written in Python, received four marks. Mark Point E was not awarded, as there is no attempt to output the second message and the email address is not output, either. There are some exercises that you can try out yourself on the screen at the moment. If you want to have a go at them, pause the video now.

We're now going to look at a longer GCSE question from towards the end of a past paper. Questions at the end are typically more complex, either from an algorithmic point of view, or because they make use of data structures. For this question, students needed to write a program that would look through an array of 100 card values to search for a run of five cards, in which each card's value was one higher than the previous card's value. There were separate versions of the question for each language, as it included some program code. On the real question paper, a figure showed an example of a run, but that is not included here.

There are two design marks in our mark scheme for this question, and four programming marks. The design marks are for using an appropriate iteration structure and selection statement – one to go through every card, and the other to compare two cards. The four programming marks are for different aspects of a working solution. Some of the mark points such as Mark Point D, which is for having a correct condition to compare two card values in the array, can be clearly mapped to a single line of code in solutions. Others, such as Mark Point F, which is for setting gameWon to 'True' in the correct place, require a number of lines of code to be awarded. The example solution in C# is fully correct. The labeled mark points to the side of the solution show where each mark was awarded. Mark E is the most complex to award, as numerous elements of the code are required to check if the five cards are in sequence.

On screen now are two other solutions written in Python. One of them is fully correct and the other is partially correct. As an exercise, you could pause the video and try marking them against the mark scheme. The second uses a slightly different solution method. How well does it map to our mark scheme?

We will move on now to look at AS and A-level. Most of the AS and A-level programming questions are about additions and changes to the skeleton program, which is provided to centers at the start of March for AS and at the start of September for A-level in the exam year. The papers also contain one unseen programming question, in Section A at AS, and Section B at A-level. The AS question paper contains a question giving pseudocode, which a student must translate into their selected programming language. Marking such questions is much more straightforward than marking the A-level Section B question, which gives a description of a problem to be solved.

This is an example AS question taken from the 2020 paper. Marks are awarded for successfully writing a program that translates the pseudocode from the question into the programming language used by the student. Marks are also awarded for testing the program and producing evidence that it works correctly. This type of mark isn't available at GCSE, as students are not able to execute their programs in the GCSE exam to test them.

This is our mark scheme for the question. As the question required the student to convert the pseudocode into program code, the mark points are focused on the student using specific programming techniques which correspond to those used in the pseudocode. This kind of mark scheme is relatively easy to write and apply, as it's not possible for a student to produce an unusual solution to the problem. There are no design marks available as the student has not had to do any design for this task. That was already done when the pseudocode for the question was written. As there is no need to declare variables in Python, the first mark point for data types is awarded in Python if appropriate values are initially assigned to the variables.

At A-level, the unseen problems are more complex. Instead of requiring a student to translate pseudocode into programming language code, A-level problems require the student to decide how to solve a previously unseen problem. Recent problems include testing if a number is prime; testing if a word can be made from the letters of another word; finding the mode of a list of numbers; and finding the nth Harshad number. As a student has to decide how to solve the problem, our mark schemes award some design mark points for those questions, as well as programming points. For example, for the Harshad question, there were design mark points for identifying that interdivision is needed when calculating the sum of the digits, and also for identifying that a loop is needed, that repeats a number of times determined by the number entered by the user. Problems usually require some input, possibly with validation, some output, and processing that may involve iteration or selection.

The mark points for programming marks are commonly awarded in two different ways. The first type is for the use of a particular programming technique called construct. This type of programming mark is awarded when the number of effective ways of solving the problem are quite limited. The second type is for the observed behaviour of the program under a range of circumstances. This type of programming mark is used when parts of a problem could be solved in a wide range of ways. More straightforward questions, such as the question from the 2020 AS paper, may only have the first type of programming mark, but more complex questions are likely to have more of the second type of mark.

On screen now is the unseen programming question from the 2020 A-level paper. Students had to write a program to find the most frequently occurring digit in a list of digits. The program then had to display the number of times the digit occurred. If more than one numeric digit had the same frequency and was the most frequently entered digit, then instead of displaying the frequency, the program had to display a special message. You could have to go at writing your own mark scheme for this question. As with the GCSE question, remember to include some design and programming marks, to keep your mark points language-neutral, and to add accept, ignore, and reject points, if they are helpful. If you're going to attempt the exercise, then pause the video now.

These two slides contain example solutions for you to try marking with your own mark scheme, to see how well it can be applied to them. If you've written a mark scheme, then have a go at doing this, while pausing the video. I'm going to move ahead through the slides to look at our mark scheme for this question. Again, pause the video now if you want to try marking this solution using your own mark scheme.

This is our mark scheme for the question. As you can see, there are four design marks for choosing appropriate techniques to use to tackle the problem. For example, Mark Point One is awarded if the student attempted to create a data structure that could store ten different frequencies – one frequency for each numeric digit. The programming marks are only awarded for syntactically correct and working code. Some of the programming marks, such as Mark Point Five, are for quite specific things being achieved, and evidence for them can be easily seen in one or two lines of code. Others, such as Mark Points Eleven and Twelve, are awarded for quite complex behaviour of the program, so cannot easily be attributed to individual lines of code.

In our mark schemes, we also include example solutions in every programming language that we offer, and label them to indicate which parts of the solution result in the award at each mark point. For example, here, Mark Point One, which is a design mark point, is awarded for creating the data structure. Whilst Mark Points Eleven and Twelve are indicated as being awarded near the end of the code, the award of them is contributed to by much of the earlier code as well.

Here you can see a student’s solution to the question. You could pause the video while you try to mark it using our mark scheme.

Despite the solution being superficially plausible, we awarded six marks, as the use of storing the value '99' to mark off numbers which had already been counted caused significant problems.

Sometimes, students produce unusual solutions to problems which may use techniques not covered by the specification, that are unfamiliar to some examiners. In this example, the student has used functional programming techniques, which are examined on Paper Two, but perfectly valid in responses on Paper One. A helper function input and the 'Group By' function have been used in the answer. Examples like this are referred to the Lead Examiner for Marking and may involve running the program. In marking this answer, our overriding principle that working code should achieve marks comes into play, although some of the code maps neatly onto our mark scheme – for example, design marks Two and Four, and programming marks Five, Six, Eleven and Twelve.

This question is the final question from the June 2020 A-level Paper One. It's based on a skeleton program, which was a simulation in which businesses competed against each other to make the most money from running a chain of restaurants. The skeleton program, named Food Magnet, and the associated materials can be downloaded from the AQA website. For the question, students had to write code to improve the route a business’s delivery vehicles took, to reduce the business's costs. The question outlines a greedy algorithm which could be used to do this, and then gives some instructions about which parts of the skeleton program should be modified when a student solves the problem. These instructions help a student to quickly focus on producing the solution.

One of the reasons that we chose to use a skeleton program in our AS and A-level assessments is that it allows us to assess complex programming skills, as students already have the basic code needed to build their solutions on top of.  Whilst a small amount of how the solution should work is specified in Task One and Task Two, how to implement the greedy algorithm is left to the student. The mark scheme has some mark points focused on very specific points included in the question, such as creating a new method with a specific name. But most of the mark points are for students achieving the required behaviour with their solutions, rather than for specific techniques being used. This reflects the fact that with complex problems like this, solutions can work in many different ways. Examiners inspect the code given by students to determine how it will behave when executed.

Thank you for watching and making it to the end. There are a few key points that I hope you can take away from this video. Firstly, unless they are specifically ruled out by a question, students are free to use any programming techniques that they know to answer questions. Secondly, we’ll always mark solutions fairly, whether they map neatly to our mark schemes or not. At GCSE, we are less stringent about requiring correct syntax for programming marks than at AS and A-level. Please read the marking guidance document that I discussed earlier to see how this works. Finally, most questions, even the hardest, contain some accessible marks. So please encourage your students to do whatever they can on every question. Thank you for watching.

Questions you may want to think about

• How can you use these insights to prepare your learners for exams?
• Do your internal assessments reflect the approach of the exam? To what extent do you want them to?
• What’s the most important or surprising thing that you’ve learned? How might it influence your teaching?