Marking Scheme

Steps to prepare a marking scheme for a marking meeting.

1. Choose the criteria to include

Not all the criteria should be included in a marking scheme. In general, more questions will be checked, and more criteria will be included in the first few assignments. Students are not familiar with the style guide at the beginning the term, so it is important to hand-mark their submissions for style more thoroughly and give sufficient feedback on as many criteria as possible in the first few assignments. However, this does not need to be carried forward, you will notice that in later assignments, there is a larger focus on code complexity in place of formatting components of design recipe. Try to assign certain criteria to appropriate questions. For example, a question with a tricky contract (eg. parametric, functions as values) is better to mark for contract correctness than one with a very simple (e.g. Num -> Num) contract.

Sometimes, there is not much to check for some criteria in a specific assignment. Omit those criteria in that case. As the term goes, you might want to omit some criteria where students have barely made mistakes (for example, purpose/contract format!). What is more, criteria that will take too much time to mark could be removed from a marking scheme since the available TA units are usually limited. You can add a new criterion as needed, as well as adjust criteria coverage based on differences between assignments and introduction of new course material. In Fall 2021, we decided what criteria to include during a weekly handmarking meeting with assignment-leading instructors before the assignment’s release – be sure to raise your concerns with grading certain criteria for assignments! – if you do not think it is possible to grade something in the way the instructors would like, make it known.

2. Choose the best fit for each criterion

Not all the questions in an assignment should be checked. You can spot-check questions for selected criteria. After thinking about what criteria will be included, you probably have ideas about which question will be checked for which criterion. You could talk to other ISAs (ideally ones who are experienced with this) before marking the marking scheme. You could also ask for instructors’ opinions at this step.

In Fall 2021, we asked for opinions after we had written a first draft of the marking scheme. There could be several questions in the assignment appropriate for a criterion (e.g.). You might not want to mark all of them. The suggestion here is from the aspect of TA’s marking experience/complexity. Try to match the criterion with some questions in the order that will avoid scrolling up and down from question to question when marking. For example, if Contract Correctness and Contract Formatting are being marked on an assignment, assign the two criteria to the same questions. In Fall 2021, these were also largely decided during the meeting with the assignment leads.

For any assignments that will be graded for code complexity it is helpful to take a look at some student submissions in order to see the variety of (both good, bad, and especially, common) examples. Using these, you can come up with specific positives and negatives that TAs should look out of in term of code complexity for the specific assignment. For example, in Fall 2021, A07 was students' introduction to mutual recursion, we created a file containing several alternatives and graded students on whether or not they used mutual recursion correctly to solve the question.

3. Make the marking scheme

The format of marking schemes is shown on the bottom of the page. Formatting should be consistent throughout the term unless otherwise indicated. It is a similar case for the wording of marking schemes. Most points in the general notes and in the criteria will not change much. The new marking schemes could and should be directly built on the last ones. Marking schemes look similar, but please do not assume that markers could remember all the points in the marking schemes. Keep the marking schemes as detailed as possible and make sure they can easily find the information in the marking scheme even if they already have marked several assignments. You will want to address some administrative information in marking schemes, as well, for example, the deadline of marking, relevant MarkUs information, etc.

In Fall 2021, we wrote both a General and a Specific Marking Scheme for each assignment using Overleaf. The General Marking Scheme provided guidelines for marking each criteria included on the assignment. It included major (deductable) and ungraded errors for each criteria. These did not reference any one assignment in particular, and thus largely remained the same from week to week. The Specific Marking Scheme was where we specified any details that were relevant only to a particular assignment such as, but not limited to, correct purposes, contract, and data definitions, constants that must be defined, specifications for code complexity, etc. We have included examples of both as well as the templates we used in early assignments. Please note that some points under certain criteria may not apply to assignments until relevant concepts are introduced to students (ex. Whitespace specifications of local definitions).

4. Feedback

In Fall 2021, we asked other ISAs and instructors who were the assignment leads to provide feedback on the marking schemes. For the most part, the marking scheme did not change too much at this step. However, there could be some back-and-forth. Sometimes, instructors may want the marking scheme to be more thorough. You need to make some judgement. If it will cost too much to do so, justify yourself! Otherwise, make changes to the marking scheme accordingly. This is also the time to start setting up rubrics and annotations on Markus.

5. Finalize the marking scheme, rubrics, and annotations. Prepare for the marking meeting.

Review the marking schemes, rubrics, and annotations again – note that the private tests and check test cases for the assignment are subject to change throughout the week, make sure that the rubric matches the correct numbers. Sample marking is also highly recommended before the TA marking meeting. Marking 3-5 submissions before could help identify some flaws or mistakes in the marking scheme or annotations so that you can make changes in time. Once everything is done, the marking scheme is ready to go. Upload the pdf version of the marking scheme for graders and set up criteria threads for each criterion on MS Teams. Don’t forget to schedule the meeting! (This is only applicable for online terms.)

6. Hold the marking meeting

The marking scheme for A01 should be carefully explained to markers because marking for CS 135 is completely new to most of them. Go through every criterion in details. It is a good idea to mark a couple submission in the marking meeting when the marking scheme is complicated, especially for A01. As the markers get more familiar with the marking schemes, you can pick out what is new and important in the marking scheme only. Have some time for Q&A before the meeting ends. Ensure that you record the meeting, as well, for graders who miss the meeting. The ISA who set-up the meeting on Teams should click “End Meeting” instead of “Leave” when it is ending. This will ensure that the ISA will receive an Attendance Report for the meeting. In Fall 2021, we had one ISA guide the TAs through the schemes and another mark submissions afterwards. More information about the marking meeting can be found in the marking meeting page.

7. Answer questions regarding marking

TAs might have questions when they are marking. Check the official marking channel and reply to them in a reasonable time. Please be patient with graders and answer questions respectfully. Sometimes TAs might find some problems with the marking scheme. If you decide to change any details in the current scheme, make sure everyone is informed of the changes. (Hopefully, this will not happen)

Fall 2020 Documentation [OLD]

Not all the criteria should be included in a marking scheme. In general, more questions will be checked, and more criteria will be included in the first few assignments. Students are not familiar with the style guide at the beginning the term, so it is necessary to hand-mark their submissions more thoroughly and give sufficient feedback on as many criteria as possible in the first few assignments. This does not need to be carried forward. Pick some criteria and questions to check in later assignments. Think about if there is any question that could be checked for a certain criterion. Sometimes, there is nothing much to check for some criteria in a specific assignment. Omit those criteria in that case. As the term goes, you might want to omit some criteria where students have barely made mistakes. What is more, criteria that will take too much time to mark could be removed from a marking scheme since the available TA units are usually limited. You can add a new criterion as needed.

Not all the questions in an assignment should be checked. You can spot-check questions for selected criteria. After thinking about what criteria will be included, you probably have ideas about which question will be checked for which criterion. You could talk to other ISAs (ideally they are experienced with this) before marking the marking scheme. You could also ask for instructors? opinions at this step. In Fall 2020, we asked for opinions after we had a draft of the marking scheme. There could be several questions in the assignment appropriate for a criterion (e.g.). You might not want to mark all of them. The suggestion here is from the aspect of TA?s marking experience/complexity. Try to match the criterion with some questions in the order that will avoid scrolling up and down from questions to questions when marking.

The format of a marking scheme is shown in this file. It should be consistent throughout the term unless otherwise indicated. It is a similar case for the wording of marking schemes. Most points in the general notes and in the criteria will not change much. The new marking scheme could and should be directly built on the last ones. Marking schemes look similar, but please do not assume that markers could remember all the points in the marking schemes. Keep the marking schemes as detailed as possible and make sure they can easily find the information in the marking scheme even if they already have marked several assignments. You probably want to address some specific issues in marking schemes. For example, the deadline of marking and common mistakes that markers made in previous marking (e.g. clearly state that (else (some-operator (cond[?]))) is not a code complexity error). Please note that some requirements in the provided table may not apply to the assignments before relevant concepts are introduced. For example, you do not need to specify global and local functions since everything is global before Module 12 is covered. Similarly, points in the ?Note? may be different. Choose the appropriate points and make changes as needed. After making all the changes, adjust the layout of the file so that one section will not be split into two parts by the page break.

ISA FEEDBACK: In Fall 2020, ISAs in the hand-marking team gave feedback to the marking scheme before sending it to instructors. You might want to do so as well. It is the time to start setting up rubrics and annotations on Markus.

INSTRUCTOR FEEDBACK: Mostly the marking scheme will not change at this step. However, there could be some back-and-forth. Sometimes, instructors may want the marking scheme to be more thorough. You need to make some judgement. If it will cost too much to do so, justify yourself! Otherwise, make changes to the marking scheme accordingly.

Marking 3-5 submissions before could help identify some flaws or mistakes in the marking scheme or annotations so that you can make changes in time. Once everything is done, the marking scheme is ready to go. Upload the pdf version of the marking scheme for graders and set up criteria threads for each criterion on MS Teams. Don’t forget to schedule the meeting!

The marking scheme for A01 should be carefully explained to markers because marking for CS 135 is completely new to most of them. Go through every criterion in details. It is a good idea to mark a couple submission in the marking meeting when the marking scheme is complicated, especially A1. As the markers get more familiar with the marking schemes, you can pick out what is new and important in the marking scheme only. Have some time for Q&A before the meeting ends.

TAs might have questions when they are marking. Check the official marking channel and reply to them in a reasonable time. Please be patient with graders and answer questions respectfully. Sometimes TAs might find some problems with the marking scheme. If you decide to change any details in the current scheme, make sure everyone is informed of the changes. (Hopefully, this will not happen)

Marking scheme template

[OLD] A comprehensive marking scheme template from Fall 2020 ( Marking_Scheme_Template.docx) is attached. It includes all the general marking notes, general notes for rubrics, criteria, major errors, and exceptions for each criterion used in all the assignments in Fall 2020. This is also a guide to making assignment marking schemes. It will help you understand how a marking scheme should look like, what criteria should be included and how to select appropriate questions for each criterion that is going to be checked. Notes for ISAs in blue boxes are for you to read to learn about the criteria.

-- Shicong Liu - 2020-12-23

[NEWER] Additionally, comprehensive marking scheme examples as well as templates are attached at the bottom of the page. These were created using LaTeX (every UWaterloo student/staff member can use their UW email for a free account on Overleaf. The basics of LaTeX can be learned within an hour or so using the Overleaf tutorial). A zip file containing important .tex files is also attached. Together, these include all the general marking notes, general notes for rubrics, criteria, major errors, and exceptions for each criterion used in all the assignments in Fall 2021. These is also a guide to making assignment marking schemes. They will help you understand how a marking scheme should look like, what criteria should be included and how to select appropriate questions for each criterion that is going to be checked.

-- Adam Abdulhamid - 2021-12-03

-- Alice Schroeder - 2021-12-14

Comments

• Marking_Scheme_Template.docx: Fall 2020: Marking_Scheme_Template.docx
• Fall2021_General_Marking_Guidelines_Template.pdf: Fall 2021 General Guidelines Template
• Fall2021_Specific_Marking_Scheme_Template.pdf: Fall 2021 Specifc Scheme Template