Communicating With Markers

There are several ways that tutors can communicate with the grad TA, the ones that are going to do the hand marking part for each assignment. The main way that the tutors in Winter 2009 used was by making a marking package for each TA to use.

The marking package (by the end of Winter 2009) had the following parts:

• Model Solution
• Questions
• Extra notes about the marking scheme

If you do not have enough time to make the extra notes, you should still print the model solutions and the questions to each TA.

For the notes you should provide information on almost any parts of the marking scheme that you think might not be fully clear on what determines the mark.

Things that should be done before starting on the notes

• The marking scheme should be final, if not close to final, as your notes will be highly based off from what is written from the marking scheme.
• You should understand what the question is asking and have thought of the different approaches that one can use to solve it. By brainstorming this first, you would have a more clear view on what the TA might receive when marking the assignments, and therefore would make a note more fitting to what the TAs might need. [Reading the model solution only is not good enough].
• ...

Feedback

You should specify what kind of feedback you want from the markers in the very beginning of the notes. This is important because by having feedback from the markers you can know some information that the markers discovered during marking, or stuff that you didn't catch when making and reviewing the marking scheme. Some of the usual feedback that you want from the markers are:

• The time that it took the markers to mark the assigned pile of submissions.
  • This way, you can monitor how much time did each marker spend on the work; to see if the workload it too much for them or not.
• The common mistakes that the students made in the submission.
  • Those can be compiled together and be put on the Postmortems in the course website
• Any typo or unexpected things that is found in the marking scheme
  • You might not be able to find some typos in the marking scheme, so it is important that you are notified of any typo in the end, as there are chances that a typo is important that would render your marking scheme incorrect.
• Any places that the markers might not understand
  • This would make sure that the markers do know what they are doing, and so that the final marked submissions would not be considered to be marked unless the markers fully know what they are doing.

You always can add more it make it more fitting to you and your marking scheme. Also, even if you told the markers to provide feedback, do not expect that they all will provide feedback, and those who provide feedback, they might not provide feedback for all of your questions.

Contract and Purpose

Contract (Scheme only) for functions might seem obvious to you at first, but there are still space for discrepancy, so you should provide a list of correct contracts in the students code. For example, some students might choose to write nat nat -> nat instead of int int -> int, depending on the nature of the assignment, the former can be correct, and therefore your notes should cover both cases, clearly saying that either one will earn the mark for a correct contract.

Just for safety, you should also provide a short description for each function so that the TA will know what to expect to search for when looking for the purpose in the student's codes.

More information on creating CS135 marking schemes is available on the CS135MarkingSchemes page.

Style

You should provide a short description and if possible, should also provide a short example on what is good style.

For Scheme, good style means that there is correct indentation and that really long lines are separated.

For C, there should be indentation (indent every time something in inside a curly bracket), and that there should be consistent style for the curly bracket (therefore, keeping the same style for where the opening and closing curly bracket are every time).

One example for bad C style is (and you can provide this in your notes saying that the following example is not get the style mark):

int main() { // Student should be consistent with one kind of styling
for(...) {
...} //should end in a newline
while (...)
{ ...} }  //should start on the same line as while, and end in a newline
// the ending } should not be grouped with other } in the same line

and hence, something like that should have been represented in the following way:

int main() {
   for(…) {
      ….
   }
   while (…) {
      for (…) { // extra code, to show how nested loops should be treated
         …
      }
      
   }
   …
   return …;
}

Or something that is as clear as that

Tests + Drivers

In the MarkingScheme, it said that there should be more detailed specific tests that the student's code should have. In order for this to happen, the markers themselves must know what cases or under what circumstances that each point will be fulfilled. Therefore, for your notes, you should also provide a short description on when each point is satisfied by the student's tests. You should also comment on things like : (add (add 3 4) 4) [for Scheme] and cons(3, cons(4, NULL)); [for C] would already be two tests to add and cons respectively.

In C, instead of having the tests with the functions (as Scheme), the student's tests would be placed in the driver files that they submitted, so any suggestions for tests that is talked about above applies for drivers also. Yet, one thing that you need to be cautious about is stating that driver will only get points if the student submitted the respective main program files. A note about that should be added for security.

Analysis

In some assignments in CS136, the students are asked to analyze the code that they wrote or was provided in the assignment. They would be asked to analyze for either C or Scheme programming. For Scheme functions, they should use a table that lists all the conditions and the running of each cond statements, in the way that is specified in during the lectures. There should be 6 columns in the table, listing which cond statement it is, how many time the question part is called and the running time for each call to the question, how many times the answer part is called and the running time for each call to the answers, lastly with the total running time.

Example for a Scheme Analysis: Define n as the number of items in the list

Statement	# Q	time Q	# A	time A	total time
1	n + 1	O(1)	1	O(n)	O(n)
2	n	O(1)	n	O(1)	O(n)
Total Time:					O(n)

In contrast, a table is unnecessary in C programs, as C programs functions in a different way and therefore making the table defined for Scheme not suitable for C programs. As of Winter 2009, CS136 had specified the students to just explain their code in a short paragraph, following up to the final running time, though we still gave the freedom for student to modify the table used for Scheme to be used in C for analysis.

In terms of marking the analysis, please bear in mind that it will take a very long time for the markers to read each individual submissions word by word or table by table. Though we still want to give student marks for correctness, you need to decide on how much work you are going to give to the markers. For example, if for an assignment you know that it is very hard to do it in a running time that is specified in the assignment, you should not be too strict what the students has but instead of giving marks if the student did the analysis already. Yet, in the very end, it is your decision on how much work you want your markers to do.

Efficiency

Again, this section is most applicable to CS136 assignments but might might also be used for other courses that tests for student's efficiency.

In CS136, there are some questions (either in C or Scheme) that require that the student to make programs in a efficient manner, and therefore should have marks allocated to mark for how efficient a student's codes are. There are several ways to know how efficient a program is, with the main way having the markers looking at the code and determining how efficient that their code is, which without any guidelines is very hard for the markers to understand. Another way is to have an automarking script that test for the running time for the student's submissions, then plotting it on a graph (the detail of the scripts are talked in BitterSuitePlottingRunningTimes), yet up to now the script is very dependent of successful runs of autotests, which would obviously fail if the student's submissions is buggy enough. This is why by the end of Winter 2009, the CS136 still use a hybrid of the first and second method for most of the problems dealing with efficiency.

In C programs, we had used both plots, BitterSuitePlottingRunningTimes, and guidelines on what to look for within the student's submissions (instead of having them look at their code line-by-line). For Scheme programs, because we have not changed the plotting script enough to include Scheme functions, we just provided guidelines on what to look for in the student's submissions.

In order to understand to know what the markers should be looking for, you should think of any ways that the question can be solved different kinds (both in efficiently and inefficiently) of build in functions and the running time of those functions. Also, you should look at the questions itself on what any hints that the questions gave to the students on what method or data structure that they should be using. Last, knowing what is should be avoided and what should be used, providing and explaining what kind of code (or what build in functions name) they should be looking for, and the consequences to the submission if such code is used or such function is called.

Room for improvement

As commented by both the graduate TAs and IAs, the best most useful part of the marking scheme notes were the examples provided talking about the different kinds of codes that might appear. By having those examples, the TA can better match the code that they are marking to what is provided in the notes. In future, more time should be used to think of more ways that a student can think of writing the codes, making even more different examples of the possible types of solution for each question (and how many points each possible solution would have). While the explanation was useful, the presence of the examples that the TA can use to match the student codes to the example was more useful for them.