Using Markus and Handmarking in CS 241

CS 241 uses Markus for handmarking of a few assignments. Markus requires a good deal of setup to get working properly: this guide should walk you through what you need to do.

A lot of the instructions on this page are out of date. See MarkUsPrepareForMarking for simpler instructions.

Initial Markus Setup

1. Log into Markus and click on the "Users" tab.
   
2. Add any instructors, ISCs or IAs for your course as administrators if they aren't already, and any TAs as graders. This is done through the various sub-tabs under the Users tab.
3. To add students, rather than hand-entering every student, you can batch-add them from a CSV classlist file. You can create this with ~cs241/markus/bin/create-markus-classlist.sh which will create a file called ~cs241/markus/classlist.csv.

Setting up an Assignment

1. Navigate to the Assignments tab and click Add Assignment.
2. The short identifier, repository folder and assignment name can all be AX (where X is the assignment number, for example A4 for Assignment 4). Even if you are marking only one problem on the assignment, you should use the name AX rather than AXPY (Assignment X Problem Y) because the public marks script expects the format AX. Note that while you can change these fields later (for example, if you break the assignment and want to create a new one since it's more or less unsalvageable) it's highly unadvised to do so and might cause weird behaviour. Just rename the long name to "Old/Ignore" or something and pick a new short identifier/repository folder for the next one.
3. Set the due date to at least a few hours in the future. This is necessary because Markus seems to be picky about due dates - submissions need to be added before the due date and graded after the due date.
4. Allow web submits (that's how you'll be uploading things), but hide the assignment from students, since they won't be submitting to Markus.
5. Next, navigate to ~cs241/markus and run bin/make-groups.sh: this should automatically generate a file called group.csv which will be necessary for Markus later. Back on the assignment page, navigate to the Groups tab and select "Upload" in the top right: upload the group.csv file you just generated. Note that most Markus uploads will take quite some time.

Uploading Student Submissions

This guide assumes that you are only marking one problem for each assignment. Previous versions of the course had a system for marking multiple problems but the scripts for this seem to have been moved or removed. Look at revision r5 of this page for information about these scripts, in case you need to bring them back in a future version of the course. The scripts are probably still in the archives somewhere.

1. Navigate to the handin directory on the course account.
2. Download the best on-time submissions for the assignment problem from Marmoset with MarmSql: ~/bin/marm_sql -v -d AXPY (for Assignment X Problem Y).
   If you already have an AXPY folder for Moss purposes, rename it to something like AXPY_moss before running this script.
3. The MarmSql script will create a folder called AXPY. If you want you can rename it to something like AXPY_handmarking.
4. Run ~cs241/markus/bin/upload-to-markus.sh, where the first argument is the name of your folder in ~cs241/handin (for example, "AXPY" or "AXPY_handmarking" if you renamed it) and the second argument is the name of the repository folder for this assignment on Markus ("AX" where X is the assignment number, if you followed the recommendations above). Note that this is more or less irreversible. This will take a while.
5. Once all submissions appear on the submissions page (click "all" to view more than 30), in the assignment settings, change the due date to some time in between the time the last submission was uploaded to Markus, and the current time (otherwise not all submissions will show up on Markus). Also disable web submits.
6. Select "Collect All Submissions" from the submissions page to collect the submissions for marking: this will, as usual, take quite some time.

Setting up TA Marking

If you want to pre-make any annotations for the TAs, you can do so under the annotations folder. It is recommended to at least make some folders, so that they can save their own annotations if they'd like to. If the handmarking involves actual marks, then you have the choice between the "Rubric" marking system (where the TAs pick from 0/1/2/3/4 based on a strict set of criteria) or the "Flexible" marking system (where there are guidelines listed and they can enter whatever mark they choose between 0 and the maximum for that part). If you are unsure how to design the marking scheme, look at the marking scheme from past offerings of the course, or ask the instructors for help.

To assign TAs to submissions, go to the Graders section. You can select all submissions by checking the "All" check box over the student names, select the TAs you'd like from the left panel. You can randomly assign TAs to submissions in an even fashion by clicking the green dice. You may want to reassign some submissions later since not all TAs will be comfortable marking Racket. If you've run Moss on the assignment, you can look at the results in the "html_scheme" folder to figure out which students used Racket. Then you can reassign those students to different TAs, in exchange for a C++ submission.

There is a script called "assignGraders.sh" in ~/markus/bin which may help with more complicated situations where you need to assign different amounts of submissions to each grader. However, this script was created in 2017 and the writer of this sentence has not tried using it.

Once TA Marking is done (everything has been marked complete, there is a green checkmark beside each submission) you can select all the submissions on the Submissions page and click "Release Marks": the icon should change to an envelope with a green arrow. Go back to the assignment settings and make the assignment visible to students so they can see their marks.

Note that if students weren't given either annotations nor marks, they won't see any submissions for that assignment under Markus. If there are no actual marks involved in your term's handmarking (text feedback only) make sure the TAs leave annotations on all submissions.

Marking Meeting

While it's tempting to just give the TAs an e-mail to go ahead, you should run a marking meeting before every assignment marked. First off, assign a few of the submissions to yourself (maybe one from each TA or something) and spend a good deal longer than usual marking them, to give you an idea of what mistakes students are commonly making, what you'd like TAs to be looking out for, and so forth. Ask your instructor(s) if they have anything in particular they want the TAs to be looking for.

Set up a marking meeting where you clarify with the TAs what exactly you expect them to do, what the deadline is, mark an assignment (ideally one which could use a fair amount of feedback but is not particularly bad either), and then have them do a few before they leave to make sure everyone is on the same page. Figure out which TAs are not comfortable marking unusual languages such as Racket and which ones are so that you can reassign those submissions appropriately. You might want to send a follow up e-mail a day or so before the marking is done to make sure everyone is still aware of the deadline and is on schedule. Furthermore, ask the TAs to provide some feedback about how the handmarking went and what they generally saw done well or poorly from students: you may want to send this e-mail after the marking is due, as otherwise people will probably have forgotten about it by the time they actually mark.