Using Markus and Handmarking in CS241

As of Winter 2015, CS241 is using Markus for some amount of handmarking (A4 and A9). Markus requires a good deal of setup to get working properly: this guide should walk you through what you need to do.

Initial Markus Setup

  1. Log into markus. You should see three accounts: For example, for me they're sharrap, sharrap-grader, and sharrap-student. The -grader and -student IDs are pretty much what they sound like: aliases so that you can see things from a marker or student's perspective. By default you should have administrative powers within your course.
  2. Make sure to add any instructors, ISCs or IAs for your course as administrators if they aren't already, and any TAs as graders. This can all be done under the "Users" tab, under the appropriate sub-tab. For students, rather than hand-entering every student, you should be able to batch-add them from a CSV classlist file. You can create this with ~cs241/markus/bin/create-markus-classlist.sh which will create a file called ~cs241/markus/classlist.csv.

Setting up an Assignment

  1. Navigate to the Assignments tab and click Add Assignment.
  2. The short identifier, repository folder and assignment name can all be A* (where * is the assignment number). Note that while you can change these fields later (for example, if you break the assignment and want to create a new one since it's more or less unsalvageable) it's highly unadvised to do so and might cause weird behaviour. Just rename the long name to "Old/Ignore" or something and pick a new short identifier/repository folder for the next one.
  3. Set the due date to at least a few hours in the future. Allow web submits (that's how you'll be uploading things), but hide the assignment from students, since they won't be submitting to markus.
  4. Next, navigate to ~cs241/markus and run bin/make-groups.sh: this should automatically generate a file called group.csv which will be necessary for markus later. Back on the assignment page, navigate to the Groups tab and select "Upload" in the top right: upload the group.csv file you just generated. Note that most markus uploads will take quite some time.

Uploading Student Submissions

  1. Edit ~cs241/handin/bin/prune.sh to only keep files with the extensions allowed on the assignment
  2. Download the best on-time submissions for each of the problems, for each student from marmoset. Then combine into a single folder each student's code for the last problem they submitted to. Finally, remove unwanted junk files. You can do so with the 3 scripts in ~cs241/hand.bin.prune
For example, for A4 if there were 6 problems you would run
~cs241/handin/bin/batch_marm_sql.sh A4P1 A4P2 A4P3 A4P4 A4P5 A4P6

~cs241/handin/bin/combine-student-submissions.sh a4handmarking A4P1 A4P2 A4P3 A4P4 A4P5 A4P6

~cs241/handin/bin/prune.sh a4handmarking

This will create ~cs241/handin/a4handmarking

  1. Run ~cs241/markus/bin/upload-to-markus.sh where the first argument is the name of your folder in ~cs241/handin (in the example above, "a4handmarking") and the second argument is the name of the repository folder for this assignment on Markus. Note that this is more or less irreversible. This will take a while.
  2. Once all submissions appear on the submissions page (click all to view more than 30), change the due date to some time between the time the last assignment was uploaded to markus and the current time (otherwise not all submissions will show up on Markus). Also disable web submits.
  3. Select "Collect All Submissions" from the submissions page to collect the submissions for marking: this will, as usual, take quite some time.
Note that you could, in theory, combine most of this into a single bash script to automate the process, but it's unadvisable since many of these scripts will require slight term-by-term editing to make sure they're still doing what they need to do, and it's easiest to do this by simply forcing you to remember which scripts you're running at which points.

Setting up TA Marking

If you want to pre-make any annotations for the TAs, you can do so under the annotations folder. I suggest at least making some folders so that they can save their own annotations if they'd like to. If the handmarking involves actual marks, then you have the choice between the "Rubric" marking system (where the TAs pick from 0/1/2/3/4 based on a strict set of criteria) or the "Flexible" marking system (where there are guidelines listed and they can enter whatever mark they choose between 0 and the maximum for that part).

To assign TAs to submissions, go to the Graders section. You can select all submissions by checking the "All" check box over the student names, select the TAs you'd like from the left panel (if you or any other administrators will participate in the marking then I suggest using your -grader account rather than your main one). You can randomly assign TAs to submissions in an even fashion by clicking the green dice. You may want to reassign some submissions later since not all TAs will be comfortable marking Racket: it's easier to just come up with a list of students who used Racket and reassign those groups to different TAs in exchange for some C++ submission.

Once TA Marking is done (everything has been marked complete, there is a green checkmark beside each submission). From there, you should be able to select all the submissions and click "Release Marks": the icon should now be an envelope with a green arrow. Go back to the assignment settings and make it visible to students.

Note that if students weren't given either annotations nor marks, they won't see any submissions for that assignment under markus, so make sure that if there are no actual marks involved in your term's handmarking the TAs leave annotations on all submissions.

Marking Meeting

While it's tempting to just give the TAs an e-mail to go ahead, you should run a marking meeting before every assignment marked. First off, assign a few of the submissions to yourself (maybe one from each TA or something) and spend a good deal longer than usual marking them, to give you an idea of what mistakes students are commonly making, what you'd like TAs to be looking out for, and so forth. Ask your instructor(s) if they have anything in particular they want the TAs to be looking for.

Set up a marking meeting where you clarify with the TAs what exactly you expect them to do, what the deadline is, mark an assignment (ideally one which could use a fair amount of feedback but is not particularly bad either), and then have them do a few before they leave to make sure everyone is on the same page. Figure out which TAs are not comfortable marking unusual languages such as Scheme and which ones are so that you can reassign those submissions appropriately. You might want to send a follow up e-mail a day or so before the marking is done to make sure everyone is still aware of the deadline and is on schedule. Furthermore, ask the TAs to provide some feedback about how the handmarking went and what they generally saw done well or poorly from students: you may want to send this e-mail after the marking is due, as otherwise people will probably have forgotten about it by the time they actually mark.

Topic revision: r5 - 2017-07-19 - PierreLouisGuidez
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback