MarkUs

MarkUs is an open-source software that lets students submit assignments online. Graders can mark the assignments online, and students can view their marked assignments online as well. MarkUs scripts are fully described and posted at MarkUsScripts.

What do I have to do and when?

The things you have to do differ from course to course. Some courses use MarkUs for both student submission and grader hand-marking. Other courses use MarkUs only for hand-marking and students submit using some other means (such as through Marmoset) and then course staff upload submitted files to MarkUs. Depending on your course, you may have to skip, modify, or do extra steps.

At the beginning of the term:

• Check that MarkUs is set up for your course and term. CSCF should have set up MarkUs for your course and term by the first day of lectures and give your ISC access. Your ISC would then add an ISA.
• Add other course staff (other ISAs, instructors, and marking TAs)
• Add students (for the first time, do this as close to the start of classes as possible for a more stable classlist; then do this daily at the start of the term until a week after the course add deadline because some students will register into the course late)
• Discuss who is responsible for what things with other ISAs tutoring the course. For example, which ISA is responsible for uploading the list of students onto MarkUs? Who will work with instructors on the marking scheme and canned annotations (comments) to go into MarkUs for each assignment?
• Discuss what marking scheme to use (flexible or rubric) for each assignment with the ISC and the course instructors. Also look at past examples that you can use as starting points.

For each assignment:

1. If your course is using RST/Bittersuite for Basic Tests (formerly called Public Tests), then create the test.pt folder in your course account's marking folder.
2. If students submit using MarkUs, then create the assignment in MarkUs. Note the folder name used in MarkUs for the Repository folder. This is the exact name (including capitalization, if used) for the corresponding folding in the course account's handin folder.
3. Test submission with your student account. Make sure that submission and public tests work as expected.
4. Create marking scheme and add some annotations (finish this before marking meeting. You may be able to modify a past marking scheme and annotation files and upload to MarkUs.
5. After the assignment is due, and before the marking meeting:
   1. Run RST and other post-deadline scripts like CheckoutAndCommit (to make RST results available in MarkUs) and Moss
   2. Assign submissions to graders
   3. Look through some sample submissions to see if the marking scheme and annotations should be changed.
6. Run the marking meeting
7. When all the graders are done marking, release the marks to the students.
8. Process any remark requests you get.

Installing and Setting Up

You likely don't need to install or set up MarkUs. CSCF will do that for you.

• MarkUsInstallation
• MarkUsSetupFall2010FullLog

Make sure these folders and files exist. If they exist, you're good. If not, create them and leave them empty.

• folder: /u/csXXX/handin/
• folder: /u/csXXX/marking/
• file: /u/csXXX/course/markus/new_submission.log

markus is an actual user/account. If you're using MarkUs for assignment submission, make sure markus can ssh into the course account without being asked for a password. This means that

• markus should be in /u/csXXX/.rhosts
• markus should be in /u/csXXX/.ssh/authorized_keys

Logging In and Role System

MarkUs is installed on markus.student.cs.uwaterloo.ca. It's reached with the URL https://markus.student.cs.uwaterloo.ca/. That page displays all the courses that use MarkUs. Do NOT post this URL for students to use. Instead, post the URL for your specific course and term, like https://markus.student.cs.uwaterloo.ca/markus_cs115_f.

MarkUs works best with Firefox. In other web browsers, some features may not work. You should use Firefox and recommend this to students.

You can login to the web interface with your Quest ID and Quest password. Omar (CSCF) can log into the markus account via ssh; others use the web interface to interact with it.

After logging in, you will take on one of three roles: Student Role, Grader Role, or Admin Role. Each role allows you to do different things:

Role	Possible Actions
Student Role	Can submit, view your own marks and graded assignment, and view class averages
Grader Role	Can only mark assignments assigned to you. Cannot release marks to students
Admin Role	Can do everything except submit assignments. Admins can mark any submitted assignment.

Most people have no choice over what role to take on. For example, students can only take on the Student Role. It would be dangerous if students can login as Grader Role and mark people's assignments. However, some users like ISAs and instructors can choose a role. This allows ISAs, for example, to both submit and mark assignments.

If you are added under multiple roles, you can switch roles as follows. Only these role switches are allowed.

• To switch from Admin Role to Student Role, click in top-right corner, and enter questid-student (ex. a3tutor-student)
• To switch from Admin Role to Grader Role, click and enter questid-grader (ex. a3tutor-grader)
• To switch from Student Role to Admin Role, click "Cancel role switch" link at top-right.
• To switch from Grader Role to Admin Role, click "Cancel role switch" link.

Tip: From the Admin Role, you can log in as any student or grader. After you click , you can type in a student's username, or any of the graders' usernames. Logging in as a grader is useful for determining how many assignments the grader still has to mark.

Adding Students and Course Staff

You have to add users at the start of the term. Login with Admin Role to add users.

• Students: Add them under the Student Role only
• Marking TAs: Add them under the Grader Role only
• Instructors, ISCs, and ISAs: Add them under all three roles.

Once someone is added onto MarkUs, that person cannot be removed. However, you can change the username to be something like "person-removed". You never have to remove students from MarkUs. For example, if a student drops out, just leave the student in MarkUs.

Adding Students

A student cannot submit unless they are added under the Student role/tab in MarkUs. The list of students in MarkUs should be a superset of the classlist. Students who are not officially enrolled in the course can still be added to Markus to submit assignments, assuming your instructor/ISC approves.

To add many students at once, use the "Upload from course account" link or upload a custom CSV file.

To add individual students, click the "Add new" link from Users->Manage Students page. For "User Name", enter the student's Quest ID with no special suffixes (ex a9stud). Enter the Quest ID in all lower case, and up to 8 characters only. Grace Credits are for allowing late submissions, so most courses will leave Grace Credits at 0.

Usually you do not have to do anything when students drop the course. However if you wish to prevent someone from accessing Markus, you can mark them as "inactive" from the drop-down menu on the Students tab.

Adding Course Staff

Add marking TAs one by one under the Grader Role (click the "Add new" link). For the "User Name", enter the TA's Quest ID with no special suffixes (ex a5grader).

Add instructors, ISCs, and ISAs under all three roles. You can do this manually. Here are the usernames to use when adding:

• Admin Role: Your Quest ID with no special suffixes (ex a3tutor).
• Grader Role: Your Quest ID with -grader suffix (ex a3tutor-grader).
• Student Role: Your Quest ID with -student suffix (ex a3tutor-student).

Setting Up and Creating Assignments

If your course uses public tests, create the public test first, ie create the test.pt folder first. When students submit on MarkUs, a public test is automatically run.

Warning: Once an assignment is added, it cannot be removed. However, you can hide the assignment from students by checking the "Yes" box next to "Hide assignment from students?" on the Assignment Properties page.

You can add an assignment by logging in with Admin Role and clicking "Add Assignment" button. Fill out the fields, and click the "Submit" button when you're done.

After the assignment is created, switch to Student Role and try submitting. Make sure everything works, and that you receive a public test email.

Field	Explanation
Short Identifier	A short name for the assignment, like "CS115 a04"
Name	A longer name for the assignment, like "Assignment 04"
Message	You can write a little message to the students, such as: When you submit on MarkUs, a public test is automatically run. You can check the public test results by checking the email you set in Quest.
Due Date	The assignment due date.
Repository Folder	If you enter Y here, then, when students submit, their code will be saved automatically on the course account under /u/csXXX/handin/Y/. You can enter something like a04. Make sure you type this field correctly! Once an assignment is created, you can't change this field from the web interface (CSCF can help change it if necessary though).
Allow Web submits?	If your students will be submitting on MarkUs, choose "Yes." Otherwise, pick "No." For example, if your students submit on Marmoset or using the submit command line tool, then you would choose "No."
Display submission and revision history to students	Checking this will show students a table of the changes made to their files. Students will be able to see all changes, including those made by course staff with tools like svn_add_dir described in MarkUsGroupsSVNRepos#Terminal_Adding_files_from_cours. The table has 4 or 5 columns (see row below): DATE AND TIME: Shows the date and time the file(s) were changed. AUTHOR: Who changed the files. Typically it's either the student, markus, or the course account. CHANGES: Describes what changes were made (ex. "Updated a03q2.rkt") DOWNLOAD FILES: Allows students to retrieve files from previous submissions. TEST SUMMARY: This column only appears if students are allowed to view basic/public test results. It will show students a short summary of their basic test mark, and students can click the link to see more detailed results.
Let students run and view basic/public test results	If you check this, students can see their basic test results in on MarkUs web site (in addition to getting the results in an email). The students can also request basic test results if they want to run basic test again.
Hide assignment?	If you check this, students won't be able to see or submit the assignment. This feature is useful if you're still developing the assignment, or if you created an assignment by mistake.
Marking Scheme	Choose the marking scheme you want to use. The two types are described below.
Required Files	List the files students have to submit.
Other fields	Typical settings are: Do not let students work in groups and accept no late submissions. Adjust settings as needed for your course.

Turning off public/basic tests

By default, when students submit on MarkUs, MarkUs will run the public/basic tests using RST/BitterSuite. If your course does not need RST/Bittersuite for public/basic tests, you can tell MarkUs to not run it.

• To disable public/basic tests for every assignment, create the file /u/csXXX/.markus_no_public_test where csXXX should be replaced with your course number. The contents of the file does not matter; if that file exists (and is readable from the course account), MarkUs will not run the public tests.
• To disable public tests for particular assignments, create the folder /u/csXXX/.markus_no_public_test/ and in that folder, create a file for each assignment that doesn't need public tests. The name of the file should match the name in the /u/csXXX/marking/ folder. The contents of the file(s) do not matter.

Groups, SVN Repositories, and Auto-testing

The GRADED_ASSIGNMENT.ss file is a file created by the ISAs. It contains the student's auto-testing results followed by their submitted code. The marking TAs like this file, because it contains everything; they don't have to switch from file to file. A sample GRADED_ASSIGNMENT.ss is attached.

Information about SVN repositories and how it relates to MarkUs is here: MarkUsGroupsSVNRepos. That page also explains the auto-testing procedure and how to add emailed solutions.

Hand Marking

MarkUsGradingInterface describes MarkUs 's grading interface. It is focused on the Grader Role, but since the Admin Role can mark assignments too, it applies to that as well. If you need to mark several assignments, mark from the Grader Role, so that you don't accidentally mark an assignment assigned to someone else. If you only need to mark one assignment (such as when handling remark requests), you can mark from the Admin Role.

Finish the marking scheme and add some starting annotations before the marking meeting. You only have to add the basic annotations; as graders mark, they will add more annotations. The annotation categories and the annotations themselves cannot be sorted, unfortunately (See Possible Improvements below).

For both the marking scheme and annotations, you can save them onto your computer (click Download link) and upload them (click Upload link). You can use this feature to copy the marking scheme and annotations from assignment to assignment. For example, if you want to copy the marking scheme from Assignment 05 to Assignment 06:

1. Go to Assignment 05's Marking Scheme tab. Click "Download" link and save the CSV file somewhere on your computer.
2. Go to Assignment 06's Marking Scheme tab. Click "Upload" link and open the file you saved in Step 1.

Copying annotations from assignment to assignment is similar.

Collecting Assignments and Recovering Lost Marking

See MarkUsGroupsSVNRepos#Collecting_Assignments_Marking_t

The Two Marking Schemes

You have to create a marking scheme before you can start marking. MarkUs offers two types of marking schemes: flexible and rubric. Ask your ISC and instructors to decide which one to use.

Flexible Marking Scheme

When you use the flexible marking scheme, you can create some criteria. For each criterion, you need to specify three things:

1. The name of the criterion
2. A description for the criterion
3. The mark that the criterion is out of

The name and description can be seen by graders and students, so don't type any sensitive information. Graders assign marks by typing them in.

Rubric (4-level) Marking Scheme

When you use the rubric marking scheme, you can create some criteria. For each criterion, you need to specify its name and a weight w. The weight w is just some positive number, and the criterion will be out of 4*w marks. The weights are used to give certain criteria more weight.

You also need to set some levels. The available levels are 0, 1, 2, 3, and 4. You set a level by providing a name and description for it. If a level has no name and no description, it is not set and graders won't be able to assign that level. Levels 0 and 4 should always be set, so that students can get zero and perfect scores.

Graders assign marks by clicking on the level they want to give. If a student gets level n, then that student gets n*w marks out of a possible 4*w marks. Students can only see the description for the level they are assigned.

Illustration: Suppose you set the weight w as 1.5 and you set levels 0, 1, 3, and 4. Then the criterion is out of 4*w = 4*1.5 = 6 marks, and the marks for each level are:

Level Given	Marks Given
0	0 out of 6
1	1.5 out of 6
2	impossible, level not set
3	4.5 out of 6
4	6 out of 6

As examples, if the weight is 1.5, then marks of 2/6 and 5/6 are impossible (even if level 2 was set).

Assign Submissions to Graders

Assign submissions to graders after the assignment deadline. Graders can only see assignments that they're assigned to, so they cannot accidentally mark a submission assigned to another grader (this is a good thing).

To assign submissions, login with Admin Role, click the assignment, and choose "Graders" tab.

You should use the markerallocation.py script to assign graders (detailed info is at MarkUsScripts).

You can also assign graders manually. This is useful if you need to make small adjustments to the output of markerallocation.py. It's also useful if you want to assign specific student submissions to a grader. On the "Manage Graders" page, there are three icons in the middle:

• : Select 2 or more graders, and select some students. This button will randomly assign the graders to the selected students.
• : Select 1 grader, and select some students. This button will assign that grader to the selected students.
• : Don't select any graders, but do select some students. This button will remove the graders from the selected students.

Releasing Marks to Students

"Releasing marks to students" means letting students see their marks. Marks can only be released from the Admin Role. You should release the marks after all the assignments have been marked, ie their marking state is . If most of the assignments are marked, but a few are incomplete, find the TA responsible for those incompletes (from the "Graders" tab) and contact him/her.

To release marks, login as Admin Role, go to the assignment, and click "Submissions" tab. Check everyone whose marking state is . You can use the "Select: All None" links near the top-left. Then click the "Release Marks" button. The will change to , meaning the marks have been released.

Binary Content

Students may submit files with binary content. For example, if they copy and paste text from a PDF file into their code, their code may have binary characters in it. When you try and mark the student's code in MarkUs, MarkUs will display "Binary content: Use download button to view file!" instead of the student's code.

You can use the Ruby script is_binary.rb (attached below) to check if a file F has binary content or not. It uses the same algorithm as MarkUs. Run the script from a terminal, giving it the path to F as an argument. The script will print a message telling you if F is a binary file or not. It also exits with code:

• 2 if F does not exist, or if the incorrect number of arguments were supplied
• 1 if F is not a binary file
• 0 otherwise ( F is a binary file)

To run is_binary.rb for every student, you can write a wrapper Bash script, similar to the following:

#!/bin/bash

assn=a10
cd /u/cs135/handin/${assn}_autotest
for stud in *; do
    # Run is_binary.rb on the student's GRADED_ASSIGNMENT.ss file
    ruby /u/cs135/u/yc2lee/is_binary.rb $stud/GRADED_ASSIGNMENT.ss > /dev/null

    # check if exit code is not 1 (ie check if file is binary)
    if [ $? -ne 1 ]; then
       echo $stud
    fi  
done

This Bash script is for CS 135's Assignment 10 (a10). For every student, it will check if the student's GRADED_ASSIGNMENT.ss file has binary content; if it does, the student's Quest ID will be printed.

Once you have identified the students who submitted binary content, you can fix their files by deleting the binary characters. Then use svn commit command to commit your changes (see MarkUsGroupsSVNRepos for more information about SVN and committing).

Known Bugs and Issues

See MarkUsIssues

Possible Improvements

Possible improvements for MarkUs 0.11.0:

• Allow the annotations to be sorted.
• Marking a different revision (by collecting a revision) will overwrite previous marking. If you want to change GRADED_ASSIGNMENT.ss, you'll have to remark the entire assignment. This makes remarking autotesting very difficult. Although MarkUsGroupsSVNRepos#Collecting_Assignments_Marking_t allows you to do the remark by manually copy and pasting the marking you had before.
• Display the average mark awarded by each grader, so that I can tell if some graders are being too lenient or too harsh. Also have a better interface to know the marking progress of each grader.

Old Material

MarkUs has evolved quite rapidly. The material below accumulated during that evolutionary phase. There may be information that is still useful, but in general the info below should be taken with a substantial grain of salt and not trusted.

OldMarkUsMaterial