Old MarkUs Material

MarkUs has evolved quite rapidly. The material below accumulated during that evolutionary phase. There may be information that is still useful, but in general the info below should be taken with a substantial grain of salt and not trusted.

Overview

MarkUs is a tool being developed by Karen Reid and others at UofT. The goal is to make submission, marking, and return of student assignments paperless. MarkUs currently includes the following features:

• web-based submission of assignments by students
• specification of assignment parameters: name, required files, due date, late policy
• support for forming and working in groups (unused in the pilot)
• establishing rubrics (marking schemes) for each assignment
• establishing a set of canned comments that can be used to annotate submissions
• tools for graders to assign marks (using the rubric) and annotate submissions with either canned comments or ad hoc comments
• administrative tools to add users, release marks, etc.

Some fallbacks of MarkUs:

• Once an assignment is created, it cannot be deleted.
• There is no support for testing student submissions.
• Difficult to include marks from auto-marking.
• There is no way to remove a student if s/he was added in error.
• No way to filter student submissions so it only includes specified files or to exclude overly-large submissions or submissions that have illegal characters or ...
• No scripting API.

Added by Bob at end of Winter 2010:

• Assignments cannot be deleted once created.
• Access may be blocked suddenly (i.e. the system cannot load new submissions when grading, and will jump out to authentication page when refreshed).
• The order gets reversed when uploading a marking scheme from a CSV file
• When using "randomly assign marker", we should be able to specify the number of assignments for each individual marker, as the tutor(s) need to mark fewer submissions.
• Bonus calculation system should be added. Student frequently complained on the bonus showing as regular marks.
• There is no direct link to return to the submission list of a specific assignment from the marking panel in grader mode. Currently we have to click the Home tag to go back to the list.
• Sometimes the assignment selection list on the top left corner will become transparent and not selectable (disappears when the mouse moves). This happens only in the most current version and can be solved by choosing another assignment tab (e.g. properties)
• The order is random on the short list of assignments on the top-left corner.
• Highlights for annotation always cover entire line, not a section of the code.
• Currently, students can only read the notes on the rubric on the specific grade given, thus they may not be clear on what we really want (e.g. 4/4). One temporary way to solve the problem is to post the rubric on the course website for every assignment, or a general one at the beginning of term.

Added by Yi Cheng (Nick) Lee at the end of Spring 2010:

• CS 115 used MarkUs 0.7.
• I added descriptions of the rubric and flexible marking schemes.
• I added a few clarifications here and there.
• The flexible marking scheme was used only on Assignments 1 and 3, because the graders felt that it was too detailed and marking was taking too much time.

Question: Was the issue the markers had with the flexible rubric an inherent property of the flexible rubric, or did it reflect the way it was being used? Would it have any uses that would both take less time and perform tasks not possible with the standard 4-level rubric?

Answer: It was an issue with the way the flexible marking scheme was being used. The flexible marking scheme is more flexible since you're not restricted to 4 levels. It is better for entering autotesting marks. The graders say the 4-level scheme is "faster", but that's only because every time we used it, we had fewer criteria. So to make the flexible scheme better, simply get rid of or combine some criteria. Then the flexible scheme will be fast to use (due to fewer criteria), and students can get the exact autotesting mark they deserve (something the 4-level scheme cannot do).

MarkUs at Waterloo

We are currently (as of 25-Feb-2010) using version 0.6.0.

MarkUs is installed on markus002.student.cs.uwaterloo.ca. It's reached with the URL https://markus002.student.cs.uwaterloo.ca/. Byron and Omar can log into the account via ssh; others use the web interface to interact with it.

Authentication

By default, MarkUs passes a userid and password to a program that is specific to the site that returns whether the user has been authenticated or not. Omar used a different approach, hooking up our standard Apache-based authentication tools. Once authenticated, he loads a customized version of the MarkUs login page with a "continue" button to fake out the system. If you use the URL https://markus002.student.cs.uwaterloo.ca?admin it will display options for logging in with your own username or -grader or -student. Assuming all these users actually exist, you can choose your role. If you log-out you'll go back to the generic "continue" screen. Add the "?admin" to the end of the URL and you get the login options again.

Users need to be identified to the system. The first admin is added using a command-line tool when the system is installed. Graders and other admins can be added by hand or uploaded. For students, there is a tool in cs135/markus/bin/parseStudentsToLoadIntoMarkus.py that reads the .classlist file and outputs a csv file that can be uploaded to MarkUs. The upload will either add or update students; so no need to exclude students already in MarkUs. However, the tool will not delete students not in the current list (i.e. student dropped out of the course).

Groups and SVN Repositories

The tutor in charge of MarkUs should know about SVN repositories and the svn command. There are many tutorials online that teach you about SVN.

MarkUs lets students work in groups. Most CS courses however require students to work individually. In this case, groups will still be created, but each student will be in a group by him/herself. The group name is the same as the student's Quest ID.

In MarkUs, login as Admin and go to Assignments > Groups. At the left, there are "Unassigned" and "Assigned" tabs. Students in "Unassigned" have never clicked the assignment link on their MarkUs homepage, while "Assigned" students have. In the right table, you can see all the groups and the members of each group. Recall that each student is in a group by him/herself.

MarkUs uses SVN repositories to manage each group's submissions. Each group has a repository with URL svn+ssh://markus@markus002.student.cs.uwaterloo.ca/u/markus/markus_csYYY_s/svn-repos-root/group_XXXX. Here, csYYY is your course name (ex. cs115) and group_XXXX is an ID unique to each group (ex. group_0322). The URL may change from term to term, because the MarkUs folder and file names may change. You can use the svn command on the URL given above. You can determine the group's group_XXXX id from the Submissions page in MarkUs.

How to add emailed solutions to MarkUs

Sometimes students will email you their solutions instead of submitting it on MarkUs. (They usually do this if MarkUs is down). After you get permission from ISC or instructor, you can add the student's solutions to MarkUs yourself:

1. In MarkUs, login as Admin and go to Assignment > Groups. If the student is in "Unassigned" go to step 2. If the student is in "Assigned" go to step 4.
2. Click the "Add new" button near the top of the right table. This will let you create a new group. For the group name, use the student's Quest ID.
3. Check the student at the left table, and check the group you just made in the right table. Then click the + icon in the middle. This will assign the student to a group by him/herself.
4. In a terminal, navigate to the handin/aXX folder on the course account. If the student already has a folder, go to next step. Otherwise, checkout the student's repository using the "svn co" command. You'll need the group_XXXX id, which you can get from the Submissions page in MarkUs.
5. Save the files that the student emailed you into the handin/aXX/questID folder (questID is the student's actual Quest ID). Commit the changes with "svn ci"
6. In MarkUs, go to the Submissions page, and find the student. Click on the group_XXXX link and make sure the student's submissions are listed.

Interaction with Public Test and Autotest Systems

How Handin Is Updated

When a student submits a file to MarkUs, a UW-added hook is run that copies the files from the MarkUs repository to the handin directory in the course account to facilitate compatibility with our testing infrastructure (public tests, autotesting). That is, MarkUs calls /u/markus/bin/post-commit. The name of this program is obtained from the config file in markus/config/environment.rb. The actual invocation is from markus/lib/repo/subversion_repository.rb. The post-commit script is used by all courses using MarkUs.

The post-commit hook does an ssh call to the course account to run /u/markus/bin/newMarkusSubmission.py with the path, username, and assignment directory as arguments. That program does one of two things. If it is the first submission for the assignment, it creates the appropriate directory in handin and checks out the files from the MarkUs repository (it's an subversion (svn) repository). For subsequent submissions, it simply tells svn to do an update.

If submission is not working as expected (for example if the students' code are not being copied into handin folder), please check the following. I will use CS 135 in Winter 2011 as an example.

• The user markus should be able to ssh into the course account without being prompted for a password.
• The user markus should be in the course account's .rhosts file.
• Check that the Python script /u/markus/bin/post-commit is correct.
• The post-commit script, which is run as user markus, will attempt to ssh into the course account and run /u/markus/bin/newMarkusSubmission.py. Make sure this latter script exists and that it has executable permissions. Also check for errors in the script (such as wrong directory paths, typos, etc.).
• Both the post-commit and newMarkusSubmission.py scripts do logging. If you read the scripts, they will tell you where the log files are. The log files may help you fix problems.

Be aware that MarkUs lets students submit late assignments. The student will get a warning, but his folder in handin will still be updated. So something like this might happen:

1. Assignment is due at 4:00pm.
2. John submits late at 4:40pm. His late submission will be copied into the handin folder. His previous submission, if any, will be overwritten.
3. Tutor runs autotest at 4:46pm. The autotest will mark John's late submission and award him marks he doesn't deserve.

It is because of this that the assignment procedure below says to run distrst "2-3 minutes after the deadline". However, if you have a large class, the autotest will take a while. So a student can submit after the autotest has been started, but before the autotest gets to him or her.

An alternative solution is this. Create a temporary folder in handin (for example a5_temp). After the deadline, checkout everyone's submission to a5_temp. You should checkout the student's last submission before the deadline (use the -r argument for svn command). Then you can run the autotest on a5_temp whenever you want.

This procedure will guarantee that you mark all assignments that was submitted on time. In Winter 2011, there was an ssh problem which prevented newMarkusSubmission.py from being run. So the handin holder wasn't getting updated.

RevertToDueDate

Script

Effect

The

RevertToDueDate

script (archived in the folder u/cs135/archives/1109/scripts) can revert a student's submission to a specific time. It should be run before

distrst

to avoid marking late submissions.

Usage

The variable "assign" and the command "svn update -r "{2010-10-20 20:05}"" should be updated each time before running the script. Double check to make sure it has the correct assignment number and due date.

  assign="/u/cs135/handin/a05/"
  cd $assign
  for d in *
  do
    cd $assign$d
    svn update -r "{2010-10-20 20:05}"
  done

Recommendations and Improvements

It is possible that a student submits after we run the

RevertToDueDate

script but before we run the

distrst

script, so that his/her late submission gets marked. Therefore, it is a good idea to choose a time that students usually do not submit assignments. For example, an early morning after the deadline.

A better solution is to integrate

RevertToDueDate

into

distrst

to produce a more powerful script, which reverts a student's submission and runs Auto-tests immediately after reverting.

Problems with Updating Handin

The post-commit hook sometimes leaves the .svn permissions in the handin directory in an inconsistent state by leaving them without write access for the user. This screws up subsequent updates. Until this can be tracked down, there is a script, /u/cs135/bin/handin_w_sweep, that detects this anomaly. It takes the assignment number as the only parameter. Directories that have the permission messed up should be fixed with the following commands:

  cd handin/<assign_dir>/<student_dir>
  chmod -R u+w .svn
  svn cleanup
  svn update

This script should be run and problems addressed before the automarking tests are run. Occasionally the fix, above, does not get everything. Likely it should be chmod -R u+w ., but that needs more testing.

Manually update OUTPUT.txt

Sometimes additional tests were run and OUTPUT files need to be manually updated to the handin folder and MarkUs repository. Procedure: copy the output file to the student's handin directory (in W10 it is OUTPUT.txt)

   
   cd handin/<assign_dir>/<student_dir>
   svn add OUTPUT.txt
   svn ci -m "Adding autotest output."

• Note: If the file was already there (i.e. update instead of add), just go directly to the last step (svn ci ...)

Updating MarkUs with Test Results

Sweep the handin directories for permissions problems; see above.

After autotests have run, a script ( /u/isg/bin/markus_testcommit <assign> <suite> <timestamp>) copies the autotest output back into the repository the handin directory on the course account. It then tells svn to add the output and commit the changes to MarkUs' svn repository. When running the command, it's a good idea to redirect the output to a file in case you need to review it. We're still having problems with permissions and you may need to look at the errors.

Graders also like to see all of the submitted files collected in one file. This is done with /u/cs135/markus/bin/cat_submitted_files. This script is still (as of 26-Jan-2010) a hack in need of work. For each assignment it needs to be edited to update the list of files to catenate together and the assignment number. Pay attention to the order of the list so it matches the order graders will want to see the files!

Important: Both of these scripts invoke svn. svn doesn't like to be invoked more than once on the same working copy. So... these scripts should not be run simultaneously. They run at different speeds and it's possible that svn will interfere with itself.

These scripts take a while -- on the order of 20 minutes to handle 150 students.

After the above files have been committed to MarkUs, the due date needs to be reset (to just after the time the last job finished) so that the graders see those files by default.

Suggestions for improvements to cat_submitted_files

I would suggest replacing this script with a specialized RST suite, as this is already duplicating effort that RST is designed to do. Then, a special location for the file list can be designated for an ordered file listing (stored in a predetermined file name?), and all of the makeStarLine+file_cat effort can be handled very naturally via RST's keepFile interface in the computeMarks script.

ie, with just a few extra parameters to keepFile and a bit of wrapping code, this essentially becomes for file in file_list; keepFile $file; done

and then an extension of the mechnanism already being used to check in OUTPUT.txt files can presumably be used to check in the generated OUTPUT.txt from this test run also (but importantly with a different file name!)

-- TerryVaskor - 01 Sep 2010

MarkUs Procedure

Beginning of term

• Install MarkUs (more detail needed as I haven't actually gone through the process)
• Add Admin: Instructors, Tutors, ISC
• Add Graders: Instructors (for debug/test), Tutors, TAs
• Add students with parseStudentsToLoadIntoMarkus.py with .csv format output. (script needs edit to suite the course) Upload the csv file to MarkUs. Also, add Instructors and Tutors as students for debug.
• Discuss and set up a general rubric / annotation pack (can do it for A01).
• edit handin hook at MarkUs system (need to ssh in)

For Each Assignment

Set up assignment:

• Create a? folder in handin/
• Create assignment in MarkUs.
• Check the "Allow Web Submit"
• Set up deadline. Make sure you disable the group.
• Test submission with your student account.
• Try public test to make sure the rst can access your files.

Before / At deadline:

• Prepare the rubric and annotation: Usually download it from last assignment and upload the same files for the current one. Change accordingly.
• Sometimes it's better to start with only the basic annotations, and let TAs add annotations according to the assignments they mark.
• As MarkUs in W10 can only accept a rubric scale of 0-4, we mark the autotests with a "conversion" rule (roughly): 0%=0, <50%=1, 50%~80%=2, 80%~95%=3, 95%~100%=4. (Make sure you put it in the rubric)
• edit the cat_submitted_files before the deadline, changing assignment name and files that need to append together.
• 2-3 minutes after the deadline, start the distrst test and wait for it.
• after the autotest is run, start the cat_submitted_files.
• After inspecting some OUTPUT.txt files to make sure your autotest is valid, run MarkUsScripts#upload_marking_sub_command.
• Set the new due time on MarkUs. If you go to the Submissions page after the deadline, you should see blue squares in the "Marking State" column (assuming you did not click on the student's Quest ID). You need to change the deadline while the blue squares are there. If you click on a student's Quest ID, the blue square will turn into a pencil. Make sure you change the deadline before clicking on the student's Quest ID. Should you click on the student's Quest ID before changing the deadline, you'll need to click the "Collect and Grade This Revision" button, which you access by clicking the group_xxxx links in the "Repository" column.

When marking, the allfiles.ss created from cat_submitted_files should become the main reference, as the changed deadline may make the MarkUs system accept some late submissions. Also, please remind the graders to set the Marking State to Completed when they are done marking.

Assigning Graders

Graders need to be assigned in the Admin -> Assignments -> Graders view. To facilitate this, there is a script in the course account called markerallocation.py (located in cs135/archives/1109/markus) that will create a group/grader map that you can upload. To run the script, first download the groups list as a csv file by going to Admin -> Assignments -> Groups and clicking download in the top right corner. Next create a csv file with the names of all you markers in column A, and the number of students assigned to each marker in column B (see cs135/archives/1109/markus/marking_list_a10.csv for an example). Put both csv files in the same directory as where you saved markerallocation.py, and then open the .py in a text editor to change the file names so that they match those of your csv files. Open the terminal, navigate to the directory and run the script (by typing python markerallocation.py). The resulting file will be your group/grader map. Upload this map to the Graders view by clicking upload in the top right corner.

-- MaxTabordMeehan - 23 Dec 2010

For CS 115, a copy of markerallocation.py is at /u/cs115/marking/Marker_Allocation/markerallocation.py

Thanks CS 135

-- Nick Lee