MarkUs: SVN Repositories, Groups, and Auto-testing

MarkUs manages the student's submissions with Subversion (SVN) repositories. If you're using MarkUs, someone in your course staff should at least glance through this page. Also, before running the auto-test (RST), it's good to have some understanding of SVN repositories.

SVN Repositories and Groups

A repository is a storage location from which files may be retrieved and added. The revision control software will remember each change you make. You can think of a repository as a special folder, where every change you make is remembered. You can view the folder contents immediately after each change. Each change creates a new revision.

MarkUs lets students work in groups. Most CS courses, however, require students to work individually. In this case, groups will still be created, but students will be in a group by themselves. Thus, if there are 20 students, there will be 20 groups, and each group contains exactly one student.

Each group (or each student) has a repository, and MarkUs is essentially just a web interface to the repository. When students submit a file on MarkUs, they are adding files to their repository. When students delete a file on MarkUs, they are removing files from their repository. Each student gets one repository for the whole term; they do not get one repository for each assignment.

Here is an example, showing some actions performed in MarkUs.

Time yyyy-mm-dd hh:mm	Action performed	Revision	Folders/Files in repository
2012-01-13 13:30	Student logs onto MarkUs for first time	0	None.
2012-01-13 14:00	Student clicks on "Assignment 00" link from MarkUs homepage	1	a00/ folder
2012-01-14 07:00	Student submits a file called a00q1.rkt	2	a00/a00q1.rkt
2012-01-14 07:30	Student submits a00q2.rkt=	3	a00/a00q1.rkt, =a00/a00q2.rkt
2012-01-14:10:00	Student replaces a00q1.rkt with a newer version	4	a00/a00q1.rkt (Modified), a00/a00q2.rkt
2012-01-15 14:30	Student submits a00q3.rkt==	5	a00/a00q1.rkt (Modified), =a00/a00q2.rkt, =a00/a00q3.rkt
2012-01-15 16:00	Student replaces a00q2.rkt with a newer version	6	a00/a00q1.rkt (Modified), a00/a00q2.rkt (Modified), a00/a00q3.rkt

As a tutor, you can view the student's repository contents at each revision. For example, you can view the contents of a00q1.rkt at revision 2 and 4. The latest revision is called the head revision. In the example above, the head revision is revision 6. MarkUs only shows students the head revision. So, students cannot go to a previous revision to retrieve an old file.

You can access a repository and its contents with a URL. Here are some examples of URLs:

• file:///u/markus/markus_cs115_f/svn-repos-root/group_0168
• file:///u/markus/markus_cs115_f/svn-repos-root/group_0168/a00
• file:///u/markus/markus_cs115_f/svn-repos-root/group_0168/a00/a00q1.rkt

Each group (or each student) has a group ID. In the example URLs above, the student has group ID group_0168. This student gets one repository for the whole term, and each assignment is a folder under that repository. Here are two ways to determine the group ID, given a student's Quest ID.

1. On MarkUs, login with Admin Role, go to an assignment, and click the "Submissions" tab. The table shows Quest and group IDs.
2. Login with Admin Role, go to an assignment, and click the "Groups" tab. The "Download" link will let you download a CSV file mapping group IDs with Quest IDs. This CSV file is useful for writing scripts.

You can perform read-only actions on the repository with the svn command from the course account. svn takes some arguments, which are similar to the normal linux commands.

Here are some examples. By default, the svn command will use the head revision.

svn ls URL
This command lists the folders and files at the given URL. For example,

ubuntu1204-002:~> svn ls file:///u/markus/markus_cs115_f/svn-repos-root/group_0168/a00
a00q1.rkt
a00q2.rkt
a00q3.rkt

svn cat URL
This command shows the contents of the given URL. For example,

ubuntu1204-002:~> svn cat file:///u/markus/markus_cs115_f/svn-repos-root/group_0168/a00/a00q1.rkt
This is my a00q1.rkt file
It's very cool.
I submitted this file at 2012-01-14:10:00.

There are other commands you can try (run svn help to see them all). From the course account, you can only run read-only commands. So svn commit will not work.

Terminal: Downloading submissions from MarkUs to course account

You can download a student's submission from MarkUs to the course account by running svn checkout -r RorD URL PATH where

• RorD is either a number or a date. If you put a number R, you will checkout revision R. If you enter a date D, you will checkout the last revision before D. The argument -r RorD is optional. If you omit it, then svn will use the head revision.
• URL is a repository URL
• PATH is where you want to save the copy.

Examples and explanations:

• svn checkout -r 4 file:///u/markus/markus_cs115_f/svn-repos-root/group_0168/a00 my_copy
  Creates a folder called my_copy, which contains revision 4 contents: a00q1.rkt and a00q2.rkt. The folder won't contain a00q3.rkt.
• svn checkout -r "{2012-01-15 09:00}" file:///u/markus/markus_cs115_f/svn-repos-root/group_0168/a00 my_copy
  Does the same thing as the first example.
• svn checkout file:///u/markus/markus_cs115_f/svn-repos-root/group_0168/a00 my_copy
  Creates a folder called my_copy, which contains revision 6 contents: a00q1.rkt, a00q2.rkt, and a00q3.rkt.

Terminal: Adding files from course account to MarkUs

There are several command-line programs for adding files from the course account to MarkUs. Run them in a terminal from the course account.

For all the commands, you need the repository name R. You can get R from the Submissions tab in MarkUs. You can also get it from the groups CSV file downloadable from the Groups tab. R is the second column in the CSV file you download.

You also need the assignment name A. You can look it up in the assignment's Properties page. It is the "Repository Folder/" field.

/u/markus/svn_add_file_stdin

/u/markus/svn_add_file_stdin R/A filename.txt

Creates a file called filename.txt in repository R, under assignment A. The contents of filename.txt will be read from standard input.

In the following example, I am in the cs115 course account. I am adding a file called filetoadd.txt to the repository group_0149, under assignment a00. The contents of the file will be "hello world." You can run this command from any folder.

cs115@ubuntu1404-012:~$ echo "hello world" | /u/markus/svn_add_file_stdin group_0149/a00 filetoadd.txt
Hello cs115! I am markus.
Adding file with name filetoadd.txt to repository group_0149/a00.
Making temporary directory...
Created /tmp/markus_cs115_adding_file_rhzj55nc
Enter file contents for filetoadd.txt:
Read 1 lines of input
Checking out file:///u/markus/markus_cs115_s/svn-repos-root/group_0149/a00 to /tmp/markus_cs115_adding_file_rhzj55nc
Writing to file filetoadd.txt
Calling svn add filetoadd.txt
A         filetoadd.txt
Calling svn commit filetoadd.txt
Deleting temporary directory /tmp/markus_cs115_adding_file_rhzj55nc
End of script. (exit code = 0)
cs115@ubuntu1404-012:~$

If you are adding a file that already exists in the student's repository, you will see warnings like these, which you can ignore. The file will still be updated (ie overwritten).

svn: warning: W150002: '/tmp/markus_cs115_adding_file_378uzr6h/filetoadd.txt' is already under version control
svn: E200009: Could not add all targets because some targets are already versioned
svn: E200009: Illegal target for the requested operation

/u/markus/svn_add_file

/u/markus/svn_add_file R A /path/to/filetoadd.txt

Adds a file called filetoadd.txt to the repository R, under assignment A. The contents of the added file will be the same as /path/to/filetoadd.txt. This script is mostly just a wrapper around svn_add_file_stdin.

In the following example, there is a file called myfile.txt located in the /tmp folder. (The cat command just shows what's in it.) A file with the same name (myfile.txt) is added to the repository group_0149 under assignment a00.

cs115@ubuntu1404-012:~$ cat /tmp/myfile.txt 
hello there!
cs115@ubuntu1404-012:~$ /u/markus/svn_add_file group_0149 a00 /tmp/myfile.txt 
Repository name: group_0149
Assignment: a00
Given file: /tmp/myfile.txt
File's full path: /tmp/myfile.txt
Filename: myfile.txt
Hello cs115! I am markus.
Adding file with name myfile.txt to repository group_0149/a00.
Making temporary directory...
Created /tmp/markus_cs115_adding_file_tdvgkeuj
Enter file contents for myfile.txt:
Read 1 lines of input
Checking out file:///u/markus/markus_cs115_s/svn-repos-root/group_0149/a00 to /tmp/markus_cs115_adding_file_tdvgkeuj
Writing to file myfile.txt
Calling svn add myfile.txt
A         myfile.txt
Calling svn commit myfile.txt
Deleting temporary directory /tmp/markus_cs115_adding_file_tdvgkeuj
End of script. (exit code = 0)
cs115@ubuntu1404-012:~$

/u/markus/svn_add_dir

/u/markus/svn_add_dir R/A /path/to/dir/to/add/

Adds the contents of the folder /path/to/dir/to/add/ to repository R under assignment A. You can use this command to add many files at once. To use this command, the course account should have SSH access in to itself.

In the example below, I add a directory to the student's account:

cs115@ubuntu1404-012:~$ /u/yc2lee/tree /tmp/dirtoadd/
/tmp/dirtoadd/
|-- dir1
|   `-- anotherfile.txt
|-- dir2
|   `-- mygamefile.txt
|-- filea.txt
`-- no

2 directories, 4 files
cs115@ubuntu1404-012:~$ /u/markus/svn_add_dir group_0149/a00 /tmp/dirtoadd/
Hello cs115! I am markus.
Adding directory /tmp/dirtoadd/ to repository group_0149/a00.
Making temporary directory...
Created /tmp/markus_cs115_adding_file_gddu3w92
Checking out file:///u/markus/markus_cs115_s/svn-repos-root/group_0149/a00 to /tmp/markus_cs115_adding_file_gddu3w92
Copying /tmp/dirtoadd/ to /tmp/markus_cs115_adding_file_gddu3w92 with scp.
Running '/usr/bin/svn add --force .' in /tmp/markus_cs115_adding_file_gddu3w92
A         dirtoadd
A         dirtoadd/no
A         dirtoadd/filea.txt
A         dirtoadd/dir1
A         dirtoadd/dir1/anotherfile.txt
A         dirtoadd/dir2
A         dirtoadd/dir2/mygamefile.txt
Running svn commit
Deleting temporary directory /tmp/markus_cs115_adding_file_gddu3w92
End of script. (exit code = 0)
cs115@ubuntu1404-012:~$ 

If you don't want the directory actually in the student's account, first cd to the directory and use . as the path. Example:

cs115@ubuntu1404-012:~$ cd /tmp/dirtoadd/
cs115@ubuntu1404-012:/tmp/dirtoadd$ /u/markus/svn_add_dir group_0149/a00 .             
Hello cs115! I am markus.
Adding directory /tmp/dirtoadd/. to repository group_0149/a00.
Making temporary directory...
Created /tmp/markus_cs115_adding_file_iesdm7xz
Checking out file:///u/markus/markus_cs115_s/svn-repos-root/group_0149/a00 to /tmp/markus_cs115_adding_file_iesdm7xz
Copying /tmp/dirtoadd/. to /tmp/markus_cs115_adding_file_iesdm7xz with scp.
Running '/usr/bin/svn add --force .' in /tmp/markus_cs115_adding_file_iesdm7xz
A         no
A         filea.txt
A         dir1
A         dir1/anotherfile.txt
A         dir2
A         dir2/mygamefile.txt
Running svn commit
Deleting temporary directory /tmp/markus_cs115_adding_file_iesdm7xz
End of script. (exit code = 0)
cs115@ubuntu1404-012:/tmp/dirtoadd$

/u/markus/svn_delete_file

/u/markus/svn_delete_file R/A file_to_delete

Deletes the file file_to_delete from folder A in repo R.

In the example below, I delete a file called test.rkt from student yc2lee-student's a01 folder:

cs115ae@ubuntu1604-008:~$ /u/markus/svn_delete_file yc2lee-student/a01 test.rkt
Hello cs115ae! I am markus.
I'm going to delete file with name test.rkt from repository yc2lee-student/a01.
Making temporary directory...
Created /tmp/markus_cs115ae_adding_file_t9a8igvp
Checking out file:///u/markus/markus_cs115ae_f/svn_repos/yc2lee-student/a01 to /tmp/markus_cs115ae_adding_file_t9a8igvp
Deleting file test.rkt with svn rm
D         test.rkt
Deleting temporary directory /tmp/markus_cs115ae_adding_file_t9a8igvp
End of script. (exit code = 0)
cs115ae@ubuntu1604-008:~$ 

/u/markus/svn_delete_all

/u/markus/svn_delete_all R/A

Deletes the entire A folder from repo R.

In the example below, I delete the entire a01 folder from student yc2lee-student's repo.

cs115ae@ubuntu1604-008:~$ /u/markus/svn_delete_all yc2lee-student/a01
Hello cs115ae! I am markus.
Preparing to delete yc2lee-student/a01
Making temporary directory...
Created /tmp/markus_cs115ae_adding_file_0pi5ujeu
Checking out file:///u/markus/markus_cs115ae_f/svn_repos/yc2lee-student/a01 to /tmp/markus_cs115ae_adding_file_0pi5ujeu
Deleting...
D         a
D         a/testing
D         juice.txt
Running svn commit
Deleting temporary directory /tmp/markus_cs115ae_adding_file_0pi5ujeu
End of script. (exit code = 0)
cs115ae@ubuntu1604-008:~$

Collecting Assignments: Marking the Correct Revision

When you mark an assignment, you have to tell MarkUs which revision to mark. You can do this in two ways:

1. On the assignment's "Submissions" page, there is a link that says "Collect All Submissions". If you click this link, then for every student, you will mark the last revision before the due date. Here, "due date" is whatever you typed in for "Due Date" on the assignment's properties page.
   
   WARNING: Clicking the "Collect All Submissions" link will erase all marking for the assignment.
   
2. On the submissions page, click one of the group IDs. In the new page, you can view all the revisions for the student. Go to the revision you want to mark, then click the "Collect and Grade This Revision" button.

Method 2 is slow because you have to do it manually for each student one by one. Method 1 is faster, but you'll have to change the due date to control which revision to mark.

Accessing lost marking: If you mark an assignment for a student, and then collect that student's assignment, the marking you did will "disappear." More specifically, if you now click on that student's Quest ID in the Submissions page on MarkUs, you will not see the marking you had before. However, you can still access the marking you had before; it's just under a different URL. Example:

1. Suppose you're marking an assignment for a student, and the URL is https://markus.student.cs.uwaterloo.ca/markus_cs135_f/en/main/results/edit/6111
2. You collect the assignment for that student, and now the URL is https://markus.student.cs.uwaterloo.ca/markus_cs135_f/en/main/results/edit/9876. At this new URL, the marking is all blank.
3. You can still go to the old URL (the one with 6111 at the end), and see the marking you had before. You can copy the marking over to the new marking page (the one ending in 9876) manually.

Auto-testing Procedure

There is new, simplified procedure: MarkUsPrepareForMarking
The following instructions should still work.

This section applies to courses that use RST for autotesting. Soon after the assignment is due, you need to run the auto-test and several other scripts. The overall plan is to checkout all the student's submissions, run the auto-test, create GRADED_ASSIGNMENT.ss, and then add it to the repository.

Here is an example using Fall CS115 a04. All commands should be run after the deadline, from the course account, and in this order. The entire procedure takes about 5 hours for a class of 800 students - most of this time is just waiting for scripts to finish.

Attached below is the file autotesting_presentation.zip, which contains a Prezi presentation ( prezi.exe) explaining the auto-testing procedure. It was created by Luc Larocque for CS 115 in Fall 2012. As you read the instructions below, follow along in the presentation.

Step 1: Create a folder where the students' code will be checked out to. Also create a symbolic link in /u/cs115/marking/ so that RST can find the test suite:

mkdir /u/cs115/handin/a04_autotest
cd /u/cs115/marking
ln -s a04 a04_autotest

We will checkout the student's code to a04_autotest.

Step 2: Log onto MarkUs with Admin Role and go to the assignment. Then click the "Groups" tab and click "Download" link. Save the CSV file as /u/cs115/marking/a04/a04groups.csv

Step 3: Set the variables in CheckoutAndCommit.py and MakeGradedAssignment.py. See MarkUsScripts for detailed info about those scripts.

Example Values:

• CheckoutAndCommit.py
  • csv_file_location = '/u/cs115/marking/a04/a04groups.csv'
  • assn = 'a04'
  • checkout_folder = '/u/cs115/handin/a04_autotest'
  • due_date = '2011-10-26 21:10'
  • markus_url = "markus.student.cs.uwaterloo.ca/markus_cs115_f/en/main"
• MakeGradedAssignment.py
  • assn_name = '04'
  • file_list = [ 'a04q1.rkt', 'a04q2.rkt', 'a04q3.rkt', 'a04q4.rkt' ]
  • assn_folder = 'a04_autotest'
  • autotest_folder = 'test.1.AUTOTESTRESULTS'
  • output_filename = 'GRADED_ASSIGNMENT.ss'

Step 4: Checkout the students' code. Using the above values, the students' code will be checked-out to /u/cs115/handin/a04_autotest/

python CheckoutAndCommit.py checkout viewcmds
python CheckoutAndCommit.py checkout runcmds

Courses that use online Stepper and MarkUs: Manually create repositories for students who complete stepping problems, but have no repository in MarkUs. See CompSci135

Step 5: At this step, you can process the assignments with your own scripts. You may want to:

• For Racket code, convert non-plain-text files to plain-text files: WxmeConverter
• Fix file names for students
• Run Moss: MossScript
• For CS115 and CS135: If you want to run CheckTestcases (a script to check if the student has the needed testcases), you can run it now. The script can be run in parallel with distrst (ie in one terminal, you run distrst, and in another terminal, you run CheckTestcases).
• For CS240: you must create your own test cases and place them somewhere in the course account. Then, run a bash script that collects students' code one by one from their handin folders and compiles and runs them against these test cases. Your bash script should take care of steps 6 and 7 below (so, when done, skip to step 8).

Step 6: Run distrst (the exact command may differ).

distrst -t t -s '*' a04_autotest 1 AUTOTESTRESULTS

The argument -t t indicates the output should be in plain-text. The -s '*' indicates that every student should be tested. The a04_autotest argument is the folder where the students' code is located. The 1 argument tells RST to use the test suite test.1 inside /u/cs115/marking/a04_autotest/. The last argument, AUTOTESTRESULTS, is a name you can pick for this test run. The name should match the end of autotest_folder variable above (in MakeGradedAssignment.py)

Step 7: Join the auto-test results and the students' submissions: python MakeGradedAssignment.py. The script will ask if you want to continue. You can enter 'y' or 'n'.

Step 8: Add and commit GRADED_ASSIGNMENT.ss:

python CheckoutAndCommit.py commit viewcmds
python CheckoutAndCommit.py commit runcmds

Take note of the completed time. We'll call this time T.

Step 9: We have to tell MarkUs to mark the revision we committed from the previous step. Logon to MarkUs with Admin Role, and go to the assignment's "Properties" page.

1. Change the deadline to a few minutes after T.
2. Go to the "Submissions" page. Every row should be white, with a blue square for marking state. Click the "Collect All Submissions" link. Wait until every row is green.
3. Change the deadline back to the posted deadline so that students don't get confused.

Step 10: MarkUs lets students submit late. Moreover, the handin folder will still be updated too. This step finds the students who submitted late. The method described is the most secure way of dealing with late submissions; with other methods, there's always a small window during which late submissions will get marked.

When students submit on MarkUs, the course account runs the script /u/markus/bin/autotest_runner_ca.rb. This script writes log messages to /u/cs115/course/markus/markus_autotest_log. You can use this log file to find the students who submit late on MarkUs. For example, you can run:

grep assn=A2 /u/cs115/course/markus/markus_autotest_log

In the output, each line corresponds to a submission. At the start of a line is the time of submission. You can use the repo_path or group_id to find the student's Quest ID.

For each student who submitted late, check if you're grading the correct revision. Also make sure you are grading from GRADED_ASSIGNMENT.ss; the individual files may contain code that was submitted late.

Step 11 (Optional): You can automatically assign correctness marks in the MarkUs rubric using MarkUsScripts#set_mark_sub_command and, if you're using RST, MarkUsScripts#set_marks_rst_sub_command.

Step 12 (Last step): Archive the submissions (check your courses's archiving policies first):

mv -i /u/cs115/handin/a04_autotest /u/cs115/archives/1119/handin/a04
rm /u/cs115/marking/a04_autotest

The second command deletes the symbolic link created in Step 1.

Why do we checkout the students' code to a04_autotest?

In the procedure above, we checkout the students' code to /u/cs115/handin/a04_autotest. The main reason is because MarkUs lets students submit late. If students submit late, the handin folder will get updated. Consider this scenario:

1. 1. Assignment is due at 9:00pm.
   2. John submits late at 9:40pm. His late submission will be copied into /u/cs115/handin/a04/. His previous submission, if any, will be overwritten.
   3. Tutor runs the autotest at 9:46pm. The autotest will mark John's late submission and award him marks he doesn't deserve.

A second reason is to be safe. Updating the handin folder relies SSH working. In some situations (ex. server crashes or gets overloaded), SSH may fail. In this case, the handin folder won't get updated correctly. If you checkout the student's code explicitly again before running the autotests, you'll know for sure you're marking every student's submission.

What exactly happens when students click the "Submit" button on MarkUs?

The Python script /u/markus/bin/post-commit is run as markus user. This script will attempt to ssh into the course account. Once in the course account, the script /u/markus/bin/newMarkusSubmission.py is run. How can we run this script located on markus account, if we've ssh'ed into course account? Answer: That script has world readable and world executable permissions.

The script newMarkusSubmission.py does two things:

1. It updates the handin folder. If it's the student's first time submitting, the script will run a svn checkout command. Otherwise, it will run svn update.
2. It runs the public test. Thus when students submit, a public test is run automatically.

Both post-commit and newMarkusSubmission.py do logging.

• post-commit 's log files are located in /u/markus/bin/logs/. That directory contains logs for every course.
• newMarkusSubmission.py 's log is located at /u/csXXX/course/markus/new_submission.log

Adding Solutions Obtained from an Outside Source

You may have to add submissions obtained from an outside source. For example, students may email you their code and you'll have to add it to MarkUs. Or, your course may use a different website for submission (like Marmoset or submit command tool) and use MarkUs only for hand-marking.

This is the general procedure for adding submissions:

1. If the student doesn't have a repository yet, create one. This can be done from the "Groups" tab in MarkUs. Creating a group will create a repository.
2. Use one of the tools described in MarkUsGroupsSVNRepos#Terminal_Adding_files_from_cours to add the student's submission to the student's MarkUs account.

Example: Adding Emailed Submissions

Sometimes students will email you their solutions instead of submitting it on MarkUs. (They usually do this if MarkUs is down). After you get permission from ISC or instructor, you can add the student's solutions to MarkUs yourself:

1. In MarkUs, login as Admin and go to Assignment > Groups. If the student is in "Unassigned" go to step 2. If the student is in "Assigned" go to step 4.
2. Click the "Add new" button near the top of the right table. This will let you create a new group. For the group name, use the student's Quest ID.
3. Check the student at the left table, and check the group you just made in the right table. Then click the + icon in the middle. This will assign the student to a group by him/herself.
4. In a terminal, navigate to the folder where you saved the student's emailed files. You have to be in the course account. You might have to copy the student's emailed files to the course account first.
5. Use one of the tools described in MarkUsGroupsSVNRepos#Terminal_Adding_files_from_cours to add the student's emailed files.
6. In MarkUs, go to the Submissions page, and find the student. Click on the group_XXXX link and make sure the student's submissions are listed.
7. Click the "Collect and Grade This Revision" button.