Automatically Assigning Correctness Marks in MarkUs

This page is about a program that can automatically assign correctness marks on MarkUs. It assumes that your course uses RST and a 4-level rubric in MarkUs.

Background Information

Some courses like CS115 and CS135 use RST to run autotests on the students' code, and then hand-mark the assignments using MarkUs's 4-level rubric. The autotests will give the students a score based on how many tests they passed. For example, they might get marks like these (this would appear in GRADED_ASSIGNMENT.ss file):

**** Automated Testing Results ************************************************

107/126   Automarking total

 ** Question 0: 0/0
 ** Question 1: 12/12
 ** Question 2: 19/19
 ** Question 3: 8/8
 ** Question 4: 6/6
 ** Question 5: 50/50
 ** Question 6: 0/10
 ** Question 7a: 0/3
 ** Question 7b: 0/6
 ** Question puzzles: 12/12

(Question 0, Test AdvancedFilter, 0 marks): Passed; Files passed basic re-
    quirements
(Question 0, Test BasicFilter, 0 marks): Passed; Files passed basic require-
    ments
(Question 1, Test 1, 1 marks): Passed
(Question 1, Test 2, 1 marks): Passed
(Question 1, Test 3, 1 marks): Passed

... and so on ...

When you go and mark the assignment on MarkUs, you have to turn the marks above to marks out of 4 (Level 0 to Level 4). You have to manually click on the correct level. This copying of marks from GRADED_ASSIGNMENT.ss to the rubric can be done automatically.

Overview of the Program

Attached below is a C++ program that can assign correctness marks automatically. It is written in C++ using the Qt framework. It uses QtWebKit, which is part of Qt.

The program emulates a human. When run, it will open a browser window, go to the MarkUs website, login, and possibly go to one student's assignment in MarkUs and assign marks for that one student. To assign marks for several students, you can run the program repeatedly on each student (probably using a script).

Setting Up the Program

The school's linux.student.cs computers do not have the needed libraries, so you will have to run the program on your own laptop or desktop at home.

In order to use the program, you'll need a C++ compiler such as g++. You also need to download and install Qt and QtWebKit. Once you have these tools, download auto_assign_correctness.zip, unzip it, and cd into the created folder.

Edit main.cc by opening it in a text editor. About 20 lines down from the top, you should see:

QString markus_base_url("https://markus002.student.cs.uwaterloo.ca/markus_cs135_f/");
QString course("CS135");

//
//  YOU CAN ENTER YOUR MARKUS LOGIN INFORMATION HERE
//  You can also leave them blank. You'll be prompted for the username and/or password.
//
#define USERNAME ""
#define PASSWORD ""

Change the URL to your course's current MarkUs URL. Be sure to leave one slash (/) at the end. Also change the course name.

You can also enter your Quest username and Quest password in the #define lines. This information will be used to log onto MarkUs. If you do enter your credentials, make sure you keep the file safe. If you do not enter your username or password, you'll be prompted for it when you run the program.

To compile the program, run qmake && make. This should create an executable markus which you can run with ./markus:

$ ./markus
Format 1: ./markus -urls 'Assignment 07'
Format 2: ./markus -urlid 4244 a06 yc2lee-student m1 m2 ... mN
Format 3: ./markus -assn 'Assignment 07' yc2lee-student m1 m2 ... mN
The student will get the marks m1, m2, ... mN. If no m's are given,
then this program will only display marks, not change them.

Using the Program

Format 1: ./markus -urls 'Assignment 07'

When run like this, the program will:

1. Go to https://markus002.student.cs.uwaterloo.ca/markus_cs135_f/en/main/assignments
2. Go to Assignment 07's Submissions tab, and obtain links to each student's assignment grading page. The program will print something like:

stud1,https://markus002.student.cs.uwaterloo.ca/markus_cs135_f/en/main/results/edit/4285
yc2lee-student,https://markus002.student.cs.uwaterloo.ca/markus_cs135_f/en/main/results/edit/4244
j9smith,https://markus002.student.cs.uwaterloo.ca/markus_cs135_f/en/main/results/edit/4306
... and so on ...

Format 2: ./markus -urlid 4244 a06 yc2lee-student m1 m2 ... mN

When run like this, the program will:

1. Go to https://markus002.student.cs.uwaterloo.ca/markus_cs135_f/en/main/results/edit/4244, which is a page to grade a student's assignment.
2. Make sure that the web page contains "CS135 a06 : yc2lee-student".
3. If the web page does contain that string, then it will give the student Level m1 for the first criteria, Level m2 for the second criteria, and so on. If any of the =mi='s are =x='s, then the program will skip that criteria.

Format 3: ./markus -assn 'Assignment 07' yc2lee-student m1 m2 ... mN

When run like this, the program will:

1. Go to https://markus002.student.cs.uwaterloo.ca/markus_cs135_f/en/main/assignments
2. Go to Assignment 07's Submission tab, and find yc2lee-student. It will go to =yc2lee-student='s assignment.
3. Once at the assignment grading page, it'll assign Level m1 for the first criteria, Level m2 for the second criteria, and so on.
4. When finished, or when 1 minutes has passed (whichever occurs first), the program will quit automatically.

When the program is finished, or when 1 minutes has passed (whichever occurs first), the program will quit automatically.

The idea is that you will first use Format 1 to obtain the 4-digit numbers at the end of the URL for each student, then run the program again on each student using Format 2. Format 3 does not require you to know the 4-digit number, but it is much slower. A helper Python script is used to run the program repeatedly on each student. In summary, you'll do something like:

1. Run ./markus -urls 'Assignment 07' > UrlList.txt which saves the output the file UrlList.txt
2. Use the Python script wrapper.py below, which will use UrlList.txt to get the 4-digit numbers and then run markus on each student using Format 2.