Stepper Questions

This page contains information about how to set up stepper questions from edX or the course website for autotesting purpose.

Steppers Through edX

To set up steppers in the course account for correctness tests, it should be treated very similarly to regular tests. Within the "in" directory of test.0 for a particular assignment, you will want to create a single directory for stepper questions if, on the assignment, they are all weighted the same. If some are weighted differently than others, than a directory should be made to group/categorize these differently weighted stepper questions.

Within each of these folders, there should be two things: an options.rkt file and a Stepper folder that will contain a test.exe script. The set-up is very similar to a regular test set-up, with the only difference being for steppers we use a test.exe script instead of a test.rkt file. Within the options.rkt file, you will want to specify only a few things as follows:

(language external)

(desc "Stepping Problem Results")

(value 4)

Note: The value here can be changed from 4 to the number of steppers being checked for that specific Stepper folder.

In order to gather the data used to grade the steppers, an ISA must go to edX and from the main menu follow 'Instructor' > 'Data Download' and from here the ISA should look for the 'CS135 Grade Download' subtitle with a 'Download Marks' button underneath it and click that. The ISA will then be prompted to fill in some information on the section, subsection and time for the data to be downloaded from. For the 'Section' an ISA will want to select 'Assignments', for the 'Subsection' choose the specific assignment that is currently being set up and for the 'Due Date' the date and time 100 minutes after the due date of the specified assignment should be chosen. Once these things have been selected, the ISA can hit submit and a download will begin for a csv that carries the data for the steppers needed to grade them.

Once the ISA has the csv, you will want to rename it to follow a naming format of 'aXX_autotest.csv' if all steppers have the same weight. If there are different steppers with different weights then the csv file name does not need to follow a naming format, but that file name needs to be specified as part of the $logfile variable in the Stepper test.exe script.

Usually it looks like 'my $logfile = "/u/cs135/course/edx_stepping/$assignment.csv";' however it should be modified such that '$assignment' is replaced with the actual name of the csv file you have.

In the special case that you have your steppers being taken care of in separate folders due to differing weights, you will also want to edit your csv to contain only the columns containing data for questions that are being marked in that question. So for example if you had part a, c, and d being marked under one stepper folder, then in your csv you would want to remove any columns containing data for question parts that are not related to a, c, or d. This also means, more often than not, in these scenarios you will be using multiple csv files to cover all the different question parts based on their different weightings and, by consequence, multiple question folders in your "in" folder that will differ from each other by the csv specified in their corresponding test.exe script and the value given in their options.rkt file.

The csv should then be placed in /u/cs135/course/edx_stepping/ and once placed you will want to make sure you check a few things. By typing 'ls -l' into the terminal while in the edx_stepping directory you will be able to see the permissions of all the files, but more importantly the csv that you just placed. Issues will occur if the csv is not given proper permissions, so by executing the 'fixallperms.sh' script, permissions for all files in the directory should be appropriately adjusted. To double check, you will want to make sure the csv is part of the cs135t group, and has both read and executable permissions.

Test.exe Details

This script is written in Perl and designed to not need any modifications during normal use. The script looks for a log file in CSV format in the following location:

/u/cs135/course/edx_stepping/aXX_autotest.csv

Where aXX is the assignment such as a04, thus a typical file might be called a04_autotest.csv.

Note that this file must be part of the cs135t group. After downloading the file from edX and moving it into the above location with the correct name, you need to change the group using the following command:

chgrp cs135t /u/cs135/course/edx_stepping/aXX_autotest.csv

WARNING: If you do not change the group or if the file is not named correctly then the Stepper questions for this assignment will not be graded correctly and you'll have to re-run Correctness Tests (from CollectAssignments).

The CSV format for the log file (produced by edX) looks like this:

id,account,email,Stepping Q1,Stepping Q2,Stepping Q3,Stepping Q4,Stepping Q5,...

...

4069,t5studen,t5student@uwaterloo.ca,1.0,1.0,3.0,3.0,...

...

Note that there are as many columns as there are stepping questions on edX. The test.exe script automatically handles all of the columns.

This script needs very minimal adjustments for use in each assignment, if any at all. The only part of this script that anyone should be touching is the $logfile variable, and that should only be done if the csv file name needs to be adjusted.

Steppers Through Course Website

(Dec 23/ update)

The options.rkt file for a stepping problem has the following format.

(language external)

(desc "Stepping Problem Results")

(value 4)

The value macro needs to be changed to match the number of steppers that will be graded on the assignment. Instead of a numbered directory under the question, there will be a directory called Stepper. Inside this directory should be a file test.exe. There are three lines in this file that will need to be changed. The first one will me similar to the following.

my $duetime = 1511316000;

The number should be set to the time that the assignment is due in Unix epoch time. You can use the date command as shown in the file to get this (either something like date +%s -d "Oct 16 21:00:00 EDT 2018" or date +%s -d "Dec 16 21:00:00 EST 2018" depending on whether daylight savings is active). If it still does not work try to remove the option that specifies the time zone or just look up the Unix epoch time online.

The second line to change looks like the following.

next unless $F[3] =~ /^(m02a_expressions_(05|10|15)_required$/;

The right side of the equal sign needs to be updated to search for the correct regex using the stepper results inside of the course directory. You can go into course/stepper and open up your own student file to see what the regex expression should look like for each stepper question (make sure you complete the steppers first so that there are records on your student file of you completing the stepper question!). If you want to see how the regex should be formatted, you can see past Stepper/test.exe file in the archive for term Fall 2020.

Finally, this line is just for aesthetic purposes:

print THREE (($#plist + 1) * 100), "/4\n";

The "/4\n" should correspond to how many steppers the question is "out of" - in this case, we had a-d so it was out of 4. You should update this number (here is 4) to match the number of steppers that will be graded on the assignment.

(pre Dec 23

The options.rkt file for a stepping problem should not need to be changed in any way. It has the following format.

(language external)

(desc "Stepping Problem Results")

(value 4)

Instead of a numbered directory under the question, there will be a directory called Stepper. Inside this directory should be a file test.exe. There are three lines in this file that will need to be changed. The first one will me similar to the following.

my $duetime = 1511316000;

The number should be set to the time that the assignment is due in Unix epoch time. You can use the date command as shown in the file to get this (either something like date +%s -d "Oct 16 21:00:00 EDT 2018" or date +%s -d "Dec 16 21:00:00 EST 2018" depending on whether daylight savings is active). If it still does not work try to remove the option that specifies the time zone or just look up the Unix epoch time online.

The second line to change looks like the following.

next unless $F[3] =~ /^assignment8q1([a-d])$/;

The assignment should be changed to the correct number and the correct question. The [a-d] part should have bounds that represents the number of stepper problems on the assignment. In this example, there are four stepper problems on assignment eight, question one.

Finally, this line is just for aesthetic purposes:

print THREE (($#plist + 1) * 100), "/4\n";

The "/4\n" should correspond to how many steppers the question is "out of" - in this case, we had a-d so it was out of 4.)

-- Yuying Li - 2020-12-23

General Steppers Setup Tips

1. If all the stepper questions have the same weight, you can only create one single test suite with a value equal to the number of steppers.

2. If not every stepper has the same weight, you need to create a test suite for tests that have the same weight. The value in options.rkt is the number of steppers that have the same weight.

• eg: If there are 2 steppers with weight 2 and 3 steppers with weight 3, you need to create one test suite called 1ab or something else with value 2 and one test suite called 1cde with value 3.

Comments