Assignment Collection

The main goal for assignment collection is to:

- Isolate the assignments that students submit before the due date from the ones that are submitted after the due date. Students can still submit their assignments after the due date, but we do not want to run our marking scripts on those submissions.

- Run our marking scripts on the assignments and generate AUTOGRADING.ss and HANDMARKING.ss files for each student.

- Upload the HANDMARKING.ss file to MarkUs for TAs to mark and AUTOGRADING.ss file.

- Run the automated grading scripts for correctness tests/check-testcases

Contained in each AUTOGRADING.ss file is each student's correctness test result, their test coverage results, and HANDMARKING.ss contains all of the files that they've submitted. The HANDMARKING.ss file makes it really easy for TAs to grade each assignment, and it is a convenient place for students to receive their feedback. Generating this file will be the bulk of what we'll need to do during the assignment collection process.

The assignment collection process can be done anytime after the assignment is due, but you'll want to finish it before your first marking meeting so that TAs can get started on their marking load as soon as possible. Be sure to ask your ISC if you need any clarifications on the steps below!

Step 1 - Collecting Submissions on MarkUs:

Log onto MarkUs and go to the assignment that you want to collect. Go to the submissions tab for that assignment. Click on Collect All Submissions to prepare MarkUs for marking. When you click on Collect All Submissions, you'll be greeted with three different options for how you want to collect the assignments. Typically we have a 10 minute buffer window for any issues that MarkUs might have at the due date. Check with your ISC if there is a buffer window for assignment collection during your term! If there is a buffer window, you'll want to select "Mark the latest files that were submitted before: ..." and select the appropriate time (i.e. 2021-12-07 5:10 PM. This will collect all assignment submissions before or at 5:10PM on December 7th, 2021). If there is no buffer window, then you can just select "Mark the latest files that were submitted before the due date."

Once you've chosen how you want to collect the assignments, MarkUs will begin the collection process. Wait until all rows turn green. Once all rows have turned green, TAs could technically start marking each assignment in MarkUs, but we still need to generate the HANDMARKING.ss and AUTOGRADING.ss file to make marking easier, and to provide supplemental information to help with the marking process.

Step 2 - Downloading Submissions to Handin:

From this point onward we'll be working on the terminal/command line. Make sure that you have ssh'd into the course account at this point. Our next goal is to generate the AUTOGRADING.ss and HANDMARKING.ss file. In order to do that, we need to take the assignments that are in MarkUs 's repository and download them into the course account. To download all student submissions that were submitted by the due date into the course account, run the following command in the terminal (you can run this command anywhere in the course account; you don't need to be in any specific directory to run it).

Update (Fall 2021): there is a new command that should be used to download submissions to the course account. Simply run:

/u/isg/bin/markus.py download_marking ASSN SAVE_FOLDER

Where ASSN is the assignment you're collecting and SAVE_FOLDER is the location where you want to save the submissions. Typically we save the submissions under handin/ASSN_autotest. As an example,

/u/isg/bin/markus.py download_marking a01 ~/handin/a01_autotest/

This will download all collected student submissions for a01 into the handin/a01_autotest/ directory.

Please see the download_marking section of MarkUsScripts for more information about this command.

If you are using edX for steppers, be sure to review the Steppers Through edX page for information on how to download edX steppers before proceeding.

This is all you need to do for Step 2. At this point, before moving onto Step 3, you can skip to Step 5 and create and upload HANDMARKING.ss as instructed below. Since HANDMARKING.ss is not dependent on correctness test results, make_handmarking.sh can be run anytime after downloading the submissions. So the ideal step would be:

• Download Submissions to Handin
• Create and Upload HANDMARKING.ss
• Run Correctness Tests/Check-TestCases
• Create and Upload AUTOGRADING.ss

OUTDATED (since Fall 2020, kept just in case script reverts back)

/u/isg/bin/markus.py download ASSN SAVE_FOLDER

Where ASSN is the assignment you're collecting and SAVE_FOLDER is the location where you want to save the submissions. Typically we save the submissions under handin/ASSN_autotest. As an example,

/u/isg/bin/markus.py download a01 ~/handin/a01_autotest/

This will download all student submissions that were submitted by the due date for a01 into the handin/a01_autotest/ directory. If you used a buffer window in step 1, you'll have to make sure you download all submissions that were submitted within that buffer window. To do that, run

/u/isg/bin/markus.py download --time 'YYYY-MM-DD HH:MM:SS' ASSN SAVE_FOLDER

This will download all student submissions before or on the specified date and time. As an example,

/u/isg/bin/markus.py download --time '2019-09-16 21:10:00' a01 ~/handin/a01_autotest/

This will download all student submissions for a01 that were submitted before or at 9:10pm on September 16, 2019 into the handin/a01_autotest/ directory.

Step 3 - Running Correctness Tests

Once you have ran the download command and have finished downloading all of the submissions into the specified handin directory, it's time to do some processing on each student submission. First, we're going to run our correctness test on all of the submissions. The first thing you'll need to do is create a symlink from your testing folder to a new one that has _autotest appended onto it. To do this, cd into the directory where you created your correctness tests for the assignment. This should be the marking directory (i.e. cd ~/marking/). Once you're in the marking directory, do

ln -s ASSN ASSN_autotest

Where ASSN is the assignment you're collecting. For example,

ln -s a01 a01_autotest

Will create a new folder called a01_autotest inside of the marking directory which links to the a01 folder where you created correctness tests for a01. a01_autotest is like an alias for a01, so anything you do to a01_autotest (like deleting files inside of this directory), it will also happen inside of the a01 directory. They're basically the same directory. Why do we need to do this? We need a folder called a01_autotest in the marking directory, which has the same name as our handin folder, in order for our correctness tests to run.

Once you have created the symlink, we need to run a command to ensure every student has a folder in the handin/aXX directory. This is done to ensure students who did not submit any files, but completed the stepper questions, will still get marks. To do this, simply run the following command

/u/cs135/markus/bin/create_markus_repositories.sh ASSN ASSN_autotest

Where ASSN is the assignment you're collecting. For example,

/u/cs135/markus/bin/create_markus_repositories.sh a01 a01_autotest

Now it's time to run the correctness tests!

Before you run correctness tests though, let's run a command called tmux. It should simply suffice to type in tmux in the command line. Why should we run tmux? Because running correctness tests could take hours, and tmux allows the script to run uninterrupted. Some issues that you might run into if you do not use tmux would be having your ssh connection time out, you might accidentally close your terminal, or your computer might shut down. All of these issues will lead to you needing to rerun all of the correctness tests again, so it is crucial that you run tmux before running correctness tests on all students.

To run correctness tests, run

distrst -t t -s '*' ASSN_autotest TEST_SUITE SAVED_RESULTS

Where ASSN is the assignment that you're collecting, TEST_SUITE is the type of tests you want to run (can either be pt for public test, or 0 for correctness test), and SAVED_RESULTS will be the name of the folder where your correctness test results will be saved to. For example,

distrst -t t -s '*' a01_autotest 0 AUTOTESTRESULTS

This will run the correctness tests on the submissions located in the handin/a01_autotest/ using the marking/a01_autotest directory that we created earlier, using test suite 0 (for correctness tests) and saving the results inside a folder called test.0.AUTOTESTRESULTS (we typically use AUTOTESTRESULTS as the SAVED_RESULTS name) located inside of the marking/a01_autotest directory. At this point, multiple servers will be used to run correctness tests on students, all in parallel to make things faster. This might take a while depending on the number of students that you have during your term. While this is running, you should run the next step in parallel.

Although there should not be many occasions where the options such as -t and -s have to be changed, more options about rst and distrust can be found here.

Step 4 - Running Check-Testcases

Note: For this step, you should also run tmux to avoid the potential issues listed in step 3, since this step can also take hours to complete.

For assignments where the design recipe is required, we typically run some scripts to test students' test coverage (how well students' check-expects test their code). If you or other ISAs have written any check-testcases scripts earlier, now is the time to run them. Go into the check-testcases directory in the course account (cd ~/check-testcases/).

Once you're there, run

./run-all.sh ASSN 

Where ASSN is the assignment that you're collecting. This will run the check-testcases scripts that were made earlier on each student submission in the handin/ASSN_autotest/ directory. This might take a while depending on the number of students that you have during your term. Make sure you run this in parallel with the correctness tests (step 3) to be efficient! At this point you can go and do something else until steps 3 and 4 are done (sleep, netflix, league).

Step 5 - Making AUTOGRADING.ss and HANDMARKING.ss Files

At this point you have all of the files necessary to create the AUTOGRADING.ss and HANDMARKING.ss files for each student. To do this, run the make_autograding.sh and make_handmarking.sh files located in the ~/markus/bin/ folder:

~/markus/bin/make_autotesting.sh aXX
~/markus/bin/make_handmarking.sh aXX

Where aXX is the assignment you are collecting. For example,

~/markus/bin/make_autotesting.sh a09
~/markus/bin/make_handmarking.sh a09

This will generate the AUTOGRADING.ss and HANDMARKING.ss files for each student and save the file inside of each student's directory inside of the handin/aXX_autotest/ directory. The AUTOGRADING.ss file will contain their autocorrectness marks, while the HANDMARKING.ss file will contain their submissions. You should pick some random students inside of the handin directory and double check that their .ss files looks correct. If there are any mistakes in the correctness test or the test coverage scripts, do not rerun the make_handmarking.sh portion, otherwise the TA's annotations may be erased.

OUTDATED (since Fall 2021)

At this point you have all of the files necessary to create a GRADED_ASSIGNMENT.ss file for each student. To do this, open up the <nop>MakeGradedAssignment.py file located in ~/markus/bin. (vi ~/markus/bin/MakeGradedAssignment.py). Once the file is open, fill in the required fields inside of the file for the assignment that you're collecting. Once that's done, save the file and run it using:

python ~/markus/bin/MakeGradedAssignment.py

This will generate the GRADED_ASSIGNMENT.ss file for each student and save the file inside of each student's directory inside of the handin/ASSN_autotest/ directory. You should pick some random students inside of the handin directory and double check that the GRADED_ASSIGNMENT.ss file looks correct. You need to catch any mistakes in the correctness test or the test coverage scripts now before you send it off for TAs to mark, otherwise it'll most likely be too late to fix.

How to Use AUTOGRADING.ss and HANDMARKING.ss

As shown above, most of the time, you have to supply which assignment it is to run both of the scripts. However, the full usage of this script is

Usage: ./make_autotesting.sh [--student questid] [--base suffix] aXX
Usage: ./make_handmarking.sh aXX [-student questid]

Both --student and -student can be used when creating AUTOGRADING.ss and/or HANDMARKING.ss for an individual student. When these scripts are run for a single student, it does not automatically save the texts. Thus, for example,

./make_autotesting.sh --student userid a## >> ~/handin/a##_autotest/userid/AUTOGRADING.ss
./make_handmarking.sh a## -student userid >> ~/handin/a##_autotest/userid/HANDMARKING.ss

will save the text into appropriate files.

--base allows us to use OUTPUT.txt from different folders. As a default, it uses OUTPUT.txt from test.0.AUTOTESTRESULTS. When you provide the suffix, the script grabs OUTPUT.txt from test.0.GIVENSUFFIX.

Step 6 - Uploading HANDMARKING.ss and AUTOGRADING.ss to MarkUs

Once you have confirmed that the each of HANDMARKING.ss and AUTOGRADING.ss files look correct, you now need to upload each HANDMARKING.ss and AUTOGRADING.ss file into each student's MarkUs repository so that TAs and students can see the file on the MarkUs interface. All you need to do is run

/u/isg/bin/markus.py upload_marking --filter 'AUTOGRADING.ss' ASSN SAVE_FOLDER

and

/u/isg/bin/markus.py upload_marking --filter 'HANDMARKING.ss' ASSN SAVE_FOLDER

Where ASSN is the assignment you're collecting and SAVE_FOLDER is where you saved the assignment submissions in step 2. For example,

/u/isg/bin/markus.py upload_marking --filter 'AUTOGRADING.ss' a01 ~/handin/a01_autotest/

will upload all of the AUTOGRADING.ss files for each student from the handin/a01_autotest/ directory into MarkUs for a01. Once this has finished running, you should go back to MarkUs and double check that the AUTOGRADING.ss and/or HANDMARKING.ss files are present for a couple of students. Please see the upload_marking section for more information about this command.

Step 7 - Autograding Correctness Marks

At this point TAs could start marking, but we can be a bit nicer to them by automating the grading for correctness tests and coverage tests for each student. We'll first automate the correctness test marking. Go into the markus directory in the course account and go into the assignment folder that you're collecting (i.e. cd ~/markus/a01/). In this directory you can store any files that you'll be using to process things in MarkUs. One file that we'll want to create is a csv file that tells MarkUs how to assign rubric levels according to each student's correctness mark score. For example, creating a file called rubric_automarking.csv, we'll want to format this file as follows,

1,Q1: Correctness,1,3,4,5

bonus,Bonus:Correctness,,,,1

Where:

1. First column is the name of the question as it appears on the AUTOGRADING.ss file (i.e. Question 1: 5/5, the 1 is what you want to put into the first column)
2. Second column is the name of the criterion as it appears in MarkUs
3. Third column is the minimum mark needed to get Level 1.
4. Fourth column is the minimum mark needed to get Level 2.
5. Fifth column is the minimum mark needed to get Level 3.
6. Sixth column is the minimum mark needed to get Level 4.

Once you're done creating this file, process it with the following command.

/u/isg/bin/markus.py set_marks_rst [--rst-results-filename RST_RESULTS_FILENAME] ASSN TEST_RESULT_LOCATION rubric_converter.csv

Where RST_RESULTS_FILENAME is an optional parameter for the name of the file we're getting the results from (leave blank for OUTPUT.txt), ASSN is the assignment that we're collecting and TEST_RESULT_LOCATION is the location of where you saved the correctness test results. For example,

/u/isg/bin/markus.py set_marks_rst --rst-results-filename 'AUTOGRADING.ss' a01 ~/handin/a01_autotest/ a01_autograding.csv

Please see the set_marks_rst section of MarkUsScripts for more information about this command.

OUTDATED (since Fall 2021)

Once you're done creating this file, process it with the following command.

/u/isg/bin/markus.py set_marks_rst ASSN TEST_RESULT_LOCATION rubric_converter.csv

Where ASSN is the assignment that you're collecting and TEST_RESULT_LOCATION is the location of where you saved the correctness test results in step 3. For example,

/u/isg/bin/markus.py set_marks_rst a01 ~/marking/a01_autotest/test.0.AUTOTESTRESULTS a01_autograding.csv

will start the autograding process in MarkUs for a01. Once it's done, double check on MarkUs that the students have been assigned the correct rubric level according to their correctness test results. This step can be done anytime after step 3 is complete and can be done in parellel with step 8. Make sure that you go onto MarkUs and double-check that the correct rubric levels are being assigned. set_marks_rst section contains detailed information about this command if needed.

Step 8 - Autograding Check-Testcases Results

To automate the coverage test marking, first go to the test-cases directory for the assignment that you are collecting in the check-testcases directory. For example,

cd ~/check-testcases/a02/test-cases/

Once you're in the test-cases directory, you should see a csv file called output with a timestamp appended to it. This file was created after running step 4 and will be used to automate the grading on MarkUs. As an example, run this file by using the following command:

/u/isg/bin/markus.py set_marks_csv a02 output-20191126_22:47:51.csv

This will assign rubric levels on markus for all of the test coverage criterions. This step can be done anytime after step 4 is complete and can be done in parallel with step 7. Make sure that you go onto MarkUs and double-check that the correct rubric levels are being assigned. set_marks_csv section contains detailed information about this command if needed.

At this point, you should be completely done with the assignment collection process and TAs can now begin to grade assignments!

Outdated Assignment Collection

(Apr/29 update) Outdated! Use if necessary!

Follow steps 1-9 of the autotesting procedure: MarkUsGroupsSVNRepos#Auto_testing_Procedure. MarkUs scripts are in ~/markus/bin/.

Step 3: I always added 10 minutes to the due time as a buffer. Use test.0.AUTOTESTRESULTS.

Step 5: It's up to the instructor if they want you to use WxmeConverter. You might use it for the first few assignments. I didn't use it at all in my second term, but make sure students know to not introduce non-text into their files. Also, basic tests should catch this: stress the importance of checking basic tests and not leaving submissions until last minute! Moss can be run later. See following about the check-testcases script.

Step 6: Remember, we're using test suite 0, not 1.

You shouldn't need Step 10, since the checkout script should check out versions that are on time. Just be sure to mark submissions in ~/handin/aXX_autotest , where the checkout occurs. Those in ~/handin/aXX could be late.

check-testcases: You can use /u/cs135/archives/1161/assns/a09/tests/check-testcases/ as a starting point. This version will count up the number of examples and tests (number of tests INCLUDES examples). The run.sh file (run this after setting the assignment specifics to run the tests) is also easier to customize. With this version, you need 2 files for each student file you want to test: aXX[filename].rkt and aXX[filename]-base.rkt. Edit the assignment specifics in each as appropriate.

(pre Apr/29:

After everything is ready to go, run the following command,

~isg/bin/rst 3 p 0 �regex� timestamp

The option �p� stands for �print�. The �0� (zero) goes with the zero in testp.0.

�regex� should be replaced by a regular expression which will match all user IDs of the students whose submissions you want to run the tests against. For example, if you wish to run the marking script for all students with their userID starting with �a� through �c�, use the following command:

~isg/bin/rst 3 p 0 �[a-c]*� testRun

The timestamp can be anything and will be become part of the name of the folder which will store the outputs. Note that running rst twice with the same timestamp will keep results under the same directory. For example,

   ~isg/bin/rst 3 p 0 �nthibode� tutor
   ~isg/bin/rst 3 p 0 �zhli� tutor

will create folders named nthibode and zhli under marking/3/testp.0.tutor. These folders will not be overwritten if the same timestamps are used, you have to delete the old outputs if you want to run the tests against the same student again.

It is a good idea to test run the marking scripts for a few students/tutors before the assignment is due, just in case an unexpected error comes up. As soon as the assignment is due, execute the following command,

~isg/bin/rst 3 p 0 �*� firstRun New Note: Whether using the new scripts or running rst the �old fashioned way,� occasionally rst will segfault for no good reason (if anyone ever figures out why, you are my hero). This means that the student�s output will not have had all the tests run properly on him/her. To try and find any of these segfault cases, the easiest method is to use the command grep.

Navigate your way to the folder testp.0.timestamp and run the following command

grep �/0   Automarking� ./*/Autotesting

This will search every student�s autotesting output and look for any which contain the phrase written above. This is the list of student who rst will have to be rerun on. There are two things that need to be done before this. First, a file called MARKS will have been created in testp.0.timestamp. Open this file in a text editor and remove any of these students from the file (find is a very handy tool). Second, remove the student�s entire folder from testp.0.timestamp. Then you can rerun rst on the student.

)

(outdated: Send print job to Graphics Services

(note: this information is also included in the marking suite usage PDF)

After the auto-marking is complete, the assignments must be printed out for the TAs to grade the documentation and design. The following command,

~isg/bin/printOut 3 0 firstRun tut �g

will send a print job to Graphics Services that contains all Assignment 3 output files found in the ~/marking/3/testp.0.firstRun directory. The tut specifies how the assignments will be sorted. Here the assignments will be sorted into piles by tutorial sections. You could change tut to N to have the assignments in only N piles (where N is some number). The �g specifies that the assignments will be printed to Graphics Services. If instead you had a small pile to print, you could change this to �7, and instead the assignments would be printed to lp7 in DC3116. (Note: Do not print to DC3116 unless it is a very small number of assignments)

Make sure that you test the command before sending the output to graphics. The -o option will output to the screen the list of postscript files that are to be printed:

~isg/bin/printOut 3 0 firstRun tut �o

Make sure to allow at least 24 hours for the job to finish and for the papers to be delivered to the DC Copy Centre. You should be notified by email when the print jobs are completed.