RST/BitterSuite Testing Using Arbitrary Scripts

RST/BitterSuite is a software testing framework which can use scripts that you write for testing. This allows for a lot of flexibility. It can be used by running the scripts /u/isg/bin/rst or /u/isg/bin/distrst (More details on it here https://cs.uwaterloo.ca/twiki/view/ISG/RST). These scripts use a certain test setup/suite which you make for assignments. Once the test setup is made, these scripts can be run to use the setup to test student submissions. We will now explain how the setup should be made. The test setup explained here is the test suite that is referred to by the link above which explains how to run RST

To start with, you will need a directory called marking in your course account. It should already be there if your course has been using RST. Otherwise you can make one: mkdir /u/csXXX/marking . marking will contain all your automated tests (public and private) for each of your assignments. In marking you will need to make a directory for each assignment. If you are using RST, then most likely you are using MarkUS for submissions for your course as well. In this case the directory name for each assignment must be the same as the assignment repository name on MarkUS so that it knows where the tests are and RST can relay the public test information to MarkUS upon student submission. Say for example we are setting up RST for Assignment 1. Then if the MarkUS repository is named A1 then we will create the directory marking/A1 . In marking/A1 we will have two directories: test.pt for public tests and test.1 for private tests. These directories are structured pretty much the same.The difference is that the tests in test.pt are run automatically upon student submission to MarkUS so that the students can view their public test results when they submit, and the tests in test.1 are run after the due date of the assignment for testing student submission on secret tests. Note that it is important for the public test suite to be names test.pt as it needs to be run automatically upon student submission. For the private tests we usually use test.1 for consistency, and test.2 if ever a secondary test suite is needed, but it doesn't really matter as long as it is test.x where 'x' is anything but 'pt' as that is reserved for public tests.

Below is an example of how a setup would look like. Don't worry! We'll go through all the things below step by step.

runTests, computeMarks, options.rkt and the test.exe files are all scripts. There is also another script that we can have: computeMarks-postprocess which we will talk more about later. RST will run these scripts in this order:

1. test.pt/runTests
   • If test.pt/runTests doesn't exist, the default_runTests defined in /u/csNNN/.rstrc will be run. The default_runTests for most courses is /u/isg/bittersuite3/runTests.
2. test.pt/computeMarks
   • If test.pt/computeMarks doesn't exist, the default_computeMarks defined in /u/csNNN/.rstrc will be run. The default_computeMarks for most courses is /u/isg/bittersuite3/computeMarks.
   • If you want to run a custom computeMarks and the default one, you can put exec /u/isg/bittersuite3/computeMarks -q at the end of test.pt/computeMarks
3. If /u/isg/bittersuite3/computeMarks is run, then any test scripts in the test.pt/in/ folder will be run.
4. If /u/isg/bittersuite3/computeMarks is run, then test.pt/computeMarks-postprocess will be run.

Running tests from runTests

The runTests script is run in a testing account named csNNNt with a 't' at the end. This is safer than running the test script in the course account ( csNNN). After you create the runTests script, make sure it is executable using the chmod command. Since this script is run on a different account, you cannot directly access student submissions and your test setup. These are some of the many useful variables RST defines which you can use in runTests and your test.exe scripts:

1. testdir : This is a path to the test setup, i.e. test.pt . This is useful if you have some course provided files you need to use for testing student submissions. In that case you can make a provided_files directory with those files in test.pt and you can access them from runTests and test.exe with the path $testdir/provided_files
2. submitdir : This is the folder that contains the student submission that you would want to test. For example if the students have submitted a file called main.cpp, you can access it with the path $submitdir/main.cpp.
3. marksheet : The contents of this file are displayed to the students. So whatever you print to this file, students will be able to see in their test results. In the example below, runTests prints to the file results , and then the contents of results are printed to the file marksheet in computeMarks. By default this file is not saved, which for public tests it isn't usually an issue since it's automatically displayed to students once they submit on MarkUS, but it is possible to save it as discussed later in this document.

Here is an example test suite that checks if students submitted a C program that prints "Hello". In this example, students submit their code in a file named main.c.

• test.pt/
  • answers/ (empty folder)
  • runTests
  • computeMarks

#!/bin/bash

# The results.txt file will be printed to students in computeMarks
results="results.txt"

echo "~~~~ Assignment Test Results ~~~~~~~~~~~~~~~~~~~~~~~~" > $results

# Check if main.cpp was submitted
if [ -e "${submitdir}/main.cpp" ]; then
    echo "main.cpp was found" >> $results
    # Compile main.cpp
    timeout 5s gcc "${submitdir}/main.cpp" -o students_program &> gcc_output
    gcc_exit_code=$?
    if [ $gcc_exit_code  -eq 0 ]; then
        echo "Compilation succesful" >> $results
        # Run the student's program
        chmod +x students_program
        timeout 5s ./students_program > student_program.output
        # Check output of student's program
        echo "Hello World" | diff - student_program.output
        if [ $? -eq 0 ]; then
            echo "Public Test Passed!" >> $results
        else
            echo "Public Test Failed. Your program printed:" >> $results
            cat student_program.output >> $results
        fi
    else
        echo "Could not compile main.cpp (gcc quit with code $gcc_exit_code)" >> $results
        cat gcc_output >> $results
    fi
else
    echo "main.cpp was not submitted" >> $results
fi

#!/bin/bash

results="results.txt"
cat $results > $marksheet
exit 0

As you might have noticed above, we didn’t need the in directory and the computeMarks-postprocess in the test.pt directory at all for the public test! In fact most of the time just runTests and computeMarks suffice since public tests are not that complicated. However, if you do have multiple public tests and/or you want to assign marks to them, then we will need the in directory, which we will talk about more below.

Note: The empty answers folder does not really do anything. You just need it there because RST complains otherwise.

Running tests from computeMarks

After RST runs runTests, it will run computeMarks. The computeMarks script is run in the course account. This lets computeMarks access files in the course account, but it can also be dangerous (ex. if a student submits code that deletes files, and you run the student's code, it can delete files in the course account).

Examples where you can use computeMarks for testing are:

• You want to check if the students have registered their clicker ID on the course website
• You want to check if the students have completed tracing exercises on the course website

Generally you do want to avoid using this script for testing. But it is useful for things like in the previous example: printing to marksheet.

Running tests from test.pt/in/ folder

After RST runs computeMarks, it will run your tests in the in folder. You can use the "External" language to make RST run scripts in the in folder. Name your scripts test.exe. The test.exe scripts will run as csNNNt testing account.

If you have a test.pt/computeMarks script, you need to run /u/isg/bittersuite3/computeMarks to process the in/ folder. Do this by putting exec /u/isg/bittersuite3/computeMarks -q at the end of your test.pt/computeMarks script.

The options.rkt file specifies which language is being used for testing. The "external" means that it's unspecified. For eg. if it was racket then instead of having test.exe shell scripts, you could straight up have racket scripts instead for each test. The "(value 1)" is for specifying how many marks the tests are out of. If for each test directory in in/1 you have an options.rkt file specifying the value of each test, then it will override any options.rkt file in the outer directories. But if all the tests have the same value then you can just have one options.rkt in the outer directory and that applies the value to all of them like in the example below.

The config.ss file is optional. You might not want it. It is useful in the case you want to use the https://cs.uwaterloo.ca/twiki/view/ISG/MarkUsScripts#set_marks_rst_sub_command script to upload marks to MarkUS as it helps print out test output in a way the script can recognise. However, there are other ways to upload marks to MarkUS if you want (Check https://cs.uwaterloo.ca/twiki/view/ISG/MarkUsScripts#MarkUs_Scripts . This is a very useful set of scripts and an essential read if you're using MarkUS).

Example 1: The following test suite checks that students submitted a C program that counts characters read in from the keyboard.

• test.pt/
  • answers/ (empty folder)
  • runTests
  • computeMarks
  • config.ss
  • in/
    • 1/
      • options.rkt
      • t01/
        • test.exe
      • t02/
        • test.exe

#!/bin/bash

# The results.txt file will be printed to students in computeMarks
results="results.txt"

echo "~~~~ Assignment Test Results ~~~~~~~~~~~~~~~~~~~~~~~~" > $results

# Check if main.cpp was submitted
if [ -e "${submitdir}/main.cpp" ]; then
    echo "main.cpp was found" >> $results
    # Compile main.cpp
    timeout 5s gcc "${submitdir}/main.cpp" -o students_program &> gcc_output
    gcc_exit_code=$?
    if [ $gcc_exit_code  -eq 0 ]; then
        echo "Compilation succesful" >> $results
        chmod +x students_program
    else
        echo "Could not compile main.cpp (gcc quit with code $gcc_exit_code)" >> $results
        cat gcc_output >> $results
    fi
else
    echo "main.cpp was not submitted" >> $results
fi

#!/bin/bash

results="results.txt"
cat $results > $marksheet
exec /u/isg/bittersuite3/computeMarks -q

(print-by-question true)

(language external)
(value 1)

#!/bin/bash

if [ ! -e students_program ]; then
    echo "Compilation failed." >&4
    echo '0' >&3
    exit 0
fi

# Run student's program
chmod +x students_program
echo -n "Test Input Case 1" | timeout 3s ./students_program > students_program.out

# Check output
echo "17" | diff - students_program.out
if [ $? -eq 0 ]; then
    # Test passed
    echo "The test passed." >&4
    echo '100' >&3
else
    # Test failed
    echo "The test failed." >&4
    echo '0' >&3
fi

#!/bin/bash

if [ ! -e students_program ]; then
    echo "Compilation failed." >&4
    echo '0' >&3
    exit 0
fi

# Run student's program
chmod +x students_program
echo -n "" | timeout 3s ./students_program > students_program.out

# Check output
echo "0" | diff - students_program.out
if [ $? -eq 0 ]; then
    # Test passed
    echo "The test passed." >&4
    echo '100' >&3
else
    # Test failed
    echo "The test failed." >&4
    echo '0' >&3
fi

Example 2: This example shows one way to check files in the course account (computeMarks and question 1's test.exe files). Question 2 checks that the student has set up personal website. Zip file:

• test.pt/
  • answers/
  • computeMarks
  • config.ss
  • in/
    • 1/
      • options.rkt
      • part_a/
        • options.rkt
        • test.exe
      • part_b/
        • options.rkt
        • test.exe
      • part_c/
        • options.rkt
        • test.exe
      • part_d/
        • options.rkt
        • test.exe
      • part_e/
        • options.rkt
        • test.exe
    • 2/
      • options.rkt
      • t01/
        • test.exe

Timeouts in options.rkt

By default, the test.exe script is run with a timeout of 15 seconds. After 15 seconds, the test.exe script will be killed/stopped, and the test case will fail (receive 0 marks) with the message "Test timed out after 15 seconds."

You can set a different timeout of N seconds by entering (timeout N) in options.rkt.

If you don't want a timeout at all, you can enter (timeout 0). Then it becomes your responsibility to esure test.exe always terminate.

Clean Up Work in test.pt/computeMarks-postprocess

If there's anything you need to do before RST finishes, you can create a script called computeMarks-postprocess and RST will run it at the end before quitting. The computeMarks-postprocess script is run by /u/isg/bittersuite3/computeMarks.

Environment Variables

The runTests, computeMarks, and test.exe scripts receive information from RST through environment variables.

Here are some of the more useful environment variables (Here the course is cs115ae, assignment is a02 , and the question is 1):

Variable Name	Available In	Description	Example
testdir	runTests computeMarks test.exe	Path to the test suite in the course account	/u1/cs115ae/marking/a02/test.pt
submitdir	runTests computeMarks test.exe	Path to folder containing the student's submitted files	/tmp/.cs115ae.a02.t.pt.ubuntu1804-00243.0829164511/20143lrssb6/20271FL4oip
student_questid	runTests computeMarks test.exe	Student's Quest ID	yc2lee
assign	runTests computeMarks test.exe	Assignment number	a02
course	runTests computeMarks test.exe	Course account name	cs115ae
test_summary	computeMarks	Stuff written to this file will be displayed in MarkUs is the submissions table	/tmp/tmp.jwmZERUzom
marksheet	computeMarks	Stuff written to this file will be included in the test results (OUTPUT.txt)	/tmp/.cs115ae.a02.t.pt.t01/19833kIwjHU/19988jpgyog/marksheet
external_test_path	test.exe	Path to the test.exe	/u1/cs115ae/marking/a02/test.ext/in/1/t01/test.exe
internal_test_id	test.exe	The test ID (question_testname)	1_t01
tmpdir	runTests computeMarks test.exe	Path to a temporary directory that RST creates, and which test scripts are run from	/tmp/.cs115ae.a02.t.pt.t01/yc2lee/7664OTxI8A

Saving Output

Output and files created by test scripts usually are not saved. One way to get around this is to use the -t c option when running rst, for example:

rst -t c -s yc2lee a02 pt

This will save all the files that RST creates while running tests.

From runTests

Standard output from runTests is not saved in the test results, but can be useful for debugging.

To save output or files created by runTests: In computeMarks, use the keepFile command to include the file in the test results folder. For example, if your runTests creates a file called "runTests_output.txt", in your computeMarks script, do:

keepFile runTests_output.txt -o

From computeMarks

Same thing as runTests.

From test.exe

Standard output from test.exe is automatically saved in the test results folder.

To save files created by test.exe, use the keepFile command.

Additional Ideas and Scripts To Make Testing Easier

Sometimes all the test.exe files may seem repetitive. For example say the students are asked to submit a module called “sorting.cpp” which asks them to define functions to sort an array according to the function declarations in the “sorting.h” file which is provided by the assignment. Then say for each test you have a C++ program to test the functions in their module called “t01.cpp”, “t02.cpp”, “t03.cpp”. Now basically all that your “test.exe” files are going to be doing is compiling “sorting.cpp” and “txx.cpp” together with the provided “sorting.h” available in the same directory and running the compiled program to determine if the test passed or failed. This logic seems a bit redundant to write out again and again. Turns out there is a workaround for this.

Imagine having just one parent script that implements the logic shared by all the test.exe scripts and having the actual test.exe scripts just be symbolic links to that one parent script: 1) test.pt/maintest.exe : The parent script which has the logic coded for testing 2) All test.exe in each test directory in/1/tXX/ are symbolic links to test.pt/maintest.exe

But now we face the problem that for each test we are compiling we need a different testing program, so when test.pt/maintest.exe is run for each test, it needs to know what test it is running. Here some environment variables defined by RST come in handy. The variable internal_test_id stores the question name and the test directory name of the currently running test. So for example if you run the test “t01” for question 1, then $internal_test_id = 1_t01. This is exactly what we want! So in our maintest.exe script we can have:

source_file="sorting.cpp"
source_name="sorting"
regular_timeout="60s"

####################################################################

submitted_files=${submitdir}
question=$(echo $internal_test_id | cut -d '_' -f 1)
testname=$(echo $internal_test_id | cut -d '_' -f 2)
results="${testname}_testing_output.txt"
TIMEOUT=$regular_timeout

# Make $results file empty
> "$results"
if [ -e "${submitdir}/${source_file}" ]; then
   mkdir temp_folder
############################# COPY FILES WHICH NEED COMPILING TO temp_folder  ###########
# submitdir contains all student submitted files
# testdir conatins a copy of the test setup, including provided files
   # Copy the student's submission to temp_folder
   cp ${submitdir}/* temp_folder/
   # Copy all the provided files for question to temp_folder
   cp ${testdir}/provided_files/${question}/${testname}.cpp temp_folder/
   cp ${testdir}/provided_files/${question}/${source_name}.h temp_folder/
#########################################################################################
   cd temp_folder/

   g++-7 -std=c++17 -DTESTING *.cpp -o run 2>>../$results >> ../$results 
   if [ -f run ]; then
           chmod +x run
                 timeout $TIMEOUT ./run &>> student.out
           if [ $? -eq 124 ]; then
                   echo "Your compiled program timed out." >> ../$results
                        echo '0' >&3
             else
                        cat student.out >> ../$results
                if grep -Fxq "Test Passed" student.out then # or whatever logic you have to determine if test passed
                
                      echo "Test $testname passed 1/1" | tee -a ../$results >&4
                                echo '100' >&3
                   else
                      echo "Test $testname failed 0/1" | tee -a ../$results >&4
                      echo '0' >&3
                   fi
             fi

   else
           echo "Compilation failed!" | tee -a ../$results >&4
           echo '0' >&3
   fi
cd ../
rm -r temp_folder
else
   echo "Not compiling $source_file as it was not found." | tee -a $results >&4
   echo '0' >&3
   exit
fi

And now the “test.exe” scripts work as expected (given the test programs are saved in the provided_files directory as referred to above in the script)! This will work if your test names and test directory names are the same. But now we have a the problem of what if each test has different weight/value. Say t01 is out of 1 mark but t02 is out of 3 marks? What we can do here is, name each testing directory txx_x where “x” is the mark the test is out of. so for t01 it would be instead t01_1 and t02 would be changed to t02_3 . So now in maintest.exe you can add:

testweight=$(echo $internal_test_id | cut -d '_' -f 3)

This gives you the weight of the test being run. Remember though that the options.rkt files are still required for each test! You can adapt the above script to whatever fits your purposes the most. Eg If the students are submitting a program then you can make a similar iotest.exe such that your input and expect files for each test, are named the same as your test directory name like above. (An example of this will be added later)

#!/usr/bin/python3
# ^^^^^^^^^^^^^^^^ In this example we use Python, but you can
# use another language by changing the first line.


# In this example test suite, we want to check the responses of a student X. For
# illustration, we'll assume that the responses are stored in the CSV file
# /u/csNNN/student_responses/X.csv and that the CSV file looks something like this:
#
# Question,Mark,Total
# A1Q1a,0,1
# A1Q1b,4,5
# A1Q1c,2,5
#
# We also want the results printed along with any other coding questions,
# which would be checked by files inside the "in" folder.
#
# Our approch will be to process the CSV file in computeMarks, and save the mark in simple
# files that test.exe (inside the "in" folder) will read, and pass the mark to RST.
#
# Another approach is to do the work in test.exe instead of here. However, the test.exe
# scripts run as user csNNNt, not csNNN, and so they cannot read files owned by csNNN.
# We would have to make sure the files are created with permissions for csNNNt, or create
# a copy of the file for csNNNt to read.

import os, sys, csv

# Open the CSV file and store all the marks into a Dict
marks = {}
try:
    responses_file = "/u/{}/student_responses/{}.csv".format(os.environ['course'], os.environ['student_questid'])
    with open(responses_file, 'r') as f:
        reader = csv.DictReader(f, delimiter=',')
        for row in reader:
            marks[row['Question']] = (row['Mark'], row['Total'])
except Exception as err:
    # Ignore any errors that occur - marks Dict will remain empty and the student
    # will get 0, and hopefully report the problem to us. Errors printed are only
    # visible when test is run manually from command line.
    print(err)

# For each required question, lookup the grade and convert to a percentage
for question in ["A1Q1a", "A1Q1b", "A1Q1c", "A1Q1d", "A1Q1e"]:
    percentage_mark = None
    if question not in marks:
        percentage_mark = 0
    else:
        try:
            percentage_mark = 100 * int(marks[question][0]) / int(marks[question][1])
        except Exception as err:
            print(err)
            percentage_mark = None

    # To pass the mark to test.exe later
    if percentage_mark != None:
        try:
            with open("student_mark_for_{}".format(question), 'w') as f:
                f.write(str(percentage_mark))
        except Exception as err:
            print(err)

# Python throws away output when we do exec. Flush the buffer
# so we can see if any debugging stuff got printed.
sys.stdout.flush()

# This will tell RST to process the test cases inside the "in" folder
os.execl("/u/isg/bittersuite3-dev/computeMarks", "-q")

(print-by-question true)

(language external)
(value 1)

(args "A1Q1a")

#!/usr/bin/python3

import os,sys

mark_writer = os.fdopen(3, 'w')
message_writer = os.fdopen(4, 'w')

try:
    # argv[1] is set by the options.rkt in the test case folder
    question = sys.argv[1]
    with open("student_mark_for_{}".format(question)) as f:
        mark = f.read().strip()
        mark_writer.write(mark)
        message_writer.write("Your mark is {}%".format(mark))
except Exception as err:
    mark_writer.write("0")
    message_writer.write("Sorry, we could not retrieve your mark: {}".format(str(err)))

(args "A1Q1b")

#!/usr/bin/python3

import os,sys

mark_writer = os.fdopen(3, 'w')
message_writer = os.fdopen(4, 'w')

try:
    # argv[1] is set by the options.rkt in the test case folder
    question = sys.argv[1]
    with open("student_mark_for_{}".format(question)) as f:
        mark = f.read().strip()
        mark_writer.write(mark)
        message_writer.write("Your mark is {}%".format(mark))
except Exception as err:
    mark_writer.write("0")
    message_writer.write("Sorry, we could not retrieve your mark: {}".format(str(err)))

(args "A1Q1c")

#!/usr/bin/python3

import os,sys

mark_writer = os.fdopen(3, 'w')
message_writer = os.fdopen(4, 'w')

try:
    # argv[1] is set by the options.rkt in the test case folder
    question = sys.argv[1]
    with open("student_mark_for_{}".format(question)) as f:
        mark = f.read().strip()
        mark_writer.write(mark)
        message_writer.write("Your mark is {}%".format(mark))
except Exception as err:
    mark_writer.write("0")
    message_writer.write("Sorry, we could not retrieve your mark: {}".format(str(err)))

(args "A1Q1d")

#!/usr/bin/python3

import os,sys

mark_writer = os.fdopen(3, 'w')
message_writer = os.fdopen(4, 'w')

try:
    # argv[1] is set by the options.rkt in the test case folder
    question = sys.argv[1]
    with open("student_mark_for_{}".format(question)) as f:
        mark = f.read().strip()
        mark_writer.write(mark)
        message_writer.write("Your mark is {}%".format(mark))
except Exception as err:
    mark_writer.write("0")
    message_writer.write("Sorry, we could not retrieve your mark: {}".format(str(err)))

(args "A1Q1e")

#!/usr/bin/python3

import os,sys

mark_writer = os.fdopen(3, 'w')
message_writer = os.fdopen(4, 'w')

try:
    # argv[1] is set by the options.rkt in the test case folder
    question = sys.argv[1]
    with open("student_mark_for_{}".format(question)) as f:
        mark = f.read().strip()
        mark_writer.write(mark)
        message_writer.write("Your mark is {}%".format(mark))
except Exception as err:
    mark_writer.write("0")
    message_writer.write("Sorry, we could not retrieve your mark: {}".format(str(err)))

(language external)
(value 1)

#!/usr/bin/python3

import os, requests

mark_writer = os.fdopen(3, 'w')
message_writer = os.fdopen(4, 'w')

student_questid = os.environ['student_questid']
try:
    r = requests.get("https://student.cs.uwaterloo.ca/~{}".format(student_questid), timeout=3)
    r.raise_for_status()
    mark_writer.write("100")
except Exception as err:
    mark_writer.write("0")
    message_writer.write("Web page did not return OK 200 status, or some other problem occurred")