Flexible Python Automated Testing with RST

The problem with test.py

The test.py method of Python automated testing described at BitterSuitePythonTesting is sufficient for simple assignments. However, when the test.py file is run, the student's Python program has already been loaded, so it gives no control over how the student's Python program is evaluated. It's also hard to run code before running the student's program. The test.py files are not good for automatically testing certain Python assignments.

This page contains an example of how to do Python automated testing when test.py doesn't work well.

Our fake Python assignment

Question 1

Create a Python file alternate.py that defines a function called alternate. The function alternate consumes a Str and returns a Str obtained by alternating the casing of all letters (a-z) in the consumed Str. The first letter of the returned Str should be capital.

Example: alternate("hello! How Are You? Doing OK") should return "HeLlO! hOw ArE yOu? DoInG oK"

Question 2

Create a Python script alternate_keyboard.py that when run, reads lines from the keyboard (standard input) until EOF is reached. The script should print out each line it has read, but with the cases alternating. Each line should be treated independently, ie apply the alternate function from Q1 on each line that's read.


$ cat input.txt
hello! How Are You? Doing OK
this is my example!!!
I hope you do Well!!
$ python3 alternate-driver.py < input.txt
HeLlO! hOw ArE yOu? DoInG oK
ThIs Is My ExAmPlE!!!
I hOpE yOu Do WeLl!!

Question 3

Create a Python script alternate_keyboard_save.py that consumes one optional command line argument. The script behaves the same as alternate-keyboard.py, except that it saves the output to a file named output.txt in the current directory if no command line argument was provided. If a command line argument was provided, use it as the path to the output file.


$ cat input.txt
hello! How Are You? Doing OK
this is my example!!!
I hope you do Well!!
$ python3 alternate-driver.py < input.txt
$ cat output.txt
HeLlO! hOw ArE yOu? DoInG oK"
ThIs Is My ExAmPlE!!!
I hOpE yOu Do WeLl!!
$ python3 alternate-driver.py /tmp/my-output.txt < input.txt
$ cat /tmp/my-output.txt
HeLlO! hOw ArE yOu? DoInG oK"
ThIs Is My ExAmPlE!!!
I hOpE yOu Do WeLl!!

Sample solutions

Question 1 - alternate.py

import check

def alternate(mystr):
    answer = ""
    makeNextLetterUppercase = True
    for i in range(len(mystr)):
        char = mystr[i]
        if char.isalpha():
            if makeNextLetterUppercase:
                answer += char.upper()
                answer += char.lower()
            makeNextLetterUppercase = not makeNextLetterUppercase
            answer += char
    return answer

# Tests
if __name__ == "__main__":
    check.expect("empty", alternate(""), "")
    check.expect("example", alternate("hello! How Are You? Doing OK"),\
        "HeLlO! hOw ArE yOu? DoInG oK")
    check.expect("test1", alternate("this is my example!!!"),\
        "ThIs Is My ExAmPlE!!!")
    check.expect("test2", alternate("I hope you do Well!!"),\
        "I hOpE yOu Do WeLl!!")

Question 2 - alternate_keyboard.py

import sys
from alternate import alternate

for line in sys.stdin:
    print(alternate(line), end='')

Question 3 - alternate_keyboard_save.py

import sys

output_file_path = "output.txt"

if len(sys.argv) >= 2:
    output_file_path = sys.argv[1]

sys.stdout = open(output_file_path, "w")
import alternate_keyboard

Test cases the instructor wants to check

Students should be able to get credit for Q2 and Q3 even if they haven't done the previous questions.

Question 1

Check the test cases in the sample solutions.

Question 2

In the testing, use Q1 sample solutions in case the student couldn't do Q1.

Use the test input in the question, and also this one:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua. Tortor
aliquam nulla facilisi cras fermentum. Venenatis urna cursus eget
nunc scelerisque. Urna id volutpat lacus laoreet non. Dolor purus
non enim praesent elementum. Dui nunc mattis enim ut tellus eleme.

Question 3

In the testing, use Q1 and Q2 sample solutions in case the student couldn't do the previous questions.

Same test cases as in Q2. First test should not pass any command-line arguments. The second test should use a command-line argument.

Our test.pt (or test.0 or test.1) test suite

To keep things organized, we will create a brand new folder for each test case. We'll copy all the files we need into that folder and work from there.

Please keep in mind that providing solutions in test suites is a security risk. Even using pyc files is useless because pyc files can be easily decompiled.

A few notes about the test.exe environment (more details at BitterSuiteExternal):

  • The test.exe is ran as csNNNt account from a temporary folder that's something like /tmp/.csNNN.a01.t.pt.t01/userid/66923sDZOaI
  • The submitdir environment variable is path to a folder containing the student's submissions (it is NOT the /u/csNNN/handin/aXX/userid folder).
  • The testdir environment variable is path to test suite in marking folder, ex /u/csNNN/marking/aXX/test.pt
  • Anything printed to standard output will be available in the same directory as OUTPUT.txt
  • If you create a file in test.exe that you want to keep, use KeepFile
  • Set the test case score using echo 100 >&3 where 100 is the percentage that the student gets.
  • Set the test case feedback using echo message >&4 where message is what you want to say to the student.

Download (attached below): https://cs.uwaterloo.ca/twiki/pub/ISG/BitterSuiteExternalPythonTesting/sample_test_suite.zip

Topic attachments
I Attachment History Action Size Date Who Comment
Compressed Zip archivezip sample_test_suite.zip r2 r1 manage 13.8 K 2019-08-06 - 02:16 YiLee Sample test suite
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r5 - 2021-09-09 - YiLee
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback