BitterSuite Scheme Testing

This page describes hints for testing in Scheme.

Also see BitterSuiteScheme for a formal description of the language options.

Related Pages

This is a list of all pages on TWiki related to BitterSuite Scheme Testing.

• BitterSuiteTestingSchemeMutation
• BitterSuiteCountingFunctionCalls
• AppendExample
• SchemeModuleCaptureAndReportExceptions
• SchemeModuleCompareInexactValues
• SchemeModuleCompareUnknownStructs
• SchemeModuleCreateDataFromInput
• SchemeModuleCreateStudentDefinedStruct
• SchemeModuleDisallowSpecificFunctions
• SchemeModuleIgnoreOrderEqual
• SchemeModuleLimitUserTime
• SchemeRationalNumbers

Respect implementation languages

Student code should either be run in the module language (which will read the PLT Scheme headers (#lang or #reader) of the submitted files to determine the implementation language), or the specified language should be exactly what the students were told to use in the assignments specifications.

While it is clear that there is Intermediate Student code that is invalid in Beginning Student, the opposite is also true. For example, a valid BS but invalid IS student submission may contain something like

(define (time arg) "Ooops!")

Testing Mutation

To test mutation in Scheme, include a begin expression (and possible local definitions) in order to create the value to mutate (if necessary), call the function, and finish with the value to be tested, as in the example below.

(result (begin (local [(define n 5)] (set! n 1) n)))
(expected 1)

See BitterSuiteTestingSchemeMutation for more information.

Extra Information for the options.ss files

In options.ss, there are a number of available keys to specify, some of which are mandatory, some which are optional; see BitterSuiteScheme for details. The keys value, desc, timeout, memory and diff work as in the base code; see the man page for details.

For a typical assignment that is only testing Scheme, the main options.ss file, the options file will include language, and then may include other options such as modules that are to be loaded with most tested code or the value to be assigned to most of the tests. Some tutors also prefer to include self-documenting descriptions (although, in many cases, these should be overridden on a test-by-test basis). These settings will apply to all of subdirectories f in/, unless they are overridden inside one of the subfolders. For example:

(desc "Assignment 1 Tests")
(modules "imageteachpack.ss")
(language scheme/beginner)

If you are not changing any of the defaults, you do not need to put that line of code in options.ss.

Each of the questions also typically hasa its own options.ss file, which holds any information specific to one question. Very frequently this is where loadcode will be found. Any settings from the main options.ss file will be kept, unless something new is given. For example:

(desc "Question 1 - Cylinder Volume - Tests")
(value 2)
(loadcode "a1q1.ss")

Note that special care must be taken for questions involving mutation in Advanced Student or Module; see BitterSuiteTestingSchemeMutation.

leaf options.ss/test.ss file formatting

Either in a leaf directory, or by the time a leaf directory is reached, two lines of information typically need to be provided: the student's code you wish to test and the expected result. For example, if there was a question that expected a function called add4, which added four to a given number, an associated options.ss (or test.ss, which is acceptable, if you want Scheme testing to have the same “feel” as the other languages) file could look like this.

(result (add4 7))
(expected 11)

If equal? (or whatever equality function you are using) applied to (result (...)) and (expected ...) evaluates to true, the student passes the test get receives whatever value the question was worth. Otherwise, the student receives 0 for that particular test.

Note that expected is optional; if it is not present, then the standard output generated from that test will be compared against the model output, which is typically a cache created by RSTA of the standard output produced from the model solutions.

Once all your test.ss=/=options.ss files are created, the autotests are ready.

Modules

The ability to provide modules when testing code gives you a large amount of flexibility in how you mark students. They can even be used to provide alternatives to using equal? to check for equality between the student result and expected value! Care must be taken, however; if names chosen in the export list of the module conflict with any names provided by the students, the tester will fail to work correctly. Also, equality functions should never fail under any circumstances, meaning they must be quite paranoid about any appropriate type handling.

Any modules that are going to be used as part of a testing suite should be in the provided directory for that suite. The module must then be referenced in the (modules ...) option of options.ss either in the same directory as the test or in one of the parent directories.

General Use Modules

• Capture and report any raised exceptions
• Disallowing specific functions
• Limiting CPU time
• Creating test data from input files
• Creating instances of student-defined structures

Equality Overrides

The following is an example of how to crete your own equality function and how to use it with your tests.

In module language in Dr. Scheme, create a file with your equality function. For example, with a function that deals with inexact numbers (like pi), you will need to take extra care to handle floating point; see SchemeModuleCompareInexactValues.

Assume that the function is called check-close and is in a file called inexactcheck.ss. Save this file in the provided folder. For whichever question is using this equality function, insert the following code into the appropriate options.ss file:

(modules inexactcheck.ss)
(equal check-close)

In the scheme/intermediate-lambda language and higher, an alternate equal function can be defined with a lambda expression, rather than in a separate module, decreasing the likelihood of name collisions with student code. This lambda expression must be valid in the teaching language being used, though. For example,

(equal (lambda (expect result) (< (abs (- expect result)) 0.001)))

• Ignoring ordering in a list when comparing for equality
• Comparing inexact values
• Comparing unknown structs, using lines as an example

Coupling external tests to Scheme

There are occasionally tasks that cannot be accomplished with the Scheme language module itself that are still desired. These involve extra scripts; as a general rule, they should very rarely be written, undergo very little custom modification, and be verified very thoroughly to ensure correctness. It is very easy to overlook mistakes in these.

• Filtering Student Code
• Counting Function Calls
• BitterSuiteValidatingStudentFiles