BitterSuite Scheme Language Documentation

Scheme in BitterSuite has a relatively complex back-end, as it attempts to support a number of languages (the HtDP languages and PLT Scheme's module language), and attempts to make use of the scheme/sandbox library in a way that is both reasonably efficient and works with our testing system.

Also see BitterSuiteSchemeTesting for tips and tricks using the options described here.

Related Pages

• BitterSuiteSchemeTesting
• BitterSuiteFilteringStudentCode

Relevant Options

The basic language is specified by (language scheme), but this does not allow a usable default for the loadcode option. A particular default language can be specified by (language scheme/sublanguage), where sublanguage is one of beginner, beginner-abbr, intermediate, intermediate-lambda, advanced, and module. It is extremely important to specify in advance a particular language that is being used for a given assignment (or, if desired, a given question); while there is clearly intermediate-level code that is not valid in beginner, the reverse is also true. While the module language will load an HtDP submission properly in whatever language is specified by the student, this puts things like teachpack specification directly in the hands of the student and takes control of the particular language being used away from course staff. This can lead to problems; for example, the student can include a teachpack that the testing suite does not support, or which is supported but potentially trivializes a question.

(loadcode file)

Creates an evaluator for the code submitted in file in the default language. Note that only one evaluator can be active at a time at any particular level of the hierarchy. It is created in a context established by previous options; so, for example, the values of modules must already be specified before this option is read. The values of timeout and memory also apply to the code loading. There are two types of loading that are done. In the intermediate and advanced families of languages, an evaluator is created by loading S-expressions from the student file one-by-one. This means that if any single S-expression causes a fatal error, this will be skipped over and the rest of the file will still be attempted. This will keep something like a student-supplied test that raises an error from preventing any code from being loaded. However, this is not possible in the beginner languages or module; in these, the entire file must be loaded at once, and any load error will mean the entire code base does not load. To accomodate potential timeout issues, the value of timeout is quadrupled for the code loading.

(loadcode language file)

As with the single-valued loadcode, but creates an evaluator for the code submitted in file in the language language, where this language is one of the sublanguages listed above.

(evaluator-reset frequency)

Valid options for frequency are never, before-test, before-expression, and standalone-program. The default is never, which means that after evaluator creation the contents of the evaluator are never re-created. However, languages like advanced and module are capable of mutating the global state of the evaluator, and this mutation will carry over to subsequent tests. As such, in cases where mutation is required, it is a good idea to specify before-test, which means that the evaluator state is reset before each test is run. The option before-expression should rarely be useful. In addition to resetting before running the student test, it also resets before evaluating any expected values and before running the equality function to compare the result of the student test and the expected value. However, these other evaluations generally should not involve mutating the evaluator's environment. The last option is a special case. Typically, standard input and standard output are effectively directed to /dev/null during evaluator load/reset, because it is undesirable for student tests to consume input and/or write to output, making their results incorrect. If standalone-program is specified as reset frequency, though, then loading the code into the evaluator must be part of the test (and often exclusively the test), and I/O will be active during this process.

(modules mod1 ... modN)

Specifies which modules (typically found in the provided subdirectory, but also potentially submitted by students) are to be included at the next loadcode. This can be used for a variety of purposes, such as rewriting functions available for students, providing a required teachpack (or an equivalent, more testable alternate to a required teachpack), or providing equality testing functions.

(equal equal-fn)

Specifies a function that compares the result of running the student code and the expected value for equality. It must take two arbitrary values and produce a boolean. Care should be taken to ensure that, no matter what type of value is produced by the student code, this function never raises an exception. Note that this function will be evaluated in the context of the student evaluator. The default is equal?.

(expected expr1 ... exprN)

Specifies a series of expressions (typically only one) which will be evaluated and then compared to the result to determine if the test passed or failed. Note that if this is not specified, then the testing will assume that standard output is to be compared to the standard output of the model solution, using the command specified by the diff option.

(result expr)

Specifies a single S-expression, which will be evaluated in the student evaluator context to determine what the student code produced as the result of a test.

Note that all options above are read into BitterSuite in a module language context. Because of this, care should be taken with numbers; you generally need to be sure you are not accidentally supplying inexact numbers where you should not be. See SchemeRationalNumbers for details.

Relevant files

For backward compatibility with BitterSuite 2, and for consistency with the conventions in other languages, the files test.ss and test.scm are understood by the scheme language. They are completely equivalent to the options.ss and options.scm files, but typically would include information relevant to a particular test (desc, expected, and result).