ISG Testing

A home for testing software used by ISG.

• RST
  • RSTPublicTests
  • RSTA
  • DistributedRST
• Marmoset
• BitterSuite

The ImprovePit contains suggestions for how each of these pieces of software may be improved.

Brief discussion of testing software

RST is a testing framework that is designed to run user-provided scripts to do testing for it, and handles the details like setting up a temporary directory, switching to a testing account, and generating a results file automatically. This is what's run from the command-line by the tutors. It is solely a testing system; the submission details are handled by MarkUs or submit (ISGScriptsManPages#Submit). It is often used for running secret autotests after submissions have been closed, and can be accessed via the RSTPublicTests front-end to provide unlimited public test runs to students.

Marmoset is an alternate to RST that uses a website frontend to control both submission and testing. It is oriented to allowing time windows in which students can request all eligible tests be run on their code; these tests are divided into public, request, and secret tests, and passing all tests in one category unlocks the next category.

BitterSuite is a system to designed to hide the testing details in a particular domain (ie, testing in Scheme or in C) and allow the test designer to specify particular testing information. For example, in Scheme a file and associated modules are specified to load in a particular language, and then a number of (expected ...) and (result ...) expressions are given for particular test cases; the tutor/TA/IA/... doesn't need to know how the tests are done. It has hooks that allow it to run in RST or in Marmoset.