The script maketests was designed by Sean Harrap in the fall term of 2015. Minor edits were made by Ten Bradley in winter 2016. The goal of the script is to make developing and maintaining marmoset tests more simple for CS 246 ISAs.

Directory Structure

To use the script, an assignment in the testing account must be setup as follows:

• Each question must have its own directory located at ~/marmoset/TERM/ASSIGNMENT/QUESTION where TERM is the current school term (e.g. W14), ASSIGNMENT is the assignment name (e.g. a3), and QUESTION is the question name (e.g a3q4a)
• Within the described directory, there must be a directory called setup. This directory will be where all changes to tests must be made.
• There are other directories which will exist in this directory by the end of making a project. They should be automatically made when maketests is first run. These directories are:
  • submissions: has subdirectories of the form "sol_name". Once tests are ready, a solution can be tested by calling "build sol_name". This tests the solution as if it was submitted to marmoset.
  • BuildServers: contains the results from building solutions. The results of the last run submission will be stored as a static link in Current.
  • test: directory containing the testing information after maketests has been run. Any changes made to testing informtaion in this directory will not be permenant.
• Calling the program "mkproj" will setup some default testing information.

maketests.conf

A file called maketests.conf must exist in the setup directory. maketests uses this file to determine the settings for the marmoset test. The following are options accepted for all tests. Each line should contain OPTIONNAME=OPTIONARG where OPTIONNAME is the name in the Option column of the table below, and OPTIONARG is the setting (usually described in the Argument column of the table below). All lists are whitespace-delimited.

Many of the actions described in the provided options are specified by an automatically-generated TestMakefile. If you put your own TestMakefile in setup/, it will be used instead of generating one automatically, which means many options here won't do anything. Writing your own TestMakefile is not recommended without reading the maketests script first.

Option	Argument	Effect	Required?
test-type	The name of the test type	See below	Yes
cpp-includes	A list of standard library header names	If this option is present, only the specified standard library headers may be used. If it is not, all STL headers may be used.	No
required-files	A list of files	The listed files must be present in all student submissions.	Yes
build-settings	false, makefile, or x11	false: Compile student submission manually (running g++) makefile: Compile by running make. x11: Like false, but supply x11 linking information.	Defaults to false
cpp-permissive	Anything (usually true)	If provided, allow STL containers and allow C (malloc etc).	Defaults to false
student-file	A file name	The name to rename the student's executable to before testing.	Defaults to submission
makefile-output	A file name	Use the specified file name as the student's executable in testing, and verify it exists.	Recommended if student makefile supplied
source-dir	A directory name	The directory containing any provided source.	Defaults to src
solution	A file name	The name of the solution executable.	Defaults to solution
soln-dir	A directory name	The name of the directory containing solution files.	Defaults to soln
input-dir	A directory name	The name of the directory containing test input.	Defaults to inp
tests-public	A list of files	The names of the public tests	Defaults to public0 if testtype is compile or handmarking
tests-release	A list of files	The names of the release tests	No
tests-secret	A list of files	The names of the secret tests	No
provided	A list of files, or yes	The specified file names (within provided/) will be accessible to student submissions. If yes is supplied, it stands for the entire contests of provided/	Only if testtype=cppretval
provided-scripts	Anything (usually true)	If provided, use custom test scripts (contained within setup/) rather than the default scripts for the test type. There is currently no way to only do this for specific tests.	Only if testtype=cppretval
file-input-dir	A directory name	Copies the contents of the given directory into test/. Warning: if there is a risk of conflicts with student-named files, use provided instead, since provided is copied by the makefile whereas file-input-dir is copied by the maketests script.	Defaults to files
TESTNAME-tests	A number ≥1	See "Testing" test type below. TESTNAME is the name of the associated test (for example, public0-tests=2)	Defaults to 1

Test Types

The following is a list of the current types of default tests that can be implemented and the format of the setup directory for each type. These types mostly differ by which testing script is used by default (see ~cs246t/marmoset/files/maketests/testsetups/) as well as in some obvious ways (bash script tests must enable execute permissions on the script, while C++ tests must first compile the source code, etc).

bash

Run a bash script (usually associated with a1p3 and a1p4) and compare both the contents of stdout and stderr with a model solution.

cpp

Used for testing C++ programs (usually used in due date 2 of assignments 2, 3, 4).

First compiles the program. Then, for each test, it looks in inp/t-TESTNAME/ (where TESTNAME is the name of the test). Different sub-tests are specified by including a file named <subtestname>.in, and any argument input can be specified by <subtestname>.args. Note that the .in file should be present for every test, so if you don't want to supply anything on stdin just make an empty .in file.

For an example setup/ see ~cs246t/marmoset/archives/F15/a4/a4q3b/

cppretval

Used for testing C++ programs (usually used in due date 2 of assignments 2, 3, 4) whose correctness is determined by the return value of the program. This was an ad-hoc testtype written to support F15 a4q1. Scripts must be provided for this testtype, and the makefile is setup in the same way as the cpp testtype above.

For an example setup/ see ~cs246t/marmoset/archives/F15/a4/a4q1/

testing

Used for testing students test suites for questions (usually used in due date 1 of assignments 2, 3, 4).

In maketests.conf settings, there must be a line for each public, release and secret test "TESTNAME-tests=n" where TESTNAME is the name of a test and n is the number of tests associated with each test. This makes it possible to have multiple programs which need to be passed for a specific test on marmoset to pass.

In the setup directory, have a directory, by default calles src, for the source code for an assignment. This code will be altered to make buggy programs (described below).

Maketests will create a bash test for each test listed in maketests.conf as normal based on the template at "~/marmoset/files/makefiles/testsetups/testing". This script should not be changed.

Maketests will also compile all buggy programs used for testing. This is done by compiling the source program with macros TESTNAME and TESTNAME_i where i = 0 to n-1 to the name TESTNAME_i. The TESTNAME macro (SECRET0 below) will be true for all of the associated tests, whereas TESTNAME_i is only true for test i.

To create a buggy program, insert various if statements encapsulated in a preprocessor if command. For example,

#include <iostream>
using namespace std;

int main (){
    int n;
    cin >> n;

    #ifdef SECRET0
    int check;
    #ifdef SECRET0_0
    check = 1;
    #endif
    #ifdef SECRET0_1
    check = 20;
    #endif
    if ( check > n ){
        cout << "Input greater than " << n << endl;
    }
    #endif

    cout << n << endl;
}

When this program is compiled with the different macros, the output will differ from what the expected output for the actual program when specific cases are met, in this case if n > 2 or n > 20 depending on the test. For a student to pass secret0, they will have to have a test in their testsuite which catches both of these tests.

Please document tests to describe what each test is testing for other coursestaff.

The template test for testing does the following:

• Publc tests: Runs the students testSuite with the compiled executable. This uses runSuite to ensure all students tests match the actual output from the exectuable.
• Secret and Release tests: For TESTNAME, the testing script will execute runSuite with the students testsuite on each compiled buggy version for TESTNAME. If a students tests find a bug, the output will be different than their expected output and the output from runSuite will not be empty. To pass a release or secret test, all programs associated with TESTNAME must fail.

For an example of test of this type, check "~cs246t/marmoset/archives/F15/a3/a3q2a".

pipeline

Run a bash pipeline command (usually associated with a1p1) and compare the output with a model solution.

egrep

Interpret the student submission and model solutions as patterns to be supplied to egrep (usually associated with a1p2), and otherwise proceed as in pipeline.

handmarking

Verifies that a file is submitted but otherwise does nothing. Usually associated with a5 documentation and other similar questions.

compile

Verifies that a file is submitted and can be compiled but runs no tests. Usually used for project submissions and graphical display questions.

-- KirstenBradley - 2016-05-18