The script maketests was designed by Sean Harrap in the fall term of 2015. Minor edits were made by Ten Bradley in winter 2016. The goal of the script is to make developing and maintaining marmoset tests more simple for CS 246 ISAs.

Directory Structure

To use the script, an assignment in the testing account must be setup as follows:

• Each question must have its own directory located at ~/marmoset/TERM/ASSIGNMENT/QUESTION where TERM is the current school term (e.g. W14), ASSIGNMENT is the assignment name (e.g. a3), and QUESTION is the question name (e.g a3q4a)
• Within the described directory, there must be a directory called setup. This directory will be where all changes to tests must be made.
• There are other directories which will exist in this directory by the end of making a project. These directories are:
  • submissions: has subdirectories of the form "sol_name". Once tests are ready, a solution can be tested by calling "build sol_name". This tests the solution as if it was submitted to marmoset.
  • BuildServers: contains the results from building solutions. The results of the last run submission will be stored as a static link in Current.
  • tests: directory containing the testing information after maketests has been run. Any changes made to testing informtaion in this directory will not be permenant.
• Calling the program "mkproj" will setup some default testing information.

maketests.conf

A file called setup.properties must exist in the setup directory. maketests uses this file to determine the settings for the marmoset test. The following are options accepted for all tests.

Option	Effect
TBD

Test Types

The following is a list of the current types of default tests that can be implemented and the format of the setup directory for each type.

Bash

Used for testing bash scripts.

TBD

Testing

Used for testing students test suites for questions.

In maketest.config settings, there must be a line for each public, release and secret test "TESTNAME-tests=n" where TESTNAME is the name of a test and n is the number of tests associated with each test. This makes it possible to have multiple programs which need to be passed for a specific test on marmoset to pass.

In the setup directory, have a directory, by default calles src, for the source code for an assignment. This code will be altered to make buggy programs (described below).

Maketests will create a bash test for each test listed in maketest.config as normal based on the template at "~/marmoset/files/makefiles/testsetups/testing". This script should not be changed.

Maketests will also compile all buggy programs used for testing. This is done by compiling the source program with marcos TESTNAME and TESTNAME_i where i = 0 to n-1 to the name TESTNAME_i.

To create a buggy program, insert various if statements encapsulated in a preprocessor if command. For example,

#include <iostream>
using namespace std;

int main (){
    int n;
    cin >> n;

    #ifdef SECRET0
    int check;
    #ifdef SECRET0_0
    check = 1;
    #endif
    #ifdef SECRET0_1
    check = 20;
    #endif
    if ( check > n ){
        cout << "Input greater than " << n << endl;
    }
    #endif

    cout << n << endl;
}

When this program is compiled with the different marcos, the output will differ from what the expected output for the actual program when specific cases are meet, in this case if n > 2 or n > 20 depending on the test. For a student to pass secret0, they will have to have a tests in their testsuite which meet each condition.

Please document tests to describe what each test is testing for other coursestaff.

The template test for testing does the following:

• Publc tests: Runs the students testSuite with the compiled executable. This uses runSuite to ensure all students tests match the actual output from the exectuable.
• Secret and Release tests: For TESTNAME, the testing script will execute runSuite with the students testsuite on each compiled buggy version for TESTNAME. If a students tests find a bug, the output will be different than their expected output and the output from runSuite will not be empty. To pass a release or secret test, all programs associated with TESTNAME must fail.

For an example of test of this type, check "~/marmoset/F15/a3/a3q2a".

Pipeline

TBD

Egrep

TBD

Handmarking

TBD

Compile

TBD

-- KirstenBradley - 2016-05-18