BitterSuite C Testing

This page will give helpful hints for how to do C testing.

• Verifying C code compiles without generating warnings

Testing Hierarchy

Solutions to C questions must be provided and be put into the solutions folder.

Any header files, equality checkers or other files that you want the students to have access to should go in the provided folder.

All options, including information relating to tests can go in the options.ss or options.scm files.

Usually the test cases for each C question should include one called wall, which would test whether the code compiles correctly. Read this.

Tests should go in the test.c files.

Extra information for the options.ss files

In options.ss, there are a number of available keys to specify, some of which are mandatory, some are optional; see BitterSuiteC for details about all specific options for C. The keys value, desc, timeout, memory, and diff work as in the base code; see the man page for details.

For a typical assignment that is only testing C, the main options.ss file, the options file will include language, and them may include other options such as timeout. These settings will apply to all subdirectories of in/, unless they are overriden inside one of the subfolders. For example:

(desc "Assignment 9")
(language C)
(timeout 6)

Each question also typically has its own options.ss file, which holds any information specific to one question. Very frequently this is where student-files and compiler-args will be found. compiler-args specifies the flags passed to gcc, and student-files specifies the files that contains the student code. See below for more details on how the tests work.

Any settings from the main options.ss file will be kept, unless something new is given. Also, if the assignment contains both Scheme and C question, language should be specified at each question instead of the main options.ss file. For example, you can specify this options.ss for the test cases for a10q2:

(desc "a10q2.c Tests")
(language C)
(value 2)
(compiler-args "-std=c99" "-O" "-Wall")
(student-files "a10q2.c" "wlpp.c" "testwlpp.c")
;; a10q2.c is the student solution
;; wlpp.c is the common code available to all students
;; testwlpp.c contains the test routines

You need to provide separate options.ss for each question if you want to specify the name of each test cases.

leaf options.ss and test.c file formatting

In a leaf directory, test.c and options.ss typically need to be provided.

test.c

In test.c, all provided header files need to be included, and the test you want to carry out should go into the main function.

For testing C codes we usually make the test.c so that the result of running a function is output to stdout. Bittersuite will do a diff check afterwards.

For example, sum.c can be tested with the following test.c:

#include <stdio.h>

int main()
{
    // int sum(int) does not output anything to stdout,
    // so we need to do this explicitly

    printf ("%d\n", sum(3));
    return 0;
}

Remember in order to make the diff test working, you need to run rsta against your test cases at least once before you actually apply the tests to the assignments. Run rsta in your terminal to learn the usage.

Leaf options.ss

In fact leaf options.ss is optional for C test cases. But if, for example, a different value is assigned to a single test, the value needs to be specified here. Also, if you want to specify a name for the specific test, you would want to make options.ss present.

For example:

(desc "Sigma3")
(value 6)

Reuse test code

Tutors usually found that the test.c files for each tests of a question look similar. For example, you have a function called testBstStruct to test the structure of a binary search tree the student code created. The only different in each test.c is the values in the tree the student code should create.

Rather than specify the values in each test.c driver, you can write a generic testBst.c file and put it into the provided folder. In side testBst.c provide a simple parser that read specifications about the test from stdin, and carry out the test as instructed by the input. Then call this function in each test.c. This allows you to design the test cases with input files rather than test.c files. With this approach the test code is easier to maintain and debug.

You can even put the main routine inside testBst.c to reuse the last piece of code inside test.c. As of Fall 2009, you have to provide a test.c for the test engine to work, so if you want to reuse the driver, just provide an empty test.c.

rsta

Since C testing is exclusively done with diff, we need to generate the correct output for the student code could to be tested against with. rsta does this for us. It runs rst against the solution code, and put the sample output into the answers folder.

Always run rsta before you make the public test public, or run distrst on student code.

See RSTA for a brief introduction.

How the tests work

Bittersuite 3 will compile the student files (specified with (student-files ...) in options.ss) into object files with gcc with flags specified in (compiler-agrs ...). Then it will compile test.c and link it with the object files, and execute the compiled executable.

For C programs we test the code by checking the output from the compiled program. That means if the student code does not output anything, then the tutor need to explicitly use printf( ... ) in test.c to output the returned value of calling the student functions.