Verifying C Code compiles without warnings

This page discusses how to design tests with appropriate feedback which will verify that code submitted by students will compile without generating any errors or warnings. This test was first developed and used in the Winter 2009 offering of CS 136.

Create an External Test

A C test is designed to test a student's code submission; it is not designed to grab the output of the compiler and use that as an explicit test. Because of this, using a script test is necessary.

For this test, we'll assume the submitted file in question is called isqrt.c, and that the functions implemented in it must conform to the interface specified in a provided isqrt.h file. The isqrt.h file would be placed in this test suite's provided directory.

As this test must run in the external language, a sample options.ss file for a group of tests may contain:
(language external)

Then, the corresponding test.exe file at the same level could contain the following:

#!/usr/bin/env bash

readonly FILEBASE="$1"

# Create a temporary file that includes the appropriate header.

if [ -e "$FILEBASE.c" ]; then
   if [ -s "$FILEBASE.c" ]; then
      mv "$FILEBASE.c" "$FILEBASE.original"

      echo "#include \"$FILEBASE.h\"" > "$FILEBASE.c"
      cat "$FILEBASE.original" >> "$FILEBASE.c"

      gcc -std=c99 -O -Wall -c "$FILEBASE.c" 2> "$FILEBASE.walltest"
   else
      echo "File $FILEBASE.c is empty" > "$FILEBASE.walltest"
   fi
else
   echo "File $FILEBASE.c not found" > "$FILEBASE.walltest"
fi

if [ -s "$FILEBASE.walltest" ]; then
   echo "$FILEBASE.c did not compile without warnings.  See compilation information below for details." >&4
   echo "0" >&3
else
   echo "100" >&3
fi

if [ -s "$FILEBASE.original" ]; then
   mv "$FILEBASE.original" "$FILEBASE.c"
fi

And individual test directories may have descriptions such as:
(args isqrt)
(desc "isqrt.c compiles without warnings")

This test file will force inclusion of the header file in case the student did not include it already, catching any differences between the return/parameter types expected for the function, and those specified in the implementation. Then, the suite will attempt to compile this file to an object file, redirecting standard error to the special file isqrt.walltest. Lastly, a pass or fail mark will be sent to file descriptor 3.

NB: This test originally did not first test if $FILEBASE.c existed before forcing header inclusion and compiling. As a result, students who submitted no code would pass this test. This highlights the importance of careful coding and very thorough testing.

This is sufficient to indicate whether compilation succeeded without generating warnings or not, and assign marks accordingly. However, in the failure case, it does not indicate to the students why their code failed. To do this, we want to include the contents of the walltest files as part of the generated assignment output, as the failure message indicates.

Warning: This test is written in a non-ideal fashion as it writes over a file that may be needed by other tests. Instead of overwriting $file.c directly (which is done so compiler warnings/errors reference the correct file name so as not to confuse students), a new copy of $file.c should be created in a temporary subdirectory, and compilation should be attempted there. Any side effects of this test will then be hidden from any other tests, and this test will be concurrency-safe with respect to other tests.

Creating a =computeMarks-postprocess hook

In order to include the compiler output, we can provide a file called computeMarks-postprocess in the same directory as computeMarks, which will be executed after test totaling has been completed, but shortly before termination of the computeMarks program. Sample contents for this are:

#!/usr/bin/env bash

for filebase in isqrt sumsqr regular; do
   if [ -s "$filebase.walltest" ]; then
      keepFile "$filebase.walltest" 128 -n -h "Errors/warnings for $filebase.c compilation"
   fi
done

This program simply calls keepFile on every compile output file that has non-zero size, truncating at 128 lines and providing a descriptive header message. For more information about keepFile, see the RST documentation.