Marmoset

Marmoset is a website that lets students submit their code and assignment online. When students submit, Marmoset will test the student's submissions and tell them how they did.

The Marmoset website is: https://marmoset.student.cs.uwaterloo.ca/

Getting Marmoset for your course

Please contact CSCF if you want to use Marmoset for your course, or to just try it out. Please also report Marmoset bugs and problems to CSCF.

Adding users

Nobody can access your course in Marmoset until they are added to the People table on your course's Marmoset homepage. On your course's homepage, there are three links:

• "Register students or groups for this course using a CSV file": Use this link to upload a CSV file containing all the students you want to add. There are more details on the upload web page.
• "Register one student for this course using a web interface": Use this link to add one student at a time.
• "Register an Instructor or TA for this course using a web interface": Use this link to add other TAs, instructors, and course staff.

Creating a project

From the course's Marmoset homepage, click "create new project" link and set the values as appropriate for your project/assignment.

Setting up test suites

After you create a project, you need to upload a test suite and canonical (sample) solutions. Students cannot submit until your canonical solutions pass your test suite. Once your canonical solutions pass the test suite, you can assign the number of points for each test. You can then make the project visible so that students can begin submitting.

There are two kinds of test setups: static and dynamic. In static testing, you upload a zip file that contains all your test scripts and test input data. In dynamic test, the test scripts are saved in the csNNNt account, and you upload a single test.properties that point to your test scripts in the csNNNt account.

If your test setup zipfile is too big to upload on to Marmoset, then you have to use dynamic tests.

If there are any environment variables you want all your test scripts to have, you can export them in the file /u/csNNNt/.marmoset_buildserver_environment. This file is sourced before running any test script for your course.

Static tests

Attached below (sample_test_setup.zip) is an example static test zipfile. This example test suite will always pass, and it'll output some information showing the testing environment (ex. output of pwd, whoami, and environment variables).

The test.properties file can define:

• build.language: Set this to UW
• test.class.public, test.class.release, and test.class.secret: Set these to your public, release, and secret tests.
• build.make.command: The make command to use (ex. /usr/bin/make).
• build.make.file: The name of the make file (ex. TestMakefile).
• test.timeout.testCase: The timeout in seconds FOR DISPLAY PURPOSES ONLY. If the student's code times out, this value will be displayed to the student in their test outcome. Setting test.timeout.testCase does NOT actually make Marmoset time out the tests; you must implement a timeout yourself, for example by using the Linux timeout command in your test scripts.
• test.class.public.points, test.class.release.points, and test.class.secret.points: You can assign what public, release, and secret tests are out of by setting these entries. For example, setting test.class.public.points=24 will make all public tests be out of 24 points. When you make a test setup active, you'll be presented with a web form where the point values are filled in automatically with these values. At that time, you can override the point values if needed.
• test.name.points.testName: Lets you assign the points a specific test case is out of. Replace testName with the name of the test case. For example, setting test.name.points.mypublictest1=7 will make the test case with name mypublictest1 be out of 7 points.

1. A buildserver which is configured to regularly check your course for untested submissions will detect the new submission after a couple minutes (usually under a minute).
2. That buildserver will extract the student's submitted zip file using "unzip -DD". The -DD means the extracted files will have the current time. The zip file is extracted in the buildserver's build folder, which, as an example, has a path that looks something like /u/csNNNt/buildserver/bs1.s18.csNNNt.ubuntu1604-002.student.cs/build/.
3. That buildserver extracts the test setup zip file using "unzip -DD", in the same folder as above step. Marmoset will replace any files that already exist (so student's files may be overwritten). Note that /u/csNNNt/bin is added to your PATH for you
4. That buildserver runs <build.make.command> -f <build.make.file> using the settings in test.properties. With the example values above, runs /usr/bin/make -f TestMakefile
5. That buildserver runs (more precisely, sources) the file /u/csNNNt/.marmoset_buildserver_environment and then runs your test scripts (according to the settings in test.properties). The .marmoset_buildserver_environment file is sourced before running each test script. Here is an example of how Marmoset does this step; the buildserver runs:

bash -c if [ -x "/u/csNNNt/.marmoset_buildserver_environment" ]; then source "/u/csNNNt/.marmoset_buildserver_environment"; fi; /u/csNNNt/buildserver/bs1.s18.csNNNt.ubuntu1604-002.student.cs/build/mypublictest1 61

The 61 argument is the test.timeout.testCase plus 1.

Marmoset runs all tests listed in the test.properties file in the current active test setup. Secret and release tests listed in test.properties are all run at the same time as public tests. Marmoset just hides the results of secret and release tests until students can see them (ex. when the deadline has passed or students spend a token).

Any output (both stdout and stderr) printed by your test script will be saved by Marmoset, and will be displayed to the students. If you don't want students to see certain output, you can redirect it to /dev/null. Each test case has 1 of 4 outcomes: correct, error, timeout, or failed. The outcome is determined by the test script's exit code. For example, if a test script exits with code 2, then it means that test case timed out.

Dynamic tests

In dynamic tests, you set build.make.file to the folder where your test scripts are saved. And you set build.make.command to a special script which basically copies the the test scripts to the current directory (the directory where Marmoset is doing its testing). An example of this special script is below. Most courses name this script "dynamic_test".

#!/bin/bash

export MAKEFILE="`grep build.make.file $2/test.properties | sed -e 's/.*=//'`"
export MAKEPATH="`grep build.make.command $2/test.properties | sed -e 's/.*=//'`"

if [ -z "$MAKEFILE" ]; then
    export MAKE_PARAMS=""
else
    export MAKE_PARAMS="-f $MAKEFILE"
fi

if [ -z "$MAKEPATH" ]; then
    export MAKEPATH="make"
fi

cp -r -f -p $2/* .

echo $MAKEPATH $MAKE_PARAMS

$MAKEPATH $MAKE_PARAMS || exit 1

exit 0

Recall from static testing section that Marmoset will run "<build.make.command> -f <build.make.file>". If you define the make command to the script above, it will copy the contents of <build.make.file>/ folder to the current directory (the directory where the testing done in).

Warning about dynamic tests: Changes you make to test scripts will take effect immediately, even if you don't upload anything to Marmoset. You need to be very careful when editing test scripts in the csNNNt account. Students can see your changes right away. To prevent problems, do your testing and development work in a separate folder from where your test scripts are. For example, create a brand new folder and do your work from there so that you don't affect the existing test scripts.

Buildserver Maintenance

Buildservers are processes that continually poll for untested submissions. If a buildserver detects that there is an untested submission, it'll go ahead and mark it. Otherwise, it'll sleep for a bit and poll again. Buildservers are configured to monitor specific courses. For example, a buildserver might only poll for untested submissions from CS136 Spring 2018. The number of buildservers launched for a course depends on the class size (bigger classes need more buildservers).

You can monitor the buildservers here: https://marmoset.student.cs.uwaterloo.ca/status/QueryBuildServerStatus

In your course's csNNNt account, there is a folder called ~/buildserver/. That folder contains a folder for each buildserver. The buildserver's folder are named something like bs2.w18.cs341t.ubuntu1604-002.student.cs. The term (w18), course, and hostname will differ from buildserver to buildserver. Inside each buildserver folder, you can find logs and a "build" folder, which is where Marmoset runs tests from.

When you request Marmoset for your course, CSCF will start up some buildservers for you, across some of the linux.student.cs.uwaterloo.ca servers. Having buildservers on multiple servers means that if one server goes down, Marmoset can still continue testing submissions.

The buildservers run as csNNNt account. The student's code are run as csNNNt, too. So please do not store any sensitive information in the csNNNt account (or make sure you sandbox the student's program when you run it).

Starting (or restarting) buildservers

Run from the csNNNt account, on the server which you want to start buildservers: /u/cs_build/bin/restartbuildservers

This script will scan the /u/csNNNt/buildserver folder and attempt to start any buildservers that aren't already running.

If a buildserver crashes, it will automatically restart after 10 minutes. This is done by a cronjob on ubuntu1804-002.student.cs.uwaterloo.ca, under the cs_build account.

Killing buildservers

You may need to kill buildservers if they get stuck testing a student's submission (ex. the student's program has an infinite loop and runs forever). Buildservers are also killed at the end of the term.

The buildservers are just processes, so you can kill them with kill or pkill command.

Running pgrep -al -u csNNNt -f 'java.*buildserver' will show you the buildservers running for your course (the command is very long...) without killing them.

You can kill all the buildservers on the server you're on with pkill -u csNNNt -f 'java.*buildserver'.

There is also a script that can kill buildservers which haven't responded after N seconds: /u/cs_build/bin/kill_stuck_buildservers N

For example, running /u/cs_build/bin/kill_stuck_buildservers 3600 will kill all buildservers that haven't responded after 1 hour on the server that the command is run on.

Regardless of how you kill the buildservers, after killing them, you can restart them by running the /u/cs_build/bin/restartbuildservers script as described above. Then check the buildserver status page to confirm that the buildserver is responding again.

If buildservers on different servers are stuck, you'll have to run the commands on each server.

Creating buildservers

CSCF will create some buildservers when you request Marmoset for your course. The buildservers are created using the script /u/cs_build/bin/make_buildserver. Run this script without any arguments to see help. Example usage for cs246t, creating 6 buildservers on the server that the command is run on:

/u/cs_build/bin/make_buildserver 6 'CS246,CS246_PROJECT' 'Spring 2018' --symlink-build --group=s18