Marmoset

Marmoset is a website that lets students submit their code and assignment online. When students submit, Marmoset will test the student's submissions and tell them how they did.

The Marmoset website is: https://marmoset.student.cs.uwaterloo.ca/

Getting Marmoset for your course

Please contact CSCF if you want to use Marmoset for your course, or to just try it out. Please also report Marmoset bugs and problems to CSCF.

Adding users

Students and course staff cannot access Marmoset for your course unless they are granted permission (ie they are added to the People table on your course's Marmoset homepage). On your course's homepage, there are three links:

  • "Register UW students for this course by uploading a classlist file": Use this link to upload a classlist file containing all the students you want to add. For example, you can upload the /u/csNNN/.classlist file in the course account.
  • "Register one student for this course using a web interface": Use this link to add one student at a time. You can use the student's Quest ID (up to 8 characters) for all the fields except first and last names.
  • "Register an Instructor or TA for this course using a web interface": Use this link to add other TAs, instructors, and course staff. You can use the staff's Quest ID (up to 8 characters) for all the fields except first and last names.

Creating a project

From the course's Marmoset homepage, click "create new project" link and set the values as appropriate for your project/assignment.

Setting up test suites

After you create a project, you need to upload a test suite and canonical (sample) solutions. Students cannot submit until your canonical solutions pass your test suite. Once your canonical solutions pass the test suite, you can assign the number of points for each test. You can then make the project visible so that students can begin submitting.

There are two kinds of test setups: static and dynamic. In static testing, you upload a zip file that contains all your test scripts and test input data. In dynamic test, the test scripts are saved in the csNNNt account, and you upload a single test.properties that point to your test scripts in the csNNNt account.

If your test setup zipfile is too big to upload on to Marmoset, then you have to use dynamic tests.

If there are any environment variables you want all your test scripts to have, you can export them in the file /u/csNNNt/.marmoset_buildserver_environment. This file is sourced before running any test script for your course.

Static tests

Attached below (sample_test_setup.zip) is an example static test zipfile. This example test suite will always pass, and it'll output some information showing the testing environment (ex. output of pwd, whoami, and environment variables).

The test.properties file can define:

  • build.language: Set this to UW
  • test.class.public, test.class.release, and test.class.secret: Set these to your public, release, and secret tests.
  • build.make.command: The make command to use (ex. /usr/bin/make).
  • build.make.file: The name of the make file (ex. TestMakefile).
  • test.timeout.testCase: The timeout in seconds FOR DISPLAY PURPOSES ONLY. If the student's code times out, this value will be displayed to the student in their test outcome. Setting test.timeout.testCase does NOT actually make Marmoset time out the tests; you must implement a timeout yourself, for example by using the Linux timeout command in your test scripts.

When students submit on Marmoset, the following actions take place:

  1. A buildserver which is configured to regularly check your course for untested submissions will detect the new submission after a couple minutes (usually under a minute).
  2. That buildserver will extract the student's submitted zip file using "unzip -DD". The -DD means the extracted files will have the current time. The zip file is extracted in the buildserver's build folder, which, as an example, has a path that looks something like /u/csNNNt/buildserver/bs1.s18.csNNNt.ubuntu1604-002.student.cs/build/.
  3. That buildserver extracts the test setup zip file using "unzip -DD", in the same folder as above step. Marmoset will replace any files that already exist (so student's files may be overwritten).
  4. That buildserver runs /usr/bin/make -f TestMakefile (according to the settings in test.properties)
  5. That buildserver runs (more precisely, sources) the file /u/csNNNt/.marmoset_buildserver_environment and then runs your test scripts (according to the settings in test.properties). The .marmoset_buildserver_environment file is sourced before running each test script. Here is an example of how Marmoset does this step; the buildserver runs:

bash -c if [ -x "/u/csNNNt/.marmoset_buildserver_environment" ]; then source "/u/csNNNt/.marmoset_buildserver_environment"; fi; /u/csNNNt/buildserver/bs1.s18.csNNNt.ubuntu1604-002.student.cs/build/mypublictest1 61

The 61 argument is the test.timeout.testCase plus 1.

Marmoset runs all tests listed in the test.properties file in the current active test setup. Secret and release tests listed in test.properties are all run at the same time as public tests. Marmoset just hides the results of secret and release tests until students can see them (ex. when the deadline has passed or students spend a token).

Any output (both stdout and stderr) printed by your test script will be saved by Marmoset, and will be displayed to the students. If you don't want students to see certain output, you can redirect it to /dev/null. Each test case has 1 of 4 outcomes: correct, error, timeout, or failed. The outcome is determined by the test script's exit code. For example, if a test script exits with code 2, then it means that test case timed out.

exit code outcome meaning
0 Correct student's code passed the test
1 Error student's code crashed
2 Timeout student's code ran too long
3 Failed student's code produced the wrong answer

Dynamic tests

In dynamic tests, you set build.make.file to the folder where your test scripts are saved. And you set build.make.command to a special script which basically copies the the test scripts to the current directory (the directory where Marmoset is doing its testing). An example of this special script is below. Most courses name this script "dynamic_test".

#!/bin/bash

export MAKEFILE="`grep build.make.file $2/test.properties | sed -e 's/.*=//'`"
export MAKEPATH="`grep build.make.command $2/test.properties | sed -e 's/.*=//'`"

if [ -z "$MAKEFILE" ]; then
    export MAKE_PARAMS=""
else
    export MAKE_PARAMS="-f $MAKEFILE"
fi

if [ -z "$MAKEPATH" ]; then
    export MAKEPATH="make"
fi

cp -r -f -p $2/* .

echo $MAKEPATH $MAKE_PARAMS

$MAKEPATH $MAKE_PARAMS || exit 1

exit 0

Recall from static testing section that Marmoset will run "<build.make.command> -f <build.make.file>". If you define the make command to the script above, it will copy the contents of <build.make.file>/ folder to the current directory (the directory where the testing done in).

Warning about dynamic tests: Changes you make to test scripts will take effect immediately, even if you don't upload anything to Marmoset. You need to be very careful when editing test scripts in the csNNNt account. Students can see your changes right away. To prevent problems, do your testing and development work in a separate folder from where your test scripts are. For example, create a brand new folder and do your work from there so that you don't affect the existing test scripts.

Buildserver Maintenance

Buildservers are processes that continually poll for untested submissions. If a buildserver detects that there is an untested submission, it'll go ahead and mark it. Otherwise, it'll sleep for a bit and poll again. Buildservers are configured to monitor specific courses. For example, a buildserver might only poll for untested submissions from CS136 Spring 2018. The number of buildservers launched for a course depends on the class size (bigger classes need more buildservers).

You can monitor the buildservers here: https://marmoset.student.cs.uwaterloo.ca/status/QueryBuildServerStatus

In your course's csNNNt account, there is a folder called ~/buildserver/. That folder contains a folder for each buildserver. The buildserver's folder are named something like bs2.w18.cs341t.ubuntu1604-002.student.cs. The term (w18), course, and hostname will differ from buildserver to buildserver. Inside each buildserver folder, you can find logs and a "build" folder, which is where Marmoset runs tests from.

When you request Marmoset for your course, CSCF will start up some buildservers for you, across some of the linux.student.cs.uwaterloo.ca hosts. Having buildservers on multiple servers means that if one server goes down, Marmoset can still continue testing submissions.

The buildservers run as csNNNt account. The student's code are run as csNNNt, too. So please do not store any sensitive information in the csNNNt account (or make sure you sandbox the student's program when you run it).

Starting (or restarting) buildservers

Run from the csNNNt account, on the server which you want to start buildservers: /u/cs_build/bin/restartbuildservers

This script will scan the /u/csNNNt/buildserver folder and attempt to start any buildservers that aren't already running.

Killing buildservers

You may need to kill buildservers if they get stuck testing a student's submission (ex. the student's program has an infinite loop and runs forever). Buildservers are also killed at the end of the term.

The buildservers are just processes, so you can kill them with kill or pkill command.

Running pgrep -al -u csNNNt -f 'java.*buildserver' will show you the buildservers running for your course (the command is very long...) without killing them.

You can kill all the buildservers on the server you're on with pkill -u csNNNt -f 'java.*buildserver'.

If a buildserver goes down, it will automatically restart after 10 minutes. This is done by a cronjob on ubuntu1604-002.student.cs, under the cs_build account.

Creating buildservers

CSCF will create some buildservers when you request Marmoset for your course. The buildservers are created using the script /u/cs_build/bin/make_buildserver. Run this script without any arguments to see help. Example usage is (for cs246t, creating 6 buildservers on ubuntu1604-002):

/u/cs_build/bin/make_buildserver 6 'CS246,CS246_PROJECT' 'Spring 2018' --symlink-build --group=s18

Topic attachments
I Attachment Action Size Date Who Comment
Compressed Zip archivezip sample_test_setup.zip manage 6.0 K 2019-04-15 - 21:44 YiLee Sample test setup suite for Marmoset
Topic revision: r7 - 2019-04-15 - YiLee
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback