Omar also maintains MarmosetCourseConfiguration covering UW-specific setup of Marmoset and running the buildservers.
Marmoset is a testing suite operated through a web interface (for both ISG and students); it provides an easy way to create programming problems and assign marks to each test case. The advantage of Marmoset over other test suites is it provides students with instant test feedback letting them know how they did. It is also possible to use Marmoset as a submission bank with no automatic testing if you would like to also have written problems thrown in (students can submit .txt or equivalent files). Student marks are also easily downloadable from marmoset into convenient .csv files.
The basic mechanism behind Marmoset is to have "public tests" and "release tests" (and sometimes secret tests). Public tests can be thought of as sanity checks for students. These tests are "free" and their results are instantly visible; students will automatically see these for every submission. Release tests require students to spend a token (these are limited and take a long time to recharge) to "release test" their submission. In general, most of the marks are allocated to release tests. Secret tests behave the same way as release tests however students do not get to see the results of these tests (hence they are 'secret').
It is worth noting that Marmoset was not created by UW and is modified from it original version to work with the UW systems. That being said, internet documentation should be trusted over external documentation.
How Marmoset Works:
To get Marmoset for your course, you should contact CSCF who can set up an instance of Marmoset for you for the term. Typically, whoever contacted CSCF will be added as an instructor to the instance.
Once this is set up, you will need to add all the tutors and instructors with 'instructor' privileges. You can do this by navigating to the home page for Marmoset and clicking 'Instructor View'. Then click on "Register an Instructor for this course using a web interface". Enter their first and last name and leave the rest of the fields as their UW userid (ex: kabob56).
Students will also need to be added with 'student' privileges. This can be done manually in the same way instructors are registered if you need to add only a few students. To add lots of people at once, use the "Register students or groups for this course using a CSV file" link.
The following videos illustrate the basics outlined below:
http://www.natidea.com/projects/marmoset/MarmosetStep1/MarmosetStep1.html (0:00 - 1:47)
http://www.natidea.com/projects/marmoset/MarmosetStep3/MarmosetStep3.html (4:49 - 7:25)
You can create a new project by logging into Marmoset as an instructor and navigating to the main page and clicking "create a new project". Fill the fields in as desired.
Notes:
After these two things have been uploaded and the Canonical Solution passes the Test Setup, you can "Activate" the Test Setup (this will cause people with student privileges to be able to submit and have their submission tested by the Test Setup). Although, until you make the project "Visible", students will not be able to see the project. WARNING: Once you make a project visible, it cannot be made 'invisible' again. Be sure to thoroughly verify that your test setup and canonical solution are correct before doing this!
This method is DEPRECATED and the Dynamic Test Setup is preferred (see below). However, understanding this method is key for understanding the dynamic method.
At bare minimum, Marmoset requires every test setup to include:
*** test.properties ***
This file contains metadata about how to build and test a submission. It must include the following lines (and these are sufficient):
build.language=UW test.class.public=(space separated list of bash scripts which correspond to public tests) test.class.release=(space separated list of bash scripts which correspond to release tests) test.class.secret=(space separated list of bash scripts which correspond to secret tests) build.make.command=(usually /usr/bin/make when the buildservers are set up on the linux servers) build.make.file=(if your makefile has a name other than makefile or Makefile you can specify it here)
Notes:
build.language=UW test.class.public=public0 public1 test.class.release=release0 release1 release2 build.make.command=/usr/bin/make
*** Makefile ***
The Makefile with respect to test suites can be thought of as a set of instructions for how to compile or pre-process a student's submission. Currently, it is processed using GNU Make located in /usr/bin/make. More can be read about GNU Make and Makefiles at http://www.gnu.org/software/make/.
Here is a very simple example of a Makefile (see the template links below for more complex Makefile setups):
TEST_INPUT_FILE = file_1.c file_2.c ... file_n.c CHECK_FILE = /u/cs_build/bin/marmoset_test_tools/checkfiles all: @$(CHECK_FILE) $(TEST_INPUT_FILE) @chmod u+x *
What it does:
*** public/release/secret test bash scripts ***
Since these tests are bash scripts, there is huge flexibility for configuring testing and test output. At bare minimum, the script should return one of the following exit codes to Marmoset:
exit code | outcome | meaning |
---|---|---|
0 | Correct | student's code passed the test |
1 | Error | student's code crashed |
2 | Timeout | student's code ran too long |
3 | Failed | student's code produced the wrong answer |
A very simple test which checks if a student's submission (called sub.txt) contains the text "hello" could look like the following:
#!/bin/bash export CORRECT=0 export ERROR=1 export TIMEOUT=2 export WRONG=3 grep 'hello' sub.txt > /dev/null RET=$? if [ $RET == 0 ] ; then echo "Your submission contains 'hello' - good job!" exit $CORRECT else echo "Your submission does not contain 'hello'. Try again." exit $WRONG fi
Notes:
The dynamic test setup is in principle the same as the static setup but instead of zipping the test configuration and submitting it, a 'stub' is submitted instead. All the 'stub' contains is the test.properties file with some subtle modifications:
Advantages:
To prevent unnecessary stress on the buildservers and to keep systems reasonably secure, ISG imposes certain limitations and security precautions.
Be aware that although the software that Marmoset is comprised of is maintained and run by CSCF, you are responsible for using it reasonably and within resource usage limits and guidelines established campus-wide. For example, your tests have the potential to cause an unreasonable load-increase on hosts running marmoset buildservers which violates established policies. So please be aware of this, especially around deadlines for your assignments/projects.
While it's difficult to define "acceptable" limits for student submissions, it has been adopted that 256-512MB ram+stack, 1MB max filesize, and 10 seconds of execution time are "reasonable". To impose these, we often use "wrapper" programs when executing student submissions which set up an environment with the imposed limitations (which submissions are executed in). An example of this is cwrapper, which appears in sample templates and (hopefully) in any test setups from previous terms.
In any case, your tests must use custom-built timeout commands!
Caveat:
As a security measure, cnNNNt accounts exist as 'sandbox' accounts. The idea is to keep these disjoint from the cnNNN course accounts to safeguard from malicious code or probing (by having all Marmoset tests run on the csNNNt account). The downside of this is that the test setups and solutions need to be kept on or duplicated to the csNNNt account which can be an inconvenience. Some scripts may exist (if they don't they are easy to write) which will duplicate the test setups for you on the course account.
If your test setup is not automatically run as csNNNt (which it probably is), then your tests MUST use sudo
to execute student submissions as csNNNt
.
sudo -u csNNNt command
command
as csNNNt
.
Sometimes buildservers will behave strangely or "hang". This is most likely to occur during submission deadlines. If you suspect unusual behavior, you can monitor the buildservers by logging onto Marmoset and following one of the two links (depending on whether your course is on marmoset or marmoset2).
https://marmoset.student.cs.uwaterloo.ca/status/QueryBuildServerStatus
https://marmoset2.student.cs.uwaterloo.ca/status/QueryBuildServerStatus
To restart hanging buildservers follow the instructions here:
https://cs.uwaterloo.ca/twiki/pub/ISG/Marmoset/RestartingMarmosetbuildservers1.pdf
Standardize cwapper and move it to cs_build.
OR
Replace cwrapper and lockout.o with 'Box'. Box was written by Martin Mares in Sweden, and has been used extensively for programming contests. No one has formally followed up on this yet, but with some testing and template creation, it may replace cwrapper and lockout.o.
More can be read here:
A slightly modified version of Box exists in the CS145 course account:
In case students submit a Makefile to be compiled, the Makefile bundled with the test setup should have a cryptic name. Recall that files in the test setup will overwrite any files submitted by students! If you do this, it's important to specify the name of your cryptically named Makefile using the following two lines in test.properties.
build.make.command=/software/gnu/bin/make build.make.file=SomeCrypticMakefileName
Avoid creating *Upload-only" Marmoset projects, even if the "tests" do nothing of value with a submission.
Whenever something unexpected happens, determine whether the tools you are using or servers you are relying on, have log files. Open them up and look through them. Most logs have time-based entries to help you home-in on error conditions. The log entries usually have integer identifiers as well, to simplify searching for problems if a particular entity (e.g. a student's submission) is experiencing problems.
/u/csNNNt/buildserver/bsN.csNNNt.*.student.cs/log/
. (use csNNN if submissions are not automatically run as csNNNt)
services14.student.cs.uwaterloo.ca
under /fsys2/tomcat5.5/logs/marmoset
.
To create and test a project, use the $user
account rather than ${user}-student
or ${user}-canonical
. After selecting a course, you will counter-intuitively be seeing a student view from the instructor perspective; you must click "instructor view" on the top bar to do anything productive. Note also that this student view is different than the real student view; you must log in as ${user}-student
to see what the students see!
All test setups and templates can be roughly categorized into two flavours:
The following is documentation from the old Marmoset documentation page. It still needs to be read through and consolidated into the above document. The contents of the old page are here: https://cs.uwaterloo.ca/twiki/pub/ISG/Marmoset/olddoc.txt
By default, you will not be able to read any files created by sudo. While appropriate permissions can be set up at startup, any careless change in permissions can prevent your tests from running. Because of this, your Makefile should set permissions similarly to the following before any calls requiring sudo: while [[ `/software/.admin/bins/bin/absolute ..` != `/software/.admin/bins/bin/absolute .` ]]; do cd ..; chmod ug+rX * 2> /dev/null; chgrp "$(whoami)t" * 2> /dev/null; done; exit 0 chmod ug+rwx . chgrp "$(whoami)t" . chgrp -R "$(whoami)t" * chmod -R ug+rx * chmod -R a-w * To make this work properly, you should make sure that your Makefile is executed in bash. To enforce this, one of the initial variables you set at the start of the Makefile should be: SHELL = "/xhbin/bash" ---++++ Your test files must be writeable This is a weird one. Marmoset was throwing completely unhelpful errors of the form Compile/Build unsuccessful edu.umd.cs.buildServer.BuilderException: Couldn't build project Dropping temporary files revealed that the Makefile was completing successfully, but that none of the tests (listed in =test.class.*= in the =test.properties= file) were being executed. The buildserver log file came to the rescue here, with the more helpful information: ERROR - IOException trying to build submission 360561 java.io.FileNotFoundException: [...path...] (Permission denied) Permissions were loosened from 500 (user-only read and execute) to 555 (everybody read and execute), but the test still would not run. As soon as the permissions were changed to 755, Marmoset happily ran them. So, even though there should be no good reason to write test files, make sure your Makefile makes them writeable. Further testing will be needed to determine if permissions like 750 or 700 are also acceptable. ---++++ Your tests must be _paranoid_ about permissions Not only do you need to ensure that files are readable from sudo (meaning enforcing group permissions beforehand), you must also ensure that files are writeable from the course account (meaning enforcing group permissions through sudo after testing is done). The current assumption is that because Marmoset reuses hardcoded build directories instead of a subdirectory of =/tmp=, it *must* have permission to modify all files in that build directory as the course account user. If it does not, it will die with an unhelpful error message; although Java exceptions regarding file permissions written are to the log, any indication of why it cares is not. ---+++ Updating the buildserver runtime environment variables While logged in as the course account: 1 Modify =~csNNN/.marmoset_buildserver_environment= and add *bash-style* =export= commands to set the variables you need. 1 Restart the buildservers by executing the following command: * =~cs_build/bin/buildserver_hosts_command pkill -u csNNN java= ---+++ Avoid using =student.cs= for any sort of work Do *not* use =student.cs= for any sort of development or testing. See [[TAUnix#AvoidingCertainHosts]] for more information. ---+++ Avoid creating "Upload-only" Marmoset Projects Marmoset allows one to create projects where students can submit files with no expectation of testing. Converting such projects into regular testable projects, after students have started to submit to them, requires a bit of database hackery that may result in more harm than good. It is therefore advisable that one never really create "Upload-only projects" and always create testable ones, even if the tests do nothing of value with a student's submission. ---+++ Get into the habit of reading Log Files Whenever something unexpected happens, determine whether the tools you are using or servers you are relying on, have log files. Open them up and look through them. Most logs have time-based entries to help you home-in on error conditions. The log entries usually have integer identifiers as well, to simplify searching for problems if a particular entity (e.g. a student's submission) is experiencing problems. * The buildservers have logs under =/u/csNNN/buildserver/bsN.csNNN.cpuXX.student.cs/log/=. * The submitserver has additional logs on =services14.student.cs.uwaterloo.ca= under =/fsys2/tomcat5.5/logs/marmoset=. ---+++ Uploading a project To create and test a project, use the =$user= account rather than =${user}-student= or =${user}-canonical=. After selecting a course, you will counter-intuitively be seeing a student view from the instructor perspective; you must click "instructor view" on the top bar to do anything productive. Note also that this student view is different than the _real_ student view; *you must log in as =${user}-student= to see what the students see*! Once you've created a project (see the documentation), you can access information by clicking the 'view' link for the project under 'Projects', and then click the 'Utilities' link near the top of the screen. It can be tested from here by uploading a file under 'Canonical solutions'. *Both of these files should be JAR files*! While Marmoset will accept zip files and they will often work, occasionally they will just mysteriously fail with Marmoset claiming the zip file cannot be extracted, even though these files can be downloaded, extracted, and run perfectly fine. No such problem has yet occurred with jar files. Note also that if you are using a browser like Safari, you will normally have to click this twice to make Marmoset acknowledge you. A link will be added under 'Canonical Submissions' near the top of the page. If you click on this immediately you will see a completely unhelpful java exception; this is Marmoset's way of telling you "I'm currently testing this; try again later." After a couple of minutes, this will contain potentially helpful error output in the event of a failure. If you have a dependency on a utility outside of the Makefile that is causing the error, you may wish to click "Retest" on the canonical solution failure page instead of re-uploading the zip file. Don't bother; "Retest" appears to change the text on the page to tell you that it's retesting, but it does not actually appear to do anything. No other status will change, and there's no evidence a retest will actually ever complete. If the project configuration fails, click 'mark broken' to make the testing setup disappear. This does not appear to be undoable if you do this accidentally, as the entry disappears entirely instead of just being flagged; because of this, be *very* careful clicking anything on any page. After uploading a new one, the retesting of the canonical solution will be queued. Often this is tested relatively quickly, but particularly if some of the build servers fall asleep this can take roughly a quarter of an hour to start testing; if you give it a bit to come back from vacation and keep clicking refresh, you'll see it start to test eventually. After a test is complete, a link with appropriate text (such as "failed") will appear in the status column of the "Testing setups" section. The line at the bottom lets you know what build server was used. If you look at the path /u/$csXXX/buildserver/$bs (substituting that string for $bs), you can examine the state of that run. The =log= subdirectory contains logs for that buildserver which may give you helpful information about what happened on the test run. The =build= subdirectory is where Marmoset ran the test, so you can examine the final state of any files there. ---+++ Interpreting Marmoset build errors
I | Attachment | History | Action | Size | Date | Who | Comment |
---|---|---|---|---|---|---|---|
![]() |
Overview.png | r1 | manage | 8.4 K | 2011-04-25 - 00:40 | AlexLake | |
![]() |
RestartingMarmosetbuildservers1.pdf | r1 | manage | 132.5 K | 2015-09-04 - 11:36 | OmNafees | |
![]() |
StaticDynamic.png | r1 | manage | 12.5 K | 2011-04-25 - 00:40 | AlexLake | |
![]() |
olddoc.txt | r1 | manage | 16.9 K | 2011-04-25 - 00:39 | AlexLake |