%TOC%
---+ Marmoset

---++ Related Pages
   * LocalMarmosetSubmissionMirror
   * MarkersMimickingMarmoset
   * PrintingMarmosetAssignments
   * PostingMarks
   * MarmocreateImprovePit

---++ Documentation

Marmoset documentation for Instructors is located at https://marmoset.student.cs.uwaterloo.ca/docs/MarmosetHelp/submitserver_usersguide.html. This document contains general instructions for adding new projects (assignments) to Marmoset and for writing test cases for Marmoset.

Omar also maintains CF.MarmosetCourseConfiguration covering UW-specific setup of Marmoset and running the buildservers.

---++ Test Development Rules and Tips (specific to our computing environment)

---+++ Test properties

There is a special 'language' we use; the language should be set as:

<pre>build.language=UW</pre>

In case students submit a Makefile to be compiled, the initial Makefile you'll be uploading for Marmoset to use to set up the environment for tests should have a cryptic name.  You should also explicitly set the Make program used to the GNU version if you're running Marmoset on a Solaris server, as that's likely the version of Make which will be  used:

<pre>build.make.command=/software/gnu/bin/make
build.make.file=SomeCrypticMakefileName
</pre>

---+++ Take responsibility for the tests you write and monitor the buildservers and submitserver, especially around deadlines

Be aware that although the software that Marmoset is comprised of is maintained and run by CSCF, you are responsible for using it reasonably and within resource usage limits and guidelines established campus-wide. For example, your tests have the potential to cause an unreasonable load-increase on hosts running marmoset buildservers which violates [[http://www.math.uwaterloo.ca/mfcf/policies/usage/account.html][established policies]]. So please be aware of this, especially around deadlines for your assignments/projects. 

The buildservers for you course can be monitored as follows: 

   1 Go to =https://marmoset.student.cs.uwaterloo.ca= and authenticate with any account you have access to.
   1 Visit the =https://marmoset.student.cs.uwaterloo.ca/status/QueryBuildServerStatus= with your browser and observe alerts on possibly hanging 
     buildservers. This page auto-refreshes.


The buildservers are *not* automatically shut-down on the hosts that they run. This means that any spinning processes they may have started due to buggy tests can be a potential source of load-increase.


The submitserver can be monitored simply by visiting your courses instructor-page and observing how many tests are in the queue. If there are many student-submissions that remain to be tested and the numbers do not seem to be going down, there is a problem. 


In any case, if you have difficulty tracing the problems to your tests, contact =omnafees@cs= for assistance.


---+++ Your tests must use custom-built timeout command

You must force students code to timeout and you must use the custom-built command that is different from the standard =timeout= command.  There are two reasons for this: the default timeout will not kill child processes of the timed process, and the default may cause your program to hang unnecessarily for up to 30 seconds.

The following command starts a process and kills it if it runs too long. 

   * =~cs_build/bin/cwrapper/cwrapper.solaris8 time command=. 

For instance =~cs_build/bin/cwrapper/cwrapper.solaris8 10 command= will run =command=, if =command= runs for longer than 10 seconds, it and any child-processes it started are killed.



---+++ Your tests must use sudo and set appropriate permissions

For security reasons, i.e., to prevent student code from harming the course account, and until this is automatically enforced, your tests absolutely must use =sudo= to execute student submissions as =csNNNt=.

   * =sudo -u csNNNt command= 

The above invocation runs =command= as =csNNNt=.

By default, you will not be able to read any files created by sudo.  While appropriate permissions can be set up at startup, any careless change in permissions can prevent your tests from running.  Because of this, your Makefile should set permissions similarly to the following before any calls requiring sudo:

<pre>
   while [[ `/software/.admin/bins/bin/absolute ..` != `/software/.admin/bins/bin/absolute .` ]]; do cd ..; chmod ug+rX * 2> /dev/null; chgrp "$(whoami)t" * 2> /dev/null; done; exit 0
   chmod ug+rwx .
   chgrp "$(whoami)t" .
   chgrp -R "$(whoami)t" *
   chmod -R ug+rx *
   chmod -R a-w *
</pre>

To make this work properly, you should make sure that your Makefile is executed in bash.  To enforce this, one of the initial variables you set at the start of the Makefile should be:

<pre>
SHELL = "/xhbin/bash"
</pre>


---++++ Your test files must be writeable

This is a weird one.

Marmoset was throwing completely unhelpful errors of the form
<pre>Compile/Build unsuccessful
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;edu.umd.cs.buildServer.BuilderException: Couldn't build project</pre>

Dropping temporary files revealed that the Makefile was completing successfully, but that none of the tests (listed in =test.class.*= in the =test.properties= file) were being executed.
The buildserver log file came to the rescue here, with the more helpful information:
<pre>ERROR - IOException trying to build submission 360561
java.io.FileNotFoundException: [...path...] (Permission denied)</pre>

Permissions were loosened from 500 (user-only read and execute) to 555 (everybody read and execute), but the test still would not run.  As soon as the permissions were changed to 755, Marmoset happily ran them.

So, even though there should be no good reason to write test files, make sure your Makefile makes them writeable.  Further testing will be needed to determine if permissions like 750 or 700 are also acceptable.

---++++ Your tests must be _paranoid_ about permissions

Not only do you need to ensure that files are readable from sudo (meaning enforcing group permissions beforehand), you must also ensure that files are writeable from the course account (meaning enforcing group permissions through sudo after testing is done).  The current assumption is that because Marmoset reuses hardcoded build directories instead of a subdirectory of =/tmp=, it *must* have permission to modify all files in that build directory as the course account user.  If it does not, it will die with an unhelpful error message; although Java exceptions regarding file permissions written are to the log,  any indication of why it cares is not.


---+++ Your tests must use valid exit codes

The test executable must exit with one of the four exit codes to inform Marmoset of the outcome of the test. The exit codes are:

| *exit code* | *outcome* | *meaning* |
|   0   |   Correct   | student's code passed the test |
|   1   |   Error   | student's code crashed |
|   2   |   Timeout   | student's code ran too long |
|   3   |   Failed   | student's code produced the wrong answer |


---+++ Updating the buildserver runtime environment variables

While logged in as the course account:

   1 Modify =~csNNN/.marmoset_buildserver_environment= and add *bash-style* =export= commands to set the variables you need.
   1 Restart the buildservers by executing the following command:
      * =~cs_build/bin/buildserver_hosts_command pkill -u csNNN java=


---+++ Avoid using =student.cs= for any sort of work

Do *not* use =student.cs= for any sort of development or testing.  See [[TAUnix#AvoidingCertainHosts]] for more information.





---++ Using Marmoset

---+++ Avoid creating "Upload-only" Marmoset Projects

Marmoset allows one to create projects where students can submit files with no expectation of testing. Converting such projects into regular testable projects, after students have started to submit to them, requires a bit of database hackery that may result in more harm than good. It is therefore advisable that one never really create "Upload-only projects" and always create testable ones, even if the tests do nothing of value with a student's submission.


---+++ Get into the habit of reading Log Files

Whenever something unexpected happens, determine whether the tools you are using or servers you are relying on, have log files. Open them up and look through them. Most logs have time-based entries to help you home-in on error conditions. The log entries usually have integer identifiers as well, to simplify searching for problems if a particular entity (e.g. a student's submission) is experiencing problems.

   * The buildservers have logs under =/u/csNNN/buildserver/bsN.csNNN.cpuXX.student.cs/log/=. 
   * The submitserver has additional logs on =services14.student.cs.uwaterloo.ca= under =/fsys2/tomcat5.5/logs/marmoset=.


---+++ Uploading a project

To create and test a project, use the =$user= account rather than =${user}-student= or =${user}-canonical=.  After selecting a course, you will counter-intuitively be seeing a student view from the instructor perspective; you must click "instructor view" on the top bar to do anything productive.  Note also that this student view is different than the _real_ student view; *you must log in as =${user}-student= to see what the students see*!

Once you've created a project (see the documentation), you can access information by clicking the 'view' link for the project under 'Projects', and then click the 'Utilities' link near the top of the screen.  It can be tested from here by uploading a file under 'Canonical solutions'.  *Both of these files should be JAR files*!  While Marmoset will accept zip files and they will often work, occasionally they will just mysteriously fail with Marmoset claiming the zip file cannot be extracted, even though these files can be downloaded, extracted, and run perfectly fine.  No such problem has yet occurred with jar files.  Note also that if you are using a browser like Safari, you will normally have to click this twice to make Marmoset acknowledge you.  A link will be added under 'Canonical Submissions' near the top of the page.  If you click on this immediately you will see a completely unhelpful java exception; this is Marmoset's way of telling you "I'm currently testing this; try again later."  After a couple of minutes, this will contain potentially helpful error output in the event of a failure.  

If you have a dependency on a utility outside of the Makefile that is causing the error, you may wish to click "Retest" on the canonical solution failure page instead of re-uploading the zip file.  Don't bother; "Retest" appears to change the text on the page to tell you that it's retesting, but it does not actually appear to do anything.  No other status will change, and there's no evidence a retest will actually ever complete.

If the project configuration fails, click 'mark broken' to make the testing setup disappear.  This does not appear to be undoable if you do this accidentally, as the entry disappears entirely instead of just being flagged; because of this, be *very*  careful clicking anything on any page.  After uploading a new one, the retesting of the canonical solution will be queued.  Often this is tested relatively quickly, but particularly if some of the build servers fall asleep this can take roughly a quarter of an hour to start testing; if you give it a bit to come back from vacation and keep clicking refresh, you'll see it start to test eventually.

After a test is complete, a link with appropriate text (such as "failed") will appear in the status column of the "Testing setups" section.  The line at the bottom lets you know what build server was used.  If you look at the path
<pre>/u/<strong>$csXXX</strong>/buildserver/<strong>$bs</strong></pre>
(substituting that string for $bs), you can examine the state of that run.  The =log= subdirectory contains logs for that buildserver which may give you helpful information about what happened on the test run.  The =build= subdirectory is where Marmoset ran the test, so you can examine the final state of any files there.

---+++ Interpreting Marmoset build errors

<dl>
 <dt>edu.umd.cs.buildServer.BuilderException: test script returned unknown exit code: 127</dt>
 <dd>127 is the bash shell's exit code when a file you attempt to run does not exist.  This likely means that one of the tests listed in your =test.properties= file is not actually in the build directory (which means it was probably not in the zip file you uploaded).</dd>

 <dt>edu.umd.cs.buildServer.BuilderException: test script returned unknown exit code: 126</dt>
 <dd>126 is the bash shell's exit code when you attempt to run a file that does not have execute permissions.  This means you likely need to add appropriate chmod (and potentially chgrp) commands to your Makefile to ensure that all test files in the build directory have appropriate permissions before Marmoset tries to run them.</dd>

 <dt>edu.umd.cs.buildServer.BuilderException: Couldn't build project</dt>
 <dd>This is a useless catch all message that translates into "look at the buildserver logs"

It can signal that Marmoset does not have write permission on files it does not need to write (see above), which has absolutely nothing to do with whether or not the project was built.</dd>
</dl>

---+++ Post-Upload

After uploading a successful canonical submission, you will be allowed to assign marks for each test by clicking the appropriate link under the "Activate/Inactivate" heading.  Once you have done this, *never click here again*.  If you accidentally click inactivate, the project becomes inactive again, and there's no clear way to re-activate it other than uploading the canonical solution and re-assigning marks yet again.

This would be far less irritating if there were an automated way to do mark assignment.  No such way is currently known.



---++ Adding Group Accounts

If your students are working on an assignment that allows them to work in groups, and they are made to be a part of a specific UNIX group on =student.cs= (e.g. =cs246_NN=), then these instructions will help you set up Marmoset accounts for group submissions. It is assumed that the UNIX groups have already been assigned to the students. 

The examples given will use cs246, but make sure you swap that course id for whatever course you are using.

*NOTE:* When you download the submissions from Marmoset, their folders will begin with dashes, which causes problems with basically every UNIX command. To rename them to not have the dashes, enter something like the following at a bash command prompt in the directory which contains all submissions (works for partners): 

=for dir in -*; do mv -- "$dir" "`echo $dir | cut -f2,3 -d-`" ; done=

---+++ How To

Since Marmoset supports adding accounts by uploading a classlist file, we will generate a classlist file that will add the group accounts. If users =pbeshai= and =tblancha= are part of the same group, say =cs246_01=, then their Marmoset group id is =-pbeshai-tblancha-=. Their entry in the class list:
=99999999:-pbeshai-tblancha-:Group cs246_01 - pbeshai, tblancha:000:study:plan:group:0:degree:initials:family:R:F:1:lec=000,tut=000,tst=000=. 

The following scripts will generate the classlist for you. All you need to do is replace cs246 with whatever course you're working on and then upload the outputted classlist to marmoset. Run the scripts by typing: =./marmosetGroupsClasslist > groupsClasslist=. Then upload the outputted file =groupsClasslist= to Marmoset.

Note that these scripts make some necessary formatting tweaks; for example, the third colon-delimited field (officially the "name" field) *must* contain a comma character, which is why =marmosetGroupsClasslist= explicitly includes one.  You must make sure that any data you modify yourself retains the comma.

---+++ Necessary Scripts

---++++ listGroups
<pre>
#!/bin/ksh
#       Lists the groups in cs246 and their members

grep "^cs246_" /etc/group | sort | cut -d: -f1,4
</pre>

---++++ marmosetGroups                                                                
<pre>
#!/usr/bin/env bash
#
# Translates unix groups into Marmoset user group syntax
#
# e.g. cs246_46:c32liu,yh2zhao becomes -c32liu-yh2zhao- 

for group in `listGroups | cut -d: -f2 | sed 's/,/-/'`; do
    echo -$group-
done
</pre>

---++++ marmosetGroupsClasslist
<pre>
#!/usr/bin/env bash
#
# Translates unix groups into Marmoset user group syntax then into classlist format
# The classlist can then be uploaded to Marmoset 
#
#id:userid:name:lec:study:plan:group:year:degree:initials:family:status:time:session:sections

echo "# cs246 classlist updated: Thursday May 28 10:14:23 2009"
echo "# Do not make changes to this file."
echo "# It will be automatically overwritten by the next update."
echo
echo "#Id:Userid:Name:Lecture:Study:Plan:Group:Year:Degree:Initials:Family:Status:Time:Session:Sections"
for group in `listGroups`; do
     
    userid="-`echo $group | cut -d: -f2 | sed 's/,/-/'`-"
    groupid=`echo $group | cut -d: -f1`
    member1=`echo $userid | cut -d- -f2`
    member2=`echo $userid | cut -d- -f3`
    echo "99999999:$userid:Group $groupid - $member1, $member2:000:study:plan:group:0:degree:initials:family:R:F:1:lec=000,tut=000,tst=000"
done
</pre>