BitterSuiteMarkSchemeWalkthrough < ISG

%TOC%
---+ Marking Scheme Walkthrough

---++ Introduction

One simple and easily overlooked element is a marking scheme. 
This will be a plain text file which will be dumped verbatim as the first portion of the output for students. 
This file must be in the directory =/u/<em>csXXX</em>/marking/<em>assignment</em>/<em>suite</em> and be called mark-scheme. 
For example, for suite =0= of assignment =a04= in cs125, this file would be =/u/cs125/marking/a04/0/mark-scheme=.
Normal content will include a header where the marker can write his/her initials and the final mark, and a body with the breakdown of all marks allocated for the assignment. Try putting in some arbitrary text to see the effect on the output for students.

The information contained in this walkthrough up to and including the _Using shell interpretation to import autotesting marks_ section is sufficient to design decent-looking and informative marking schemes.  The information afterwards is primarily of interest to power-users who wish to take advantage of scripting capabilities to reduce the amount of time required to create marking schemes and/or to reduce the potential for making errors when constructing the marking scheme.


---++ Content Considerations

This marking scheme will be the first thing shown to students, 
and should contain a brief summary of how marks are allocated for the assignment and what the student earned on each portion (typically filled in by hand by a marking TA).  
Some general style pointers to take into consideration to facilitate readability are given in MarkingScheme; note that in order to make marking more effective, some [[CommunicatingWithMarkers][additional marking information]] should be provided to the TAs.  If this suite is being used for RSTPublicTests, then take into consideration the suggestions at MarkSchemesForRSTPublicTests.


---++ Using nroff for style

If the =nroff-mark-scheme= option is set to true in =config.ss= in the suite directory (see BitterSuiteInitialSetup), then the provided marking scheme will be formatted with nroff.  You can then use nroff formatting commands to format the output for the TAs, as discussed in MarkingScheme.  BitterSuite will automatically provide the following nroff formatting commands before the start of the marking scheme processing as a courtesy:

<pre>
.ll ${linelength}
.ls 1
.pl 1
</pre>

These commands set the length of each line appropriately (dependent on whether or not RST is generating printouts), directs the output to be single-spaced, and sets the page length to 1 to prevent nroff from generating a large amount of unnecessary trailing whitespace.

The reason to use nroff is that it will help auto-format the marking scheme for you independent of the width.  
This gives you a flexibility not possible with plain ASCII test.
The primary feature it provides is automatic line-wrapping so words will appear cleanly on the next line if you go beyond the maximum width.
It also allows you to set indents, so you can make submarks of questions stand out cleanly in the marking scheme without spacing yourself.


---++ Using shell interpretation to import autotesting marks

Another option that can be configured in =config.ss= is =interpret-mark-scheme=, which instructs BitterSuite to have the shell interpret the marking scheme as it is being output.

If the marking scheme is entirely enclosed within double-quotes, then the effect of this option is that the shell will substitute values for any specified variables in the marking scheme.  The reason this is potentially useful is that the autotesting results are exported for use in the marking scheme in a series of variables whose names map to the testing directory hierarchy.  This allows, for example, the question 1 autotesting results to be included as a subcomponent of all of question 1's marks, making the marks earned for each question much more clean.

The total earned marks are available in the environment variable $te, and the total possible marks ("out of") are stored in the environment variable $to.  To obtain any subdirectories, the directories are listed in order with underscore as a separator in the variable name.  So, the earned mark for a test located at =in/1= would be stored in $t1e, a test at =in/2/a= would be in $t2_ae, a test at =in/3/treetest/emptyleafnodes= would be in $t3_treetest_emptyleafnodese, and so on.

A sample mini-scheme would be as follows:
<pre>"--- Marking scheme for 1-question assignment ---

Q1: ____ / 10
  Design recipe: ____ / 4
  Autotesting:    $t1e / $t1o"</pre>


---+++ Combining with nroff

If the =-n= flag is also provided, this can also be combined with =nroff= tags:
<pre>".ad c
===================================================================
.br
CS 135 Fall 2009 Assignment 4
.br

Marked By:

.br
TOTAL    /123
.br
===================================================================
.br
.ad l
.in 13

.ti -13
___/15  Question 1: Finding robots

.ti -9
___/1 Correct Contract

.ti -9
___/1 Purpose: description of the program with reference to the values consumed and produced

.ti -9
___/4 Examples and Tests: At least 4 examples and tests in total which test all conditional expressions

.ti -9
___/3 Style (readability, white space, parameter names, layout, etc.)

.ti -8
$t1e/$t1o Correctness

[...]
"</pre>

See MarkingScheme for more discussion about formatting in this fashion.


---++ Advanced Shell Interpretation: Scripting  

*WARNING*: The following information is relatively complex.
It is likely that everything you will be using to create your marking schemes is in the section above.

The =-i= option gives much more power than just importing autotesting marks.  Specifically, this runs your marking scheme through the folowing command:

=eval echo "`cat ${provided_mark_scheme}`"=

The shell that will be used to evaluate the marking scheme is bash.

A more powerful option is to provide a script to be interpreted to produce a marking scheme.  This could, for example, store the marks each component is out of and sum them automatically, reducing the likelihood that anybody on course staff will make an arithmetic error and provide the wrong total for the marking scheme.  Instead of being enclosed in double-quotes as in the simple substitution example, the script should begin with =$(= and end with =)=.


---+++ Marking Scheme Math

If, for example, the components of a mark for question 1 were the autotesting results and the design recipe, then a total mark for question 1 could be pre-calculated by having the shell perform the arithmetic automatically:

=q1tot=$(( q1o + q1dr ))=

However, the shell typically does not support floating point arithmetic (and as of the time of this writing, the student.cs environment does not).  Because of this, it is much safer to pass the work to bc.  But, bc does not always format numbers very nicely; as such, they should then be passed to some sort of postprocessor.  This process can be simplified by defining the following function before doing any calculations:

<pre>
         domath () {
            # Restrict to two trailing decimal points for the floating point value,
            # then chop off trailing zeroes, then chop off a trailing decimal point.
            echo "scale = 2; $1" | bc | sed -e 's/\(\.[1-9]*\)\(0*\)$/\1/g' | sed 's/\.$//g'
         }
</pre>

Then the calculation of q1tot may be expressed as:

=q1tot=$( domath "q1o + q1dr" )=


---+++ Marking Scheme Output

Due to the nature of the echo statements that the marking scheme passes through, newlines will be swallowed.  As such, it is necessary either to make the output marking scheme less readable initially by using character codes, or to provide some post-processing instructions.

The following code allows more natural expression of the marking scheme itself:
<pre>echo "
This is a multi-line marking scheme.
When it is used in the student output,
the newlines contained within it
will be preserved properly.
" | perl -ne 'chomp; print "$_\\n";'</pre>



---++ A Combined Example

The following example taken from the man page illustrates a marking scheme combining the use of nroff and scripting facilities:

<pre>
 #!/bin/bash
$(

domath () {
   # Restrict to two trailing decimal points for the floating point value,
   # then chop off trailing zeroes, then chop off a trailing decimal point.
   # Note: because of truncation, do not accumulate domath results.
   echo "scale = 2; $1" | bc | sed -e 's/\(\.[1-9]*\)\(0*\)$/\1/g' | sed 's/\.$//g'
}

q1dr=4
q1tot=$( domath "$t1o + $q1dr" )

q2dr=5
q2tot=$( domath "$t2o + $q2dr" )

alltot=$( domath "$q1dr + $q2dr + $to" )

# The innermost echo allows preservation of space for evaluation later
# by replacing them with hex codes.
# This delayed evaluation is handled by the echo that is echoed by
# 'eval echo'
# There really must be a simpler way to do this...
tempvar=$(eval echo "echo -e \"$(echo "
.ti 5
This is a marking scheme that incorporates both nroff
directives and shell interpretation, including the pre-assignment of variables
that are used internally.  This should help ensure, for example, that course staff
doesn't make any arithmetic errors when creating assignment totals.

.ad c
=========================================

Total Mark Assigned:   _______  / $alltot

Marked By:  ________

=========================================

.ad l
.in 8
.ti -8
Q1: _____ / $q1tot

.ti -4
Design recipe: $(printf "%11s" "_____ / $q1dr")

.ti -4
Autotesting: $(printf "%13s" "$t1e / $t1o")

.ti -8
Q2: _____ / $q2tot

.ti -4
Design recipe: $(printf "%11s" "_____ / $q2dr")

.ti -4
Autotesting: $(printf "%13s" "$t2e / $t2o")
.ti 12
" | sed 's/ /\\x20/g')\"" | perl -ne 'chomp; print "$_\\n";' )

# Debug
#echo $tempvar >& 2
#echo -e $tempvar >& 2

echo $tempvar
)
</pre>
This topic: ISG > WebHome > ISGScripts > Testing > BitterSuite > BitterSuiteGeneralUse > BitterSuiteMarkSchemeWalkthrough
Topic revision: r9 - 2010-08-27 - PeterBeshai