Public Marks Uploader (Deprecated)

The following system for automatically uploading marks to the course website was deprecated in Spring 2018. Currently mark uploading is done manually. If you want to write a system for automatic mark uploading, you may find it useful to read about the design of this system. In particular, it's probably not enough to just set up a cron job that runs the existing scripts - an automatic system should have a mechanism to detect when marks fail to download properly (due to a connection error, for example) and avoid overwriting the existing mark data. This system used a verification script for this purpose.

---

The CS 241 website has a page that allows students to view their current assignment and midterm marks, using the displayMarks.py script that is used by many courses (see OnlineMarkUpload). This script reads marks from a file called public-marks.txt stored in /u/cs241/marks, which is intended to be created manually using Excel. This page describes a system for automatically downloading marks from Marmoset and generating the public-marks.txt file. Prior to the creation of this system, the method described on OldPublicMarks was used.

How It Works

The system consists of six scripts:

• pubmarks.sh, the main script that downloads all the marks from Marmoset and generates the public-marks.txt file
• verify-pubmarks.sh, a script for verifying that the generated public-marks.txt has the correct format
• pubmarks-wrapper.sh, a script that calls pubmarks.sh then calls verify-pubmarks.sh, and overwrites the main public-marks.txt file with the newly generated one if the verification passes
• marm_sql (see MarmSql), used by pubmarks.sh as a helper script to download the marks for each Marmoset project in .csv format
• gen_csv.pl, another helper script used by pubmarks.sh to combine the .csv files for individual projects into .csv files for assignments
• pubmarks-status, a script that reports the current progress of mark downloads

The pubmarks.sh, verify-pubmarks.sh, pubmarks-wrapper.sh and gen_csv.pl scripts are all stored in /u/cs241/marks/bin. The marm_sql and pubmarks-status scripts are stored in /u/cs241/bin so they can be run from anywhere on the course account.

The system is started from the CS 241 crontab, which allows tasks to be automatically executed at scheduled intervals (see the Crontab section in CompSci241ISADuties). The following command is executed

/u/cs241/marks/bin/pubmarks-wrapper.sh

The first thing pubmarks-wrapper does is check whether pubmarks.sh is already running. If it is running, then the wrapper exits quietly. If it's not running, the wrapper calls pubmarks. This was put in place when the marks were to be updated every minute. The automatic update allowed students to see their test results indirectly without waiting for Marmoset tokens so now the command is executed a few times a week instead.

All output for pubmarks.sh will be in /u/cs241/marks/bin/pubmarks-dir, so the contents of pubmarks-dir can be cleared if there are any problems

The pubmarks.sh script downloads marks for each CS 241 assignment by calling the marm_sql script as follows, for each number N from 1 to 10:

marm_sql -m N

If you try to call marm_sql with a project that doesn't exist (e.g. if you do marm_sql -m 11 before the Assignment 10 projects are uploaded), it will produce an error message. However, it is likely that an ISA will need to enable the mark upload system before uploading every project to Marmoset, so the pubmarks.sh script is designed to ignore this error message and give everyone a mark of 0 for non-existing projects. You should not have any problems enabling the system before every Marmoset project has been uploaded.

After downloading marks for each project on an assignment, the gen_csv.pl script is used to combine these project marks into assignment marks. After all the marks are downloaded and .csv files have been created containing marks for each assignment, pubmarks.sh creates the public-marks.txt file. The script takes each student in the classlist and looks up their names in the assignment mark .csv files, and outputs a line indicating that student's mark for each assignment. It also looks under /u/cs241/marks/midterm.csv to find the student's midterm mark; if this file does not exist yet, then a mark of 0 is output. This midterm.csv file needs to be created manually by the ISA after the midterm marks are entered (more details in the next section).

After pubmarks.sh is finished generating the public-marks.txt file, the pubmarks-wrapper.sh script then calls verify-pubmarks.sh to check that the generated public-marks.txt file is valid. The following things are checked by verify-pubmarks:

• The first line is precisely ,A1,A2,A3,A4,A5,A6,A7,A8,A9,A10,A11,,Midterm
• The second line has the format ,N,N,N,N,N,N,N,N,N,N,N, ,N where each N is some number (they are not necessarily all the same number)
• For each student in the classlist, there exists a line of the form U,N,N,N,N,N,N,N,N,N,N,N, ,S where U is the student's user ID, each N is some number, and S is a string of non-comma characters (the reason the midterm mark S can be a string is because things like "DNW", "dnw", "Did not write" should be allowed as well as numbers)

If verification passes, the final thing pubmarks-wrapper does is overwrite the current public-marks.txt file in /u/cs241/marks with the newly generated public-marks.txt. The marks will then be immediately visible to students on the course website.

If you want to check what the pubmarks.sh script is currently doing, the pubmarks-status script can tell you. This script is stored in the CS 241 course account's bin directory, so you should be able to run it from anywhere by just typing pubmarks-status. The script is rather limited; it simply looks at the last-modified times of files in /u/cs241/marks/pubmarks-dir to determine which .csv file containing project marks was most recently updated. It will always display a message of this form:

Currently downloading marks for A2P6.
129 students processed.

It will not display a different type of message when the script finishes downloading all the marks and starts generating the public-marks.txt file, or when the script is idle and waiting to be called by pubmarks-wrapper . This would be a nice addition. Also, if called from a server other than the one where pubmarks.sh is actually executing, it may give slightly inaccurate results simply due to the time it takes for changes to propagate to the other servers.

Usage

To use this script, you simply need to run the /u/cs241/marks/bin/pubmarks-wrapper.sh script whenever you need to update the marks on the Public Marks page for students to see. The crontab will do this for you a few times a week on the days you specify. You may also run this script manually after the midterm marks are uploaded, or if there are holidays which change the day an assignment is due for a particular week.

You can confirm that the system is working by visiting the public marks page. Add the string ?user=USERID (where USERID is the user ID of some student from the classlist) to view a student's marks, then compare the marks on the page with their marks on Marmoset. Viewing a student's marks will only work if you have set up the .coursestaff file correctly (see the Beginning of Term section in CompSci241ISADuties). If the .coursestaff file is not set up correctly, you should be able to tell easily as the public marks page will say "Contact the ISC, None , if you have questions about a UR grade" (rather than displaying the ISC's name).

Uploading Midterm Marks

After the midterm, edit the pubmarks.sh script so that the header of the public-marks.txt file it outputs displays the correct maximum mark for the midterm. Look for the following lines:

# output first two lines of public-marks
echo ",A1,A2,A3,A4,A5,A6,A7,A8,A9,A10,A11,,Midterm"
echo ",60,61,60,60,60,60,60,60,55,60,60, ,43"

The last number on the last line should be changed to the maximum mark that students could obtain on the midterm. (Or if there are bonus marks and students can get above 100%, it should be whatever the "total" mark listed on the front of the midterm was.)

This ensures the header is correct, but we also need to make pubmarks.sh actually output each student's midterm mark. The script looks for midterm marks in the file /u/cs241/marks/midterm.csv. This file is just a series of lines of the form user_id,midterm_mark. Assuming you have entered the midterm marks in the Excel marks spreadsheet, you can create this file as follows:

• Create a new sheet.
• In the first column, copy and paste the list of all student IDs from the "Midterm" sheet.
• In the second column, copy and paste the list of marks from the Midterm sheet (or use VLOOKUP). If you are not using VLOOKUP, be very careful that you line up the students with their marks correctly.
• Select "Save As", and for the file format select "Comma Separated Values (.csv). Do not save the file as midterm.csv! Excel does not use Unix-format newlines in the .csv files it outputs, and this causes problems for pubmarks. Pick another name (say, midterm-excel.csv).
• Generate the midterm.csv file using the following bash pipeline:

  ( cat midterm-excel.csv | tr '\r' '\n' && echo ) > midterm.csv     

  Note that the fromdos or dos2unix commands will not work here, because Excel does not output .csv files in Windows/DOS format (carriage return + line feed). Instead, it uses a format where each line (except for the last) ends with a carriage return, but no line feed. The tr '\r' '\n' command replaces all carriage returns with line feeds, and the echo adds one last line feed to the end. Enclosing the pipeline in brackets causes the entire output of the pipeline to be sent to midterm.csv; otherwise only the output of echo would be sent. This converts the file to Unix format (line feed at the end of every line, including the last line).
• Alternatively, you can save the original file as midterm.csv and then use a Unix text editor to replace all newline characters with proper Unix-style newlines. In Vim, this can easily be done with the command :%s/\r/\r/g.

Once you have set up this file, next time the pubmarks.sh script finishes running it should grab the midterm marks and include them in the public-marks.txt file it generates.

Uploading Handmarking Marks

Place a file called a4-handmark.csv or a9-handmark.csv (depending on the assignment) into the same directory as pubmarks. The file should be a csv file downloaded from Markus after the handmarking has been completed (Submissions -> CSV Report), without any changes. The handmarking marks will be automatically added to the students' public marks the next time pubmarks-wrapper.sh is run.

Uploading Quiz Marks

Some offerings of the course will have tutorial quizzes that are worth marks. To have the mark uploader output a line for the quizzes, have the file quiz.csv in /u/cs241/marks. It is recommended that the marks are stored on a spreadsheet.

• Create a new sheet.
• In the first row, label the columns: , , Quiz1, .... , Total.
• In the second row, label the first two columns Name, ID and fill out the maximum marks for each of the quizzes. Where Name is the questID (truncate to 8 characters), ID is the student number.
• Fill out the Name and ID columns (you can open up /u/cs241/.classlist and get it from there)
• To generate quiz.csv , follow the steps in the Midterm section above

Look for the lines in pubmarks.sh

QUIZTOTAL=$(grep "^Name," /u/cs241/marks/quiz.csv | cut -f13 -d,)



QUIZ=$(grep "^$USERID," /u/cs241/marks/quiz.csv | cut -f13 -d,)

and change -f13 to -f# where # is the Total column. (In the example above, the thirteenth column was the total column). You will have to generate quiz.csv again after each quiz is marked.

Adding Exceptions

There will be times where a student would get an assignment remarked. Some reasons could be: printing an extra character (which fails the tests), writing an infinite loop by accident but otherwise code is correct or code works when they test it but not on marmoset. It is up to the instructor whether a student should get the marks. Since pubmarks get the marks directly from marmoset each time it is ran, any manual changes you make to public-marks.txt will not persist.

To give a student more marks, run the command:

add_exception questid assign# mark

Where questid is the student's questid, assign# is the assignment number that the student is being remarked and mark is the additional marks that the student got. For example

add_exception k29wu 5 10

would grant me (k29wu) 10 more marks on assignment 5 the next time pubmarks is ran.

These remarks are stored in the hidden file /u/cs241/marks/.except and the contents of this file should be reset at the end of each term.

Troubleshooting

If something goes wrong, the public mark upload system may send an email to the course account detailing the problem. Also, students may notice their marks are incorrect and complain about it. Here are some explanations of errors that may occur and suggestions for resolving them. If you run into any errors that are not documented here, update this section with notes on how to resolve the error.

General Issues

The following tips apply in many situations.

• If the mark upload system suddenly becomes "broken" in some way—e.g. constantly producing error messages, creating a public-marks.txt with incorrect mark values—disable it temporarily by commenting out the line that calls pubmarks-wrapper.sh in the crontab.
• If the public-marks.txt file becomes incorrect or corrupt, after disabling the system, look in the .snapshot directory for /u/cs241/marks and try to find a earlier version of public-marks.txt that is correct. Replace the incorrect/corrupt public-marks.txt with the older version so students can still check their marks while you are fixing the error.
• Post an announcement if you have to disable the mark system temporarily or revert to an older marks file. This will reduce the number of emails you get.
• You can do a manual run of the system by navigating to /u/cs241/marks/bin and executing the following commands:

  ./pubmarks.sh -a > public-marks.txt
  ./verify-pubmarks.sh public-marks.txt     

  This will generate a new public-marks.txt file in the current directory and check it for errors without overwriting the real one in /u/cs241/marks. Alternatively, comment out the line in pubmarks-wrapper.sh that copies the generated file over the one in /u/cs241/marks and then just type ./pubmarks-wrapper.sh. You can run the system manually like this until you are convinced it is working again, and then re-enable the line in the crontab that runs it automatically.

Error reading input file: Stale NFS file handle

This error message sometimes occurs when you modify any of the scripts used by the public mark upload system (including marm_sql) while the scripts are running. This error can be safely ignored. Alternatively, it seems like if you make sure you are on the server where the scripts are running (i.e. the server where the crontab is stored) before you modify the scripts, you won't get this error.

A student's mark does not match their Marmoset mark

If a student emails you with a complaint like this, first check that their claim is actually true. Sometimes students may make a mistake when adding up the marks for the Marmoset tests they passed.

If their claim is true, check the last-modified time on the public-marks.txt file using ls -l. It's possible that the public-marks.txt file just has not been updated to reflect the most recent Marmoset results.

If the file was modified after the student's email was sent, then it's likely the mark is not being updated correctly. It's hard to say what could cause this to happen. Try following the instructions in the "General Issues" section above to disable the auto-updating and test the scripts manually. The public mark upload system relies on MarmSql, so there could be a bug in the marm_sql script—it may be a good idea to test the mark-downloading function of marm_sql manually as well.

The verify-pubmarks.sh script is rejecting the generated file

This should really only happen if some freak accident causes the generated file to be corrupted. In fact, the purpose of verify-pubmarks.sh is to prevent the main public-marks.txt file from being overwritten if a random corruption happens. You should usually be able to ignore these errors unless they start happening repeatedly and are preventing the marks from being updated. If so, follow the instructions in the "General Issues" section to disable auto-updating and test the scripts manually. Examine the generated public-marks.txt file to figure out which part is invalid.

One reason verify-pubmarks.sh could repeatedly produce error messages is if some assumption that was made when the script was written has changed in recent terms. For example, maybe an additional assignment was added. You will need to modify pubmarks.sh to change the format of the public-marks.txt file it produces. However, you will also need to modify verify-pubmarks.sh, or it will repeatedly reject your public-marks.txt files for not following the old format. The pubmarks.sh and verify-pubmarks.sh scripts are very tightly coupled; if you change the output format of pubmarks.sh at all you will almost certainly need to change verify-pubmarks.sh to recognize the new format.

Potential Improvements

One fairly serious error that is currently not caught by verify-pubmarks.sh is if a student's mark mysteriously decreased or was overwritten with 0, but each line still had the correct format. This would be a good addition for future terms. However, it's possible there could be some valid reasons for a student's mark to decrease (for example, if there is an error in the tests and submissions are mass-retested), so this case would need to be handled carefully. Here is a proposed solution ( not currently implemented):

• If a student's mark in the newly generated public-marks.txt is lower than their mark in the current public-marks.txt, produce an error message (which cron will send to the course email account) detailing which student and which mark, and reject the generated public-marks.txt ( but don't delete it!).
• Stop the public mark upload system temporarily. We don't want the ISA to get the same error message over and over. One easy way to stop the system would be to create a file called PUBMARKS_STOP, and then have pubmarks-wrapper look for this file and exit silently (without calling pubmarks) if the file exists.
• While the system is stopped, the ISA can manually verify whether the mark change is correct. If it is correct, the ISA can overwrite the current public-marks.txt with the generated file. If it is incorrect and was caused by an error in the public mark upload scripts, the ISA can try to identify and fix the error.
• Once the ISA has dealt with the issue, they can delete the PUBMARKS_STOP file and the system will automatically resume.