Utilizing submit / public test logs

Assuming the course account is set up properly as described in PreparingSubmitAtStartOfTerm (particularly with the use of ISG's SubmitFileFilter and SubmitSubversionHook), you will be able to see a number of activity logs. This will allow you to track information such as successful and unsuccessful submit attempts (along with an indication of why files are rejected if this is the case) for each student, the number of students who requested public tests for a given assignment, and the marks achieved on various public test requests.

Submission log examples

Submission logs are found in files following the pattern /u/csXXX/course/submitlog/assignment/student. The submission logs are kept as CSV files with the following fields (as specified in the man page):

filename,timestamp,filter exit code,file size,MD5sum of file

If the filter exit code is non-zero, it should specify that a particular error happened; in the case of the ISG filter, it is guaranteed that the file was rejected and the student was given an explanation. A meaning of the error codes is given in the man page for file_filter. If a student is claiming they submitted a file but, for example, the public tests are claiming the file cannot be found, this log should be checked to see if submit / Odyssey would have indicated a rejection message.

In the unfortunate case that a student's submission accidentally gets deleted (which is exteremely rare, but unfortunately is quite possible given the unfortunate permissions that submit leaves behind on submission directories), the combination of MD5sum and file size can allow for a determination of whether files that are subsequently given to course staff match those that were submitted by the deadline. It also should be possible to svn checkout the repository /u/csXXX/course/submitlog/assignment/student.svn and grab the files directly; this should reflect what the contents of the handin folder would have been as of the last submission (verifiable by checking md5sums and exit status codes). Of course, if a submission from the student with timestamps after the deadline is visible in .snapshot, that should be used as an authoritative source to restore the assignment.

The subversion logs could also potentially provide some evidence as part of an academic integrity violation investigation.

Public Test log examples

The public test logs are kept in the files /u/csXXX/course/publictests/termcode/assignment.{completedrun,run}.log. There is also a requesets directory at the same level of termcode which contains files indicating pending requests. When a test runner reads a request, it records a timestamp, the userid, and its identification in a tab-delimited run.log file. After completion, it logs the same information but with an updated timestamp and a mark for the student (if it can determine one) in completedrun.log. The use of the command writeToMarkList mark inside of computeMarks for a given RST suite will guarantee a mark is output that the logger understands.

The first obvious use of this is to check whether a student submitted any public test request for a particular assignment (when handling remark requests potentially, for example). This would be done by issuing the command grep "[[:space:]]student[[:space:]]" path/to/assignment.run.log | wc -l. The number printed represents the number of requests the student made.

Of potential interest is the history of marks obtained on the public tests for a given student; this could be obtained with the command grep "[[:space:]]]student[[:space:]]" path/to/assignment.completedrun.log | cut -f4.