MarkUs Issues

The page describes issues you might encounter when using MarkUs. Some of these issues are not caused by MarkUs per se, but are rather caused by the way MarkUs is installed and/or some other software not interacting properly with MarkUs.

For ISAs: Do not do anything with the MarkUs account, source code or server unless you have a very good understanding of what you're doing (ideally you should have a good understanding of Linux, Apache, Ruby, and Rails). Furthermore, you should get permission from CSCF and your ISC before doing anything. (CSCF has to give you ssh access to the MarkUs server so that you can restart the MarkUs mongrels to see your changes.) If you make a big enough mistake, it could affect every course that uses MarkUs and Marmoset (the Marmoset web interface is run off of the same server as MarkUs).

MarkUs 0.11.0

As of April 2015, all courses using MarkUs are using MarkUs version 0.11.0. The following are issues you may encounter when using this version.

Logout does not work

Term(s) issue occurred in: Issue occurred in Fall 2014 and Winter 2015 (and possibly before Fall 2014 too). Fixed for CS135 Fall and Winter instances.

Description/Background: MarkUs web pages all have a "Log out" link at the top right, but that link does not work.

Expected behavior that isn't happening: Clicking the "Log out" link should log the user out of MarkUs (and redirect them somewhere reasonable, like the UW CAS "Logout successful" page). However, instead of this, the user sees a purple web page containing a trace of some Python script that failed. The URL in the user's web browser when this purple error page is open is https://www.student.cs.uwaterloo.ca/cas-interface/logout

Impact: Affects everybody who uses MarkUs. For people with Admin and Grader roles (i.e. the course staff), not being able to log out isn't a big issue. However, it can be a big problem for students who share computers. If students cannot log out of MarkUs, then two people sharing the same computer cannot switch accounts.

How to reproduce: Click the "Log out" link in MarkUs.

Systems/Software involved:

• MarkUs ( LOGOUT_REDIRECT variable in MARKUSAPPROOT/config/environments/production.rb)
• mod_auth_cas
• Maybe CAS logout on www.student.cs.uwaterloo.ca web server?

How to fix and/or workarounds: Google searching "mod_auth_cas sign out" seems to suggest that mod_auth_cas doesn't handle signing out very well. Some work arounds are:

• Restarting the web browser (close all tabs and windows)
• Clearing all markus.student.cs.uwaterloo.ca cookies manually.
• Using a different web browser (which doesn't have anyone logged into MarkUs yet)
• Make it so that the "Log out" link clears the user's cookies (ex. with Javascript or Rail's cookies.delete function). CS 135 tried this in Fall 2014, and it only kind of works. Sometimes when students log out, they can still get back into MarkUs without having to log in again.

Duplicate groups being created ("Whoops - something went wrong, please contact your course instructor")

Term(s) issue occurred in: Winter 2015. This issue started happening in Winter 2015 term. It has never been observed or reported before Winter 2015. The Javascript quick fix described below is implemented for CS 115, 116, 135, and 240 in Winter 2015. At the end of April 2015, it seems that even without this fix, the problem doesn't happen anymore. So it's possible CSCF changed something that fixed the problem.

Description/Background: The students' MarkUs homepage has a list of assignments they can view. When students click on the link for Assignment X for the first time, MarkUs checks if the student has a group or not for Assignment X (you can see all the groups in the "Groups" tab in MarkUs). If the student does not have a group, MarkUs will create a group for the student automatically. This is why the first time a student goes into an assignment in MarkUs, it takes a bit longer then subsequent accesses to the assignment.

Expected behavior that isn't happening: MarkUs is supposed to create exactly one group, but for some reason, it's creating multiple groups. In the Groups tab, the table at the right should have unique rows. When this problem affects a student, you will see one of two things:

• The student will have two rows. One row will have a green check mark, but the other row will have a red cross.
• The student will have two rows, but both rows will have green check marks and look identical. In the "Submissions" tab, there will be two rows for the student, but these two rows can be different (ex. have different marks).

When an affected student tries to access an assignment (not necessarily the assignment they got bugged on), he may see the error "Whoops - something went wrong, please contact your course instructor". Normally, he'd be taken to the assignment's overview page.

Impact: Affects students who submit using MarkUs. An affected student may not be able to submit assignments in MarkUs (a very big problem!). This problem should not affect courses that use MarkUs only for online grading, because in these courses, the ISA will create the groups by uploading a file, instead of letting MarkUs create groups automatically.

How to reproduce: One way to reproduce this bug is:

1. In MarkUs, go to the Users tab, then Students tab. Add a new student with a fake and impossible user name (ex. Test_Student_1).
2. Switch role to that student you just created.
3. Rapidly click on the link for an assignment. For example, double click the link.
4. Switch back to admin role, and go to the Groups tab for the assignment you click on in previous step. The student may have two rows. If not, switch back to the student role and try double clicking another assignment.

Systems/Software involved: Not entirely sure yet. The transaction block in the function create_group_for_working_alone_student in file app/models/student.rb may be being run multiple times concurrently, leaving the database in an inconsistent state. Since this problem started happening in Winter 2015, it may be due to some software upgrades that took place in January 2015.

How to fix and/or workarounds: To allow affected students to submit assignments, you can use the following workaround.

1. For every assignment that has been created in MarkUs, check if the student already has a group for that assignment (use the Groups tab).
   • If the student already has at least one group, then the student should be able to submit this assignment. Go on to the next assignment.
   • Otherwise, the student does not have a group yet. Create a new group by clicking the "Add new" link in the top of the right pane. For the Group Name, you can use StudentQuestId-group (ex. yc2lee-group). Once the group has been made, assign the student to that group (check the student in the left pane, check the group you just made in the right pane, and click the green + button in the middle).
2. Every time you create a new assignment, manually create a new group for the affected student and assign him to it, like in the previous step. If you have several affected students, you can keep them in a CSV file and just upload that file to save time.
3. Switch role to the affected student, and try clicking on a links to go to an assignment. Make sure that the links work and that the "Whoops" error does not happen.

The above steps will allow affected students to submit assignments again, but it will not prevent the problem from happening again to other students. It's still being investigated what the root cause is. In the meantime, one quick fix is to add some Javascript that prevents the assignment links on the student's MarkUs homepage from being clicked multiple times.

Cannot assign marks for a criteria when marking ("criterionMarkable has already been taken")

Term(s) issue occurred in: Winter 2015. This issue started happening in Winter 2015 term. It has never happened before Winter 2015. Doesn't affect many students and an easy workaround is available, so a fix is still being investigated.

Description/Background: When marking in MarkUs, you can assign a mark for a criteria by clicking on the level (for rubrics), or my entering a number (for flexible marking schemes). Once you click a level or enter a number, MarkUs will register the mark and save it automatically.

Expected behavior that isn't happening: Instead of saving the mark, Markus gives an error. An example error message is (for flexible): "Mark must be between 0 and the maximum mark of the criterionMarkable has already been taken".

Impact: Affects markers, because they cannot assign marks. Affects students too because their assignment cannot be marked. This is a serious issue, but there's a workaround for it.

How to reproduce: Not sure how to reproduce. The problem seems to occur randomly.

Systems/Software involved: Under investigation. Might be related to the duplicate groups bug above.

How to fix and/or workarounds: You can fix the problem for an affected student as follows:

1. Go to the marking page for the student. When you try to assign a mark, you should get an error. Now, copy the URL in your web browser down somewhere (ex. in Notepad). Keep this browser tab open.
2. Open a new browser window/tab, and go to the Submissions tab for the assignment. Find the student. Click the link in that row, second column from the left.
3. In the new page that opens, click "Collect and Grade This Revision" button. You should be redirected to the marking page. If not, go back to the submissions page, click refresh, and access the marking page from there.
4. In the new marking page, you should now be able to assign marks for the student. To save time, you can copy the marks and annotations you had before (in the tab you kept open from Step 1) over to the new marking page.

You can repeat these steps for each affected student. The actual cause of the problem is being investigated.

Multiple submissions being collected

Term(s) issue occurred in: Fall 2014.

Description/Background: When marking in MarkUs, you can navigate between submissions by:

• Clicking the "Previous Submission" and "Next Submission" links from the grading interface
• In the Submissions tab, click the student's Quest ID (leftmost column).

Expected behavior that isn't happening: Suppose you're marking a bunch of submissions. You click "Next Submission" to go to student S's assignment, then you give S a mark of 70%. If you go back to the Submissions page, the big table should show that S has 70%. However, when this bug occurs, S does NOT have a mark of 70%. You may also have trouble setting the marking status to Complete, and/or releasing the mark for S.

Impact: Affects the marking process. Markers will have difficulty marking, because it'll appear like they haven't marked S's assignment yet. Students may have to wait longer to get their assignment back.

How to reproduce: One way this issue can happen is if you click the "Collect All Submissions" link (in the Submissions tab) multiple times rapidly. For example, if you click the link many times without waiting for MarkUs to finish loading the web page.

NOTE: Do NOT click the Collect All Submissions link for a real assignment that's been marked! If you want to try this out, do it for an assignment that won't be marked (ex. create a test assignment), and ideally do it from a MarkUs instance that isn't currently being used.

Systems/Software involved: MarkUs only. The collection process reads and writes to the submissions table in the MarkUs database. If you click the "Collect All Submissions" link multiple times, it can create several processes that all read and write to this same table. It looks like the processes might not be synchronized properly.

In the submissions table in the MarkUs database backend, there may be multiple rows with the same grouping_id (this is normal). However, an invariant is that exactly one of those rows should have submission_version_used field set to true; all other rows should have that field set to false. When this issue occurs, however, more than two rows will have submission_version_used set to true. So MarkUs cannot determine which submission version to use. In this situation, MarkUs picks one at random.

How to fix and/or workarounds: Once this issue occurs, to repair the database, you will have to use the command line (ie use a terminal). You will need to use the terminal just to find the affected students; trying to find the affected students from the web interface will take too much time, even for small courses. Nick Lee (yc2lee) has written scripts to find and fix the grouping problems. These scripts are in markus@assignments.student.cs, under the paths:

• /u7/markus/u/yc2lee/markus_find_bad_groupings.rb
• /u7/markus/u/yc2lee/markus_fix_grouping.rb (can set "run" variable to false, if you don't want the script to actually write to the database).

These scripts can be run by cd'ing into /u7/markus/markus_csXXX_T/markus-0.11.0/, where XXX is your course number, and T is one of lowercase f, w, or s (for Fall, Winter, Spring, respectively). Once in the markus-0.11.0 folder, run:

script/runner --environment=production /u/markus/u/yc2lee/markus_find_bad_groupings.rb

The scripts are a bit messy, but if you don't see anything between the square brackets, then there are no problems for that student for that assignment.

Uploading Groups files take a very long time

Term(s) issue occurred in: Fall 2014 (this issue affected CS246), Winter 2015 (this issue affected CS 136)

Description/Background: Courses that use MarkUs only for online marking (i.e. students don't submit on MarkUs) will upload a groups CSV file in MarkUs. This will create a group for each student.

Expected behavior that isn't happening: Uploading the groups file should be relatively fast (ex. 10 min is reasonable for a large class with 1000 students). However, MarkUs instead takes many hours to upload the CSV file.

Impact: Severely delays marking. Until the groups are uploaded successfully in MarkUs, marking cannot start.

How to reproduce: There is some randomness to this issue. The randomness is that for some assignments, the uploading doesn't take that long. Then suddenly, for the next assignment, it will take many hours to upload the CSV file.

Systems/Software involved: When a Group is created, the build_repository function in the file app/models/group.rb is called. The function build_repository will try and give each admin and each TA read and write permissions on the repository. These permissions are stored in the file /u/markus/markus_csXXX_T/svn-repos-root/svn_authz.

If you actually look at the svn_authz file, you'll notice that most groups don't actually have each admin and each TA listed. This is because of a bug in the __parse_authz_file function, which is defined in lib/repo/subversion_repository.rb. The function __parse_authz_file assumes that admins and TAs have usernames which are matched by the regular expression (\w+). However, UW uses the convention that admins who also want to be graders will use the grader username QuestID-grader. The dash is not matched by \w, and this causes the svn_authz file to be incorrectly parsed and incorrectly written back.

Basically, svn_authz is supposed to contain, for each group, all the admins and TAs. But due to a regular expression matching bug, it does not. Instead, svn_authz only has some of the admins and TAs listed for each group.

Now, uploading a groups CSV file may cause the build_repository function to be called. This function will call set_bulk_permissions, defined in lib/repo/subversion_repository.rb. set_bulk_permissions iterates over each line in svn_authz. For a large class, svn_authz can reach 15000 lines. In the case of CS 246 Fall 2014, their svn_authz file was 47596 lines. It's parsing this long file which causes groups CSV upload to take so long.

It turns out that the regular expression bug actually makes the CSV file upload faster. The bug makes it so that not all admins and TAs are listed for each group in svn_authz, so svn_authz is shorter thanks to the regular expression bug.

How to fix and/or workarounds: To fix the problem:

• Empty svn_authz. That is, delete it, and recreate the file with nothing in it. This can be done with rm svn_authz and touch svn_authz Bash commands.
• In the file app/models/group.rb, in the function build_repository, comment out the following lines:

    Admin.all.each do |admin|
      user_permissions[admin.user_name] = Repository::Permission::READ_WRITE
    end
    # Each grader will have read and write permissions on each repo
    Ta.all.each do |ta|
      user_permissions[ta.user_name] = Repository::Permission::READ_WRITE
    end

Commenting these lines out will prevent svn_authz from having the admins and TAs listed for each group, and it will significantly reduce the size of svn_authz file. The read-write permissions that MarkUs was setting don't seem to be needed for anything that UW uses MarkUs for; even without these permissions set, admins and graders can still mark and checkout and commit repositories.

After these two changes, uploading the CSV file should take about 10 min for large classes (ex. CS 136 in Winter), instead of hours.

Students cannot see submitted files, OR marking web page doesn't load student's code

Term(s) issue occurred in: Winter 2015 and Fall 2014 and several terms before that.

Description/Background: When students use MarkUs, they can see the list of files they have submitted. Students need to see these files in order to replace or delete these files. If students have already submitted a file named F, they cannot upload a new file named F. They must either use the replace function, or delete F first.

When you start marking a submission, MarkUs will take a second or two to load the student's code in the bottom left pane in the grading interface.

Expected behavior that isn't happening: Instead of showing students the files they have submitted, nothing appears.

When marking, instead of loading the file to be marked, MarkUs will say "Loading Code.." with a spinning cicle, but it never finishes loading.

Impact: This is a very serious issue for students, because they will be unable to replace a previously submitted file. If students have already submitted a file called F, they need to use the delete or replace feature to upload a new copy of F. When this issue occurs, they cannot do either action.

For markers, the problem will prevent them from marking an assignment.

How to reproduce: Delete the MOD_AUTH_CAS_S cookie and try to mark an assignment. Instead of loading the file, it will freeze with "Loading Code.." meassage.

Another way to reproduce the problem is to open MarkUs and leave it open for 2 hours without going back to the MarkUs homepage or assignments list (you don't have to be idle in MarkUs). TAs are more likely to experience this problem because they have to spend more than 2 hours marking, and they navigate from submission to submission using the "Previous/Next Submission" links in the grading interface.

Systems/Software involved: The problem is caused by the Apache module mod_auth_cas, which is used for CAS/WatIAM authentication. According to the mod_auth_cas documentation, there is a configuration variable called CASTimeout, described as:

Directive: CASTimeout
Default: 7200 (2 hours)
Description: This is the hard limit, in seconds, for a mod_auth_cas session (whether
it is idle or not). When a session has reached this age and a new
request is made, the user is redirected to the CASLoginURL to
obtain a new service ticket. When this new ticket is validated,
they will be assigned a new mod_auth_cas session.

On the MarkUs server, this variable is not set, so it has the default value of 2 hours. That's why if you open MarkUs for 2 hours, you will get this issue, regardless of whether you're idle or not.

How to fix and/or workarounds: One way to fix the issue is to set a very large timeout value. A workaround is to manually delete all MarkUs web cookies.

For markers, you can actually just go back to the Assignments home page (click the Assignments tab or Home tab at the very top). This will give you a new set of cookies, and the "Loading Code.." freeze issue should go away.

Marking page doesn't load student's code (different from timeout issue)

Term(s) issue occurred in: Winter 2015 for CS 230

Description/Background: This problem will appear the same as above: When grading, the student's code doesn't load and the "Loading Code.." spinning circle will appear to freeze.

Expected behavior that isn't happening: Instead of showing the grader the student's file, MarkUs freezes with a "Loading Code.." message.

Impact: Affects certain students, who will not be able to get their assignment marked properly.

How to reproduce: Submit a text file containing the byte 0xEA.

Systems/Software involved: Rails seems to process the byte 0xEA (and a few others) in strings in a strange way. For example, from the Rails command line interface:

>> "abc\x61\ndef"
=> "abca\ndef"
>> "abc\xEA\ndef"
=> "abc?
def"
>> 

The 0xEA byte seems to make \n be processed, whereas if you use a more "normal" byte like 0x61 (which is just 'a'), the \n appears normally in the string.

In the end, this behaviour makes MarkUs send syntactically incorrect Javascript code to the client's web browser. It's because of the incorrect Javascript code that makes MarkUs freeze.

In Winter 2015 CS230, the affected assignment involved Unicode, so a number of students had the byte 0xEA in their submitted file.

How to fix and/or workarounds: No fix yet. A workaround is to filter non-printable characters out of the students' files before uploading them to MarkUs. This can be done if those bytes are not important for the assignment.

Alternatively, you can click the "Download" button when grading to see the file. However you won't be able to make annotations in it.

You can combine the above two workarounds: Create a copy of the student's submission with no weird bytes. Put annotations in this copy, but assign marks based off of the student's original submission (with the weird bytes).

In Racket .ss files, '< and '> symbols have semicolons after them

Term(s) issue occurred in: This issue has likely always existed, but it was first observed in Winter 2015 in CS 135.

Description/Background: When you mark a file ending in .ss (ex. the GRADED_ASSIGNMENT.ss file), MarkUs will do some Racket syntax highlighting.

Expected behavior that isn't happening: When marking, the displayed file contents should be exactly what is in the file. However, if the file ends in .ss and contains the symbol '> or '<, MarkUs will add a semicolon at the end. Thus markers will see '<; and '>; instead of just '< and '>. The student did not write the semicolon; MarkUs incorrectly placed it in when it was doing the syntax highlighting.

Impact: Markers may accidently give students a lower mark. The issue is strictly with the MarkUs grading interface incorrectly displaying the file. The file on the course account is correct (i.e. if you view the file in a terminal on the course account, there is no semicolon). Thus RST and any scripts you run on the course account will still work correctly.

How to reproduce: Mark a file that ends in .ss and contains '> or '<.

Systems/Software involved: The problem is partly in the file /u7/markus/public_html/javascripts/syntaxhighlighter/shBrushScheme.js, which contains the regular expression '.[a-zA-Z0-9_\-]* used to match symbols. When the MarkUs server sends the file contents to the user's web browser (through Ajax), the MarkUs server will first sanitize the file contents. In particular, it will replace < and > with &lt; and &gt; respectively, because < and > have special meaning in HTML. Thus the symbols '< and '> are turned into '&lt; and '&gt;. The syntax highlighter interprets the former as the symbol '&lt, followed by a semicolon to start a comment. This is because the & matches the dot in the regular expression, and lt matches the character class. Semicolon is not in the character class, so it's not matched by the regular expression. MarkUs will try to display '&lt and the semicolon in different HTML elements so that the two can be in different colours.

Now, in the function dp.sh.Highlighter.prototype.AddBit defined in the file /u7/markus/public_html/javascripts/syntaxhighlighter/shCore.js, it assigns some DOM element's innerHTML to some strings. In particular, it will assign a span element's innerHTML attribute to the string "'&lt". However, most web browsers will incorrectly think you made a typo and forgot to add a semicolon, and the web browser will add a semicolon at the end for you. Thus the innerHTML will be set to the string "'&lt;" instead of "'&lt". Meanwhile MarkUs still has a semicolon to display (the semicolon it thinks is starting a comment). Thus there will be two semicolons, one of them being extra.

How to fix and/or workarounds: One way to fix the problem is to change the regular expression in shBrushScheme.js to be ('&((amp)|(lt)|(gt));)|('.[a-zA-Z0-9_\-]*). This will tell MarkUs that the semicolon is actually part of the symbol, and not starting a comment.

PDFs showing up all black when grading

Term(s) issue occurred in: Fall 2013 for CS 240. Fixed sometime before Fall 2014.

Description/Background: Some courses require students to upload PDFs containing their solutions. When you mark these PDFs, MarkUs will convert them to JPEG images. MarkUs does this in the background using ImageMagick's convert program/command. As of March 30, 2015, the version of ImageMagick installed on the MarkUs server is: ImageMagick 6.6.9-7 2014-03-06 Q16.

Expected behavior that isn't happening: MarkUs should show the contents of the PDF, but instead, it shows a bunch of large, black areas. Markers cannot see the student's solutions.

Impact: This issue is a big problem for marking, because markers cannot see the student's solutions in the PDF.

How to reproduce: The problem seems to affect only certain PDFs. Try uploading PDFs that contain images (ex. CS240's Latex Assignment 0). You may not be able to reproduce this problem anymore, because it's already been fixed a while ago.

Systems/Software involved: The issue stems from the function convert_pdf_to_jpg in the file app/models/submission_file.rb. It's the convert_pdf_to_jpg function that runs the convert command. The following web page describes the problem with convert and the black areas:

http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=20234&start=0

How to fix and/or workarounds: To fix, add the -flatten argument to the convert command in the convert_pdf_to_jpg function. Add -flatten after the -append argument.

Bug/Issue Reporting Template

Term(s) issue occurred in: List the terms where the issue happened. If fixed, when was it fixed?

Description/Background: Provide a general description, like which specific page in MarkUs the problem occurs on.

Expected behavior that isn't happening: Describe what you expected to happen, and what is happening instead.

Impact: Describe the users affected (ex. just students? just ISAs?). Also describe how severe the problem is (ex. is the problem just annoying and kills time, or will it make MarkUs completely unusable?)

How to reproduce: Give specific instructions on how to reproduce the problem, OR state that the problem occurs randomly.

Systems/Software involved: List the software that partly or fully cause the problem. Make specific references to files and code line numbers if possible.

How to fix and/or workarounds: Provide any fixes or workarounds you know of.