Resources and Information for CS 240 ISAs

This twiki is for technical details only. Any other duty information can be found in Waterloo LEARN (D2L): CS Instructional Support Group -> Course Admin & Duties for ISAs.

Important Info [cs240E]:

Timeline

First Day:
  • Enter the course account and familiarize yourself with the locations of files, the directory structure, all the HTML files under public_html
  • Upload assignment 1 to the course webpage after checking the questions and verifying that it's the final version with the prof
Week 1:
  • After becoming familiar with the course account, go onto Markus and familiarize yourself with creating and modifying assignments, marking schemes, adding students, and managing graders. Provide a list of the file names that students are expected to submit, but allow for any any file names to be accepted to account for typos.
  • Go on Piazza and make announcements about assignment 0 and 1 being released.
  • Talk to the instructor by email and upload module 1 onto ~/public_html/TERM/modules and display it by modifying lectures.phtml.
  • At the end of the week, prepare for tutorial 0 by familiarizing yourself with LaTeX.
Week 2:
  • Post your office hours and make sure to attend them
  • Conduct your first tutorial and add tutorial 0 slides/pdf/tex file to the course webpage
  • Learn the procedure to collect assignments and how to use markerallocation.py in ~/markus/bin to assign graders to student submissions.
  • Check MarkUs Scripts on ISG Twiki to see how the course's scripts are used.
  • Email and meet with the TAs to discuss marking assignment 0
  • Ask for assignment 2 from the prof and start doing its questions
Week 3-4:
  • Upload the marking scheme for assignment 1 to MarkUs after receiving it from the prof -After assignment 1's due date has passed, release assignment 2 as quickly as possible.
  • An hour after assignment 1's due date has passed, upload solutions to the course webpage by putting the solution file in ~/protectPDF/cs240a1solns and modifying ~/public_html/TERM/protect/requestAssgnSolnCommon.php -Learn the autotesting procedure for upcoming assignments that have programming questions.
  • MarkUs does not handle autotesting. You must create your own bash script to test students' submissions.
  • Meet with the instructor to discuss the test cases that they want.
  • https://cs.uwaterloo.ca/twiki/view/ISG/MarkUsGroupsSVNRepos#Auto_testing_Procedure
Continually throughout the term:
  • Prepare for tutorials.
  • Attend your office hours to help students
  • Do the assignments the prof gives you before you release them
  • Collect assignments about 10 minutes after their due date
  • Run markerallocation.py to assign TAs to submissions
  • Host meetings with TAs to discuss marking
  • Make autotesting scripts when required
  • Upload marking schemes to MarkUs
  • Upload assignment solutions an hour after the due date
Week 8 is usually the midterm. You'll have to proctor it and then do midterm marking. Similarly for the exam

Assignments

Setup

Public tests

  • files can be found in ~/marking/A#/test.pt The main file to edit is runTests, which you need to ensure is updated before the assignment is made public on Markus. It checks student submissions and verifies that the required files are included. Make sure that the file names and the term name are up to date.
  • For assignments with programming questions the public test also runs a basic test of their code. This is generally a test using sample input/output that was included with the question.
Markus
  • Assignment needs to be createdand set up on Markus.
  • Short identifier: should be exactly A# (one digit) so that it matches with the public test directories on the course account
  • (they also receive emails, but this is a nice alternative way of viewing the test results)

    (when assignment is released, uncheck)

  • Required Files: Generally specified in the assignment, but for A0 it should be "a0.pdf" and "a0.tex", and for A1-5 it should be of the form "a##wp.pdf" (for single file submissions) and "a##q#w.pdf" for each question (second number being question number).
  • Submission Rules:
  • A0 of CS 240 always uses a rubric marking scheme (i.e. NOT flexible). Most assignments after A0 use a flexible marking scheme. If the assignment has not changed from the previous term, upload the rubric used in the previous term. If it has changed, either modify the csv file from the previous term and then upload (this can be easier if there are many changes) OR upload the previous term's rubric and modify in MarkUs.
  • Likewise to above for annotations. Annotations can be created as the assignment is marked, but for this to be possible there at least need to be annotation groups created (suggested: one called "General" and one for each question). Uploading annotations from a previous time a question was used can be a useful starting point, though note that you should probably go through them and remove duplicates, fix spelling/punctuation as necessary etc.
Marking Scheme
  • Get sample marking scheme from instructors. This will depend on the instructor but is often a very bare-bones marking scheme which you will need to flesh out to ensure that it covers alternative solutions and allows for part marks when possible.
  • The modified marking scheme should be typed up (ideally in LaTeX, see archives for examples) and the updated version should then be sent to the instructors to ensure they are okay with your modifications.
BEFORE the first marking meeting:
  1. If not done already, add TAs as Graders in MarkUs (see ISG Twiki for how to)
  2. Set up Markus (see above)
  3. Mark 2 or 3 to see if marking scheme and/or annotations need refining (and if so for the marking scheme for A1 and after, check with instructor) and to help with the next point.
  4. Create a marking notes document (there should be past samples in the course archives) and sample solutions to hand out to each grader (also copies for the ISA(s) and ISC) at the meeting and to email to graders AFTER the marking meeting (we want to make sure they come to the meeting to get the important verbal notes). It should have the marking scheme and any other special things to watch for when marking (e.g., allowed, disallowed, things that might speed up marking, etc.).
  5. Assign students to graders. See the ISG Twiki, https://cs.uwaterloo.ca/twiki/view/ISG/MarkUs#Assign_Submissions_to_Graders, for instructions. If the markerallocation.py script is not available, let the ISC know. The ISA should mark about 10-15 for A0 and 5-10 for other assignments. Remember to make sure if there are any double TA units since they get twice the marking as single units (we have none thsi term). TAs should not have more than 35 students to mark each assignment for a single TA unit and the ISA marks any excess if TAs are at 35 (the ISA should always be marking 5-10 and if that leaves <35 each for TAs, that is fine).
Also include in the marking package:
  • a printout of the Assignment questions
  • for A0: a printout of the LaTeX file (for reference to see if students have made changes)
  • Sample solution note: A1 and on: if the instructor supplies sample solutions, check that they meet all assignment guidelines. If the ISA is providing, have the instructor check them. These sample solutions will also be the ones posted in the protected area (student authentication required
AT the first marking meeting:
  1. ISC: says a few words of welcome and other notes about expectations
  2. ISA: hands out copies (enough for each TA/grader, the ISA and an extra copy for the ISC) of the marking notes the ISA prepared beforehand, goes through the general points of the marking scheme and expectations for the answers, and especially, important notes that may be in the marking notes but that we want to make sure the graders don't miss.
AFTER the first marking meeting:

Email the marking notes and sample solutions to the graders (cc: cs240@uwaterloo.ca and the instructor) with Subject: "[cs240] A# Marking Documents" (fill in "#" appropriately) . Also note that graders should reply all to this email whenever they run into marking questions so everyone will be able to be fair and consistent in marking and that graders should email cs240@uwaterloo.ca when done marking and with the common problems encountered. See course email for past emails that could be used as a template.

For all Future Assignments,
  • post sample solutions (protected, students authentication required -- see protectPDF/README.txt on the course account) an hour after an assignment deadline (not needed for A0).
  • if there are bonus marks, the criteria in MarkUs MUST have "Bonus" in the criteria name. This is to make the marking clearer to students and for the processing later that results in the web page marks posting (so bonus marks are correctly taken into account in what the assignment is out of)
  • At least 7 hours before (preferably the previous day) a marking meeting: send an email to the graders (cc: cs240@uwaterloo.ca and the instructor) with Subject: [cs240] Assig x marking meeting reminder (fill in "x" appropriately) and remind them of the meeting day, time and location, and to bring their laptops (if possible) and to go through the assignment beforehand.
  • During marking: You (the ISA) or the instructor (or if multiple instructors, the instructor responsible for this assignment) should be answering the marking questions (it will be the ISA's job to act first and not wait for the instructor to respond but if you are unsure how to answer, send the instructor a separate email to inquire). The ISA should also be spot checking, for every assignment (including A0), at least 2-3 students where marking is completed for each grader, to ensure the grader is following the marking scheme, clearly indicating where marks are lost, and leaving good comments (if comments are needed because it is not obvious from the rubric message the student would receive). This is best checked during marking so graders can be notified before they are done if they need to do things differently (and/or go back and re-do something they have already marked if the marking problem is important).
  • As markers finish and when all marking is done: You (the ISA) are to put together the common problems in a post-mortem, have the instructor check it over (make any suggested modifications), then post when you release the marks (if the post-mortem is not ready shortly after marking is done, it is okay to release the marks first but the post mortem should be up within a day of that). Let the ISC know when marking is done so the ISC can get the web page marks posting updated
Additionally For Future Assignments:
  • The ISA will lead the marking meetings alone.
  • The ISA also takes attendance at every meeting and after the last meeting, gives a copy of the attendance records to the ISC (keep the original to help with TA evaluations)
  • the ISA keeps track of which TAs get the marking done on time and which don't (including how many days late, as well as reasons, if any given, for the delay) as well as any marking quality issues.

Testing Student code for Assignments with Programming Questions

  • MarkUs does not do any autotesting.
  • An automatically run script places all student submissions from MarkUs to handin folder of course account.
  • For assignments with programming, you must run a bash script to test the student submissions in handin, then record the marks for every student individually within their folders before reuploading all the marks to students' MarkUs. See Markus Prepare For Marking for details.
Testing Submissions

There is a script in the course account called ~/bin/test-student which can be used for testing student submissions. running it without arguments will give a help message describing how to use it.

There are also similar scripts test-parallel, test-upload, test-stats which run test-student in parallel, upload autotest results to markus and calculate mark distribution over the tests respectively.

TODO:
  • Moss scripts (for checking plagiarism)
  • Tutorial question archive on the course account, categorized by topic/module and including tex sources and pdfs with the question, ideas for how to present it in the tutorial and a sample solution.

Key Links

Non-Technical (To be moved to D2L)

Outdated

No Longer Relevant

-- PatrickLee - 18 Aug 2011

Topic attachments
I Attachment Action Size Date Who Comment
Texttxt TimelineREADME.txt manage 2.5 K 2017-08-18 - 15:14 PeilinWang  
Texttxt informationForCS240EISAs.txt manage 0.7 K 2017-08-18 - 14:30 PeilinWang  
Topic revision: r18 - 2018-08-17 - BenjaminWinger
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback