Resources and Information for CS 240 ISAs

This twiki is for technical details only. Any other duty information can be found in Waterloo LEARN (D2L): CS Instructional Support Group -> Course Admin & Duties for ISAs.

Important Info [cs240E]:

• Information For CS240E ISAs

Timeline

First Day:

• Enter the course account and familiarize yourself with the locations of files, the directory structure, all the HTML files under public_html.
• Upload assignment 0 and assignment 1 to the course webpage after checking the questions and verifying that it's the final version with the prof.

Week 1:

• After becoming familiar with the course account, go onto MarkUs and familiarize yourself with creating and modifying assignments, marking schemes, adding students, and managing graders. Provide a list of the file names that students are expected to submit, but allow for any any file names to be accepted to account for typos.
• Go on Piazza and make announcements about assignment 0 and 1 being released.
• Make the weekly post on Piazza about upcoming deadlines and an overview of this week + next week. Look in archives for references on how to format it and what to include.
• Add modules 0 and 1 to the course website (see Uploading Modules to the Course Website below).
• Familiarize yourself with the course email account and reply to any relevant emails so far (see Email Guidelines below).
• Add new students to Markus, Piazza and Marmoset (every day for first few weeks).
  • For Markus, go to Users->Students and click "Upload from course account".
  • For Piazza, run (odyssey classlist | cut -f2 -d, | sed 's/.*/&@uwaterloo.ca/ > piazzaClasslist.csv) in the course account to generate a csv in the right format. Then go to Manage Class->Enroll Students in Piazza and upload it.
  • For Marmoset, follow the instructions in the Instructor View of a course, People->Register students or groups for this course using a CSV file.
• Update the course website with all relevant details to the new term. Let ISC/instructors know if something is missing.
  • Check all pages and links (e.g. correct term and year, dates)
  • Update list of course staff (+ pictures) and assignment deadlines
  • Update lecture and tutorial times (can see the Schedule of Classes for information)
  • Previous terms can be used as a reference.
• At the end of the week, prepare for tutorial 0 by familiarizing yourself with LaTeX.

Week 2:

• Post your office hours (as well as IAs' and instructors') on the Google Calendar and make sure to attend them.
  • See ~/.googlelogin in the course account for login details.
• Conduct your first tutorial and add tutorial 0 slides/pdf/tex file to the course webpage.
• Learn the procedure to collect assignments and how to use markerallocation.py (helpful link here) in ~/markus/bin to assign graders to student submissions. (You can also do this in MarkUs by clicking "Users -> Graders")
• Check MarkUs Scripts on ISG Twiki to see how the course's scripts are used.
• Email and meet with the TAs to discuss marking assignment 0.
• Ask for assignment 2 from the prof and start doing its questions.
  • Check for typos and any complicated language that you think could be made clearer.
  • Make sure the assignment is a good length. Students should be spending ~5 hours a week on assignments, ISAs less.
  • Make sure the assignment is a good difficulty. Hints can sometimes be good for a difficult question.
  • Check for alternative solutions to questions.

Week 3-4:

• Upload the marking scheme and annotations for assignment 1 to MarkUs after receiving it from the prof -After assignment 1's due date has passed, release assignment 2 as quickly as possible.
• An hour after assignment 1's due date has passed, upload solutions to the course webpage by putting the solution file in ~/protectPDF/cs240a1solns and modifying ~/public_html/TERM/protect/requestAssgnSolnCommon.php -Learn the autotesting procedure for upcoming assignments that have programming questions.
• MarkUs does not handle autotesting. You must create your own bash script to test students' submissions.
• Meet with the instructor to discuss the test cases that they want.
• https://cs.uwaterloo.ca/twiki/view/ISG/MarkUsGroupsSVNRepos#Auto_testing_Procedure

Continually throughout the term:

• Prepare for tutorials.
• Attend lectures and tutorials
• Attend your office hours to help students
• Do the assignments the prof gives you before you release them (proofread)
• Collect assignments about 10 minutes after their due date
• Run markerallocation.py to assign TAs to submissions (You can manually do this now since MarkUs introduces a new feature)
• Host marking meetings with TAs to discuss marking
• Make autotesting scripts when required
• Upload marking schemes and annotations to MarkUs
• Upload assignment solutions an hour after the due date

Week 8 is usually the midterm. You'll have to proctor it and then do midterm marking. Similarly for the final exam.

If you want to access the course folder when not using the office computer, here is a guide you can follow: https://cs.uwaterloo.ca/twiki/view/ISG/SambaWindows.

Uploading Modules to the Course Website

1. Pull the latest changes from the git repository in ~/modules (i.e. git pull).
2. Run ./buildHandout.pl <xx> where xx is the module number to build. (e.g. ./buildHandout.pl 00 to build module 0)
   This will generate two files, modulexx.pdf and handoutxx.pdf.
3. Copy both files to ~/public_html/w22/modules.
4. Make sure the files have read permissions for others in order to be viewable on the website. If not, run chmod o+r to add the permission.
5. Modify lectures.phtml to link to the modules.

Assignments

Setup

Markus

• Assignments needs to be created and set up on Markus.
• Short identifier: should be exactly A# (one digit) so that it matches with the public test directories on the course account
• Allow web submits
  Display submission and revision history to students
  Let students run and view basic/public test results (requires RST/Bittersuite) (they also receive emails, but this is a nice alternative way of viewing the test results)
  Display grader names to students
  Hide assignment from students (when assignment is released, uncheck)
• Required Files: Generally specified in the assignment, but for A0 it should be "a0.pdf" and "a0.tex", and for A1-5 it should be of the form "a##wp.pdf" (for single file submissions) and "a##q#w.pdf" for each question (second number being question number).
• Submission Rules:
  Late submissions receive no marks
• A0 of CS 240 always uses a rubric marking scheme (i.e. NOT flexible). Most assignments after A0 use a flexible marking scheme. If the assignment has not changed from the previous term, upload the rubric used in the previous term. If it has changed, either modify the csv file from the previous term and then upload (this can be easier if there are many changes) OR upload the previous term's rubric and modify in MarkUs.
• Likewise to above for annotations. Annotations can be created as the assignment is marked, but for this to be possible there at least need to be annotation groups created (suggested: one called "General" and one for each question). Uploading annotations from a previous time a question was used can be a useful starting point, though note that you should probably go through them and remove duplicates, fix spelling/punctuation as necessary etc.

Marking Scheme

• Get the sample marking scheme from instructors. This will depend on the instructor but is often a very bare-bones marking scheme which you will need to flesh out to ensure that it covers alternative solutions and allows for part marks when possible.
• The modified marking scheme should be typed up (ideally in LaTeX, see archives for examples) and the updated version should then be sent to the instructors to ensure they are okay with your modifications.

BEFORE the first marking meeting:

1. If not done already, add TAs as Graders in MarkUs (see ISG Twiki for how to)
2. Set up Markus (see above)
3. Mark 2 or 3 to see if marking scheme and/or annotations need refining (and if so for the marking scheme for A1 and after, check with instructor) and to help with the next point.
4. Create a marking notes document (there should be past samples in the course archives) and sample solutions to hand out to each grader (also copies for the ISA(s) and ISC) at the meeting and to email to graders AFTER the marking meeting (we want to make sure they come to the meeting to get the important verbal notes). It should have the marking scheme and any other special things to watch for when marking (e.g., allowed, disallowed, things that might speed up marking, etc.).
5. Assign students to graders. See the ISG Twiki, https://cs.uwaterloo.ca/twiki/view/ISG/MarkUs#Assign_Submissions_to_Graders, for instructions. If the markerallocation.py script is not available, let the ISC know. The ISA should mark about 10-15 for A0 and 5-10 for other assignments. Remember to make sure if there are any double TA units since they get twice the marking as single units (we have none thsi term). TAs should not have more than 35 students to mark each assignment for a single TA unit and the ISA marks any excess if TAs are at 35 (the ISA should always be marking 5-10 and if that leaves <35 each for TAs, that is fine).

Also include in the marking package:

• a printout of the Assignment questions
• Make sure no marking notes are inside the sample solution
• for A0: a printout of the LaTeX file (for reference to see if students have made changes)
• Sample solution note: A1 and on: if the instructor supplies sample solutions, check that they meet all assignment guidelines. If the ISA is providing, have the instructor check them. These sample solutions will also be the ones posted in the protected area (student authentication required

AT the first marking meeting:

1. ISC: says a few words of welcome and other notes about expectations
2. ISA: hands out copies (enough for each TA/grader, the ISA and an extra copy for the ISC) of the marking notes the ISA prepared beforehand, goes through the general points of the marking scheme and expectations for the answers, and especially, important notes that may be in the marking notes but that we want to make sure the graders don't miss.

AFTER the first marking meeting:

Email the marking notes and sample solutions to the graders (cc: cs240@uwaterloo.ca and the instructor) with Subject: "[cs240] A# Marking Documents" (fill in "#" appropriately) . Also note that graders should reply all to this email whenever they run into marking questions so everyone will be able to be fair and consistent in marking and that graders should email cs240@uwaterloo.ca when done marking and with the common problems encountered. See course email for past emails that could be used as a template.

For all Future Assignments,

• post sample solutions (protected, students authentication required -- see protectPDF/README.txt on the course account) an hour after an assignment deadline (not needed for A0).
• if there are bonus marks, the criteria in MarkUs MUST have "Bonus" in the criteria name. This is to make the marking clearer to students and for the processing later that results in the web page marks posting (so bonus marks are correctly taken into account in what the assignment is out of)
• At least 7 hours before (preferably the previous day) a marking meeting: send an email to the graders (cc: cs240@uwaterloo.ca and the instructor) with Subject: [cs240] Assig x marking meeting reminder (fill in "x" appropriately) and remind them of the meeting day, time and location, and to bring their laptops (if possible) and to go through the assignment beforehand.
• During marking: You (the ISA) or the instructor (or if multiple instructors, the instructor responsible for this assignment) should be answering the marking questions (it will be the ISA's job to act first and not wait for the instructor to respond but if you are unsure how to answer, send the instructor a separate email to inquire). The ISA should also be spot checking, for every assignment (including A0), at least 2-3 students where marking is completed for each grader, to ensure the grader is following the marking scheme, clearly indicating where marks are lost, and leaving good comments (if comments are needed because it is not obvious from the rubric message the student would receive). This is best checked during marking so graders can be notified before they are done if they need to do things differently (and/or go back and re-do something they have already marked if the marking problem is important).
• As markers finish and when all marking is done: You (the ISA) are to put together the common problems in a post-mortem, have the instructor check it over (make any suggested modifications), then post when you release the marks (if the post-mortem is not ready shortly after marking is done, it is okay to release the marks first but the post mortem should be up within a day of that). Let the ISC know when marking is done so the ISC can get the web page marks posting updated

Additionally For Future Assignments:

• The ISA will lead the marking meetings alone.
• The ISA also takes attendance at every meeting and after the last meeting, gives a copy of the attendance records to the ISC (keep the original to help with TA evaluations)
• The ISA keeps track of which TAs get the marking done on time and which don't (including how many days late, as well as reasons, if any given, for the delay) as well as any marking quality issues.

MOSS:

• We use Moss to detect plagiarism for all programming assignments
• The instructions for running Moss can be found at MossBasics
• Note that the ISA needs to run Moss right after the deadline for each programming assignment.

Testing Student code for Assignments with Programming Questions

• MarkUs does not do any autotesting. There are a few examples from previous terms for public test of file names and possibly the programming questions.
• An automatically run script places all student submissions from MarkUs to handin folder of course account.
• For assignments with programming, you must run a bash script to test the student submissions in handin, then record the marks for every student individually within their folders before reuploading all the marks to students' MarkUs. See Markus Prepare For Marking for details.

<!-- <h6 class="TML">Testing Submissions</h6> <p>There is a script in the course account called ~/bin/test-student which can be used for testing student submissions. running it without arguments will give a help message describing how to use it.</p> <p>There are also similar scripts test-parallel, test-upload, test-stats which run test-student in parallel, upload autotest results to markus and calculate mark distribution over the tests respectively.</p> -->

Academic Integrity Declaration Check:

• In handin you can do ./check-ai.bash Midterm_AID (or whatever directory has the AIDs in it) and it will log which students are being checked to stderr, then print a report of potential AID issues to stdout at the end. If you don't clear out the old submissions though, there will be some entries in the report for past students. Also, some of the entries in the report will probably be minor issues you can ignore, so hand-checking is still needed.

TODO:

• Tutorial question archive on the course account, categorized by topic/module and including tex sources and pdfs with the question, ideas for how to present it in the tutorial and a sample solution.

Key Links

• Examination Duties - Everything you need to prepare for exams.
• Office Tips
• MarkUs
• CourseWebpage
• PiazzaTips -- Q/A forum
• Autotesting Procedure
• Distributed RST

Non-Technical (To be moved to D2L)

Outdated

RST Public tests

• files can be found in ~/marking/A#/test.pt The main file to edit is runTests, which you need to ensure is updated before the assignment is made public on Markus. It checks student submissions and verifies that the required files are included. Make sure that the file names and the term name are up to date.
• For assignments with programming questions the public test also runs a basic test of their code. This is generally a test using sample input/output that was included with the question.

RST Programming assignments:

• Normal procedure is the same as normal assignments: The ISA needs to create a marking scheme and hold the marking meeting if there is hand-marking component.
• The MarkUs setting depends on if this is an individual assignment or an embedded question in a normal assignment
• Normally, the ISA needs to come up with all test cases and set up each test case in private tests. You can follow the previous terms' format to make this or use a script called make-in. To use the make-in script, You need to first copy-paste the provided_files folder from some previous terms' folders, and change the number of folder and the .in , .out files accordingly. Note here you'll need to update the computeMarks-postprocess and the iotest accordingly (test name and test program name)
• To assign more than 1 mark to one test cases, rename the folder that you need to change under the ./in/7b/testx_1, change the 1 here to the value you want
• After all submissions are collected and all test cases set up, you can run RST on this assignment. Make sure you check the option: "Let students run and view basic/public test results (requires RST/Bittersuite)". The detailed procedure can be seen at RST page, or use this command:

rst -t t -s 'a*' aXX suitename testrunname

Note here "a*" is the assignment name. suitename is the number/character you use to identify the test suite, normally it is just "1". testrunname can be anything.

• After you finish running the test, follow this website to upload marks: https://cs.uwaterloo.ca/twiki/view/ISG/MarkUsScripts#set_marks_rst%20sub-command. Use this command to add the OUTPUT.txt file to the students MarkUs directory: (Here I use the AUTOTESTRESULTS as the testrunname)

  /u/isg/bin/markus.py upload_marking --prefix=testresults --filter=OUTPUT.txt AX /u/cs240/marking/test.1.AUTOTESTRESULTS

• If your marking scheme has a different weighting for the programming question than the number of test cases you have, use the conversion script:

~/bin/rst2csv ~/marking/PQX_autotest/test.1.AUTOTESTRESULTS ~/PQX_marks_combined.csv WW

• Where X is the programming question number such as 3 and WW is the weight of that programming question such as 12.
• After generating the CSV file use the following command to upload the marks to MarkUs:

~isg/bin/markus.py set_marks_csv PQX ~/PQX_marks_combined.csv

• Where X is the programming question number such as 3
• When everything finishes and post-mortem is ready, make a public note on Piazza with brief information for each test case indicating what they are.
• To get some stats on which tests everybody failed, use the stats script:

~/bin/stats PQX_autotest

• Where 'PQX_autotest' is the _autotest folder for the PQ you want stats on, such as 'PQ1_autotest'

No Longer Relevant

-- PatrickLee - 18 Aug 2011

Academic Integrity Declaration