Course Evaluation database project

Documentation

• CourseEvaluationSysadminDocs
• CourseEvaluationUsers

CSCF Master ST Item

ST#70069

Project Charter

Introduction

The Faculty of Math is responsible for collecting and reporting student evaluation statistics for all Math instructors. Historically, a faculty member in CS has been responsible for generating statistics for all instructors in the School of Math (the "Tompa Scores"), and this responsibility has carried forward to CSCF on that faculty member's retirement.

Motivation

This project will directly support the School and Faculty mandate to collect and report on student course evaluations.

Project Goals

1. reduce brittleness by replacing an existing workflow which depends on a faculty member maintaining and running a system of make/awk/sed scripts
2. increase usability by outputting data in flexible formats
3. prepare for faculty self-service data lookups via a web front-end

Project Objectives

The project goals translate into the following objectives:

1. convert the evaluation workflow into documented python code
2. store historic and current evaluation data in a SQL DBMS back-end
3. investigate user requirements for output, minimally including text and csv.
4. generate plans for a second phase of the project including a web front-end.

Project Sponsor

Ed Lank, Associate Director, School of Computer Science

Project Leader

Daniel Allen, CSCF

Project advisory group

Ed Lank (Associate Director of CS), Dan Brown (Director, Undergrad Studies), Charles Clarke (Associate Director, Undergrad Studies) , Steve Furino (Faculty of Math, Associate Dean Undergrad Studies), Frank Tompa (Emeritus Faculty and creator of the Tompa Scores)

Risks, assumptions and mitigating factors

1. Risk: Failure to complete project by January 2014 when evaluations are reported. Mitigation: continue to run Frank Tompa's scripts until the project is complete.
2. Risk: Improper understanding of Tompa Score logic, leading to differences in calcuated data. Mitigation: recalculate statistics for each historical term, and compare them against Tompa statistics.
3. Assumption: We assume we can correctly understand all of our users' needs- but the description by Frank Tompa was amorphous and other users have suggested additions since the original needs assessment. Mitigation: provide service equal or greater to the service provided by Frank Tompa, and iteratively collect requirements and enhance the service as necessary to meet needs.

Project Architecture Overview

Flowchart

The following overview is illustrated by a flowchart of the architecture and processes performed by IST, MFCF, CSCF, and the Dean's Office.

Inputs

A few weeks before the end of term, students fill in evaluations for each of their Math faculty courses. After the end of each term, data for each Math faculty course corresponding to individual student responses is collated by the Math Undergraduate Office (MUO). The MUO forwards this data to MFCF, which runs a program with two functions:

1. merge the data with an output template to produce evaluations per instructor/class-section to be printed for the MathSoc
2. translate the data to an XML-like format to be forwarded to CSCF (formerly to Frank Tompa)

Additionally, data for courses evaluated via Desire2Learn (Fall 2013, consisting of all First-Year courses) will be supplied by IST in a similar text format as 2) above.

Lastly, normalized class/instructor/section data are provided from Quest via our data-feed into postgres.odyssey.uwaterloo.ca

Data Processing and Storage

Both course input sources (MUO/MFCF and IST) are validated against Quest records of instructors/sections for a given term in order to normalize instructor IDs and to add class IDs where missing. This validated data is stored in a postgres table containing one record for each class section per term, with summary reports of responses to particular questions. The data is then processed to produce summary statistics for each class section per term, and then to produce per-instructor and per-course summary statistics. All statistics will be kept indefinitely in the course evaluation database.

Outputs

CSCF produces:

1. evaluations per instructor/class-section (for use by CS Director for Undergraduate Studies, Math Associate Dean for Undergraduate Studies, CS Director, and Math Department Chairs; and eventually available to each instructor directly)
2. aggregation per instructor over previous 5 years and over all years available (for use by CS Tenure and Promotions Committee, Math Department Chairs, and eventually available to each instructor directly)

Functional Milestones

• 2013-11-15: code that can import a new term's worth of data, as specified
• 2013-11-30: generate a complete set of statistics to compare with Frank Tompa's data
• 2014-01-20: command-line tool to replace Frank Tompa's "findall" command
• 2014-01-30: Fall 2013 data is imported into the system; user documentation is complete
• 2014-02-28: sysadmin documentation is complete, requirements-gathering for second phase of project is complete.

Meetings

2013-06-26 - Ed Lank and Daniel Allen: summary: agreement that this project is a school priority.

2013-09-09 - Daniel Allen and Frank Tompa: summary: Daniel will update his validation code from using the old postgres database to the new postgres.odyssey Quest database; Frank will review Daniel's questions from January (relating to Frank's draft of an excel spreadsheet to produce calculations; agreement that we probably want to convert the excel spreadsheet into python code for server-side operation.

2013-10-28 - Daniel Allen, Sean Warren, Lori Suess: summary: Sean is prepared to produce a CSV containing anonymized individual-student results for each of the 49 sections corresponding to 10 first-year courses. The format and content are different from IST's old data-feed. The only substantial deficiency is he cannot reproduce the old IST feed's format, which means Lori can't run their existing process to generate the printed evaluations for MathSoc. A minor deficiency is he cannot insert the class_id which the MUO had recently added for us (as a verification/normalization tool). Lori will follow up with Cyntha (cc'ing Daniel) on the question of process for printed evaluations. The evaluations for 1st-year courses in Fall 2013 will run for ~7 days closing 2013-12-04, and the data will be available 1 week later on Wednesday 11 December.

2013-11-08 - Daniel Allen, Lori Suess: summary: Lori has emailed with Cyntha; Cyntha has discussed with the dean that: "A person with Excel programming skills is needed to create output which is as similar as possible to the data that was previously provided to faculty members, department chairs, etc. and which can be integrated with the course evaluations still conducted on paper. [...] Summaries should be created that look like those in the pink course evaluation book which is produced each term. The data that is used to create the Tompa's scores should go to the appropriate person as well." - Lori suggests a meeting between her, Daniel, Cyntha, and Wayne to figure out process. For clarity, Daniel will email her, Wayne, and Bill a short description/flow-chart of his understanding of process and options.

• course-evaluations-process-option-A.pdf: course-eval process option A: Dean's Office produces mathsoc printed output
• course-evaluations-process-option-B.pdf: course-eval process option B: CSCF produces data so MFCF can produce mathsoc printed output

2013-11-11 - Ian Goulden has emailed to say the Dean's Office will produce mathsoc printed output, and asked Jack Rehder to look into possible uses for our data for the Dean's office. He cautions the Fall 2013 process using LEARN isn't guaranteed after this term- "that will be decided after we see how this term goes. Thus, I am especially leery about introducing new procedures and a lot of work for various people that might only be used one time."

2013-11-21 - Daniel Allen, Frank Tompa: reviewed discrepencies between Frank's generated data and Daniel's, resulting in two bugs discovered: one in Daniel's, and one in Frank's, slightly affecting the number of hours per week reported for students (a very minor error but one that had been there since the beginning of his reporting). A simple diagnostic was produced for evaluating differences that Daniel can use without Frank's assistance. Lastly, discussed the fact that Postgres rounds half-to-even instead of standard unix "sprintf" rounding half down- Frank asserts that these rounding differences are unimportant, but Daniel will still replace code to round his results the same as Frank's.

2013-12-2 - Daniel Allen, Frank Tompa: reviewed process and output. Many pieces of additional information were recorded, some which affect the timeliness of finishing. (see ST #70069)

2013-12-12 - Daniel Allen, Frank Tompa, Dan Brown: While Frank thought the data going back to 2005 would be sufficient, Dan Brown points out that Tenure and Promotion questions can go back to the earliest of Frank's data, 98. To be addressed in a subsequent phase. (See ST#70069).

2014-01-24 - Mail from Daniel Allen to Ed Lank, Dan Brown, Charles Clarke, Steve Furino, Frank Tompa reporting the new app would be ready on time by the end of January; a link to this document; and that for continuity, Frank's old process would run on the Fall 2013 data as well.

2014-01-31- VERSION 1.0 RELEASE. Mail from Daniel Allen to Ed Lank, Dan Brown, Charles Clarke, Steve Furino with instructions on using the application. Clarification questions have led to new ST items for dealing with tenuring and promotion (ST#91970 and descendents). -- DanielAllen - 2013-10-18