Course Evaluation

Data Processing and Storage

Course Evaluation input sources (MUO/MFCF and IST) are validated against Quest records of instructors/sections for a given term in order to normalize instructor IDs and to add class IDs where missing.

Normalization and Import process

Data is stored in postgres table raw_course containing:

  • one record for each class section per term with summary reports of responses to particular questions:

course_id term_id instructor user_id course_label section nresp rresp q1.1 q1.2 q1.3 q1.4 q1.5 q1.6 q2.1 q2.2 q2.3 q2.4 q2.5 q2.6 q3.1 q3.2 q3.3 q3.4 q3.5 q3.6 q4.1 q4.2 q4.3 q4.4 q4.5 q4.6 q5.1 q5.2 q5.3 q5.4 q5.5 q5.6 q6.1 q6.2 q6.3 q6.4 q6.5 q6.6 q7.1 q7.2 q7.3 q7.4 q8.1 q8.2 q8.3 q8.4 q8.5 q8.6 q9.1 q9.2 q9.3 q9.4 q9.5 q10.1 q10.2 q10.3 q10.4 q10.5 q11.1 q11.2 q11.3 q11.4 q11.5 q12.1 q12.2 q12.3 q12.4 q12.5 q13.1 q13.2 q13.3 q13.4 q13.5 q13.6 q14.1 q14.2 q14.3 q14.4 q14.5 q14.6 q15.1 q15.2 q15.3 q15.4 q15.5 comment tag
34093 2005.1 instr1 instr1 ACTSCnnn 001 40 53 5 13 9 10 2 0 1 17 18 2 0 2 11 19 7 2 1 0 2 16 11 9 2 0 5 14 13 5 1 1 11 14 5 3 1 6 6 26 6 2 6 14 13 6 1 0 26 10 2 0 2 14 23 2 0 1 4 19 16 0 1 0 4 9 25 2 0 8 30 1 1 0 0 1 38 0 0 0 8 25 6 1 0
34094 2005.1 instr2 instr2 ACTSCnnn 001 68 78 21 29 12 2 2 1 1 8 47 8 3 0 14 27 17 4 2 2 17 26 18 3 2 1 16 34 11 3 2 0 4 11 16 8 0 28 20 33 11 1 21 23 16 5 2 0 53 12 2 0 0 25 35 6 0 1 6 8 2 42 5 9 34 18 2 4 2 8 49 7 1 0 1 3 52 8 2 1 20 39 5 2 0 online
34420 2005.1 instr2 instr2 ACTSCnnn 002 38 40 12 18 8 0 0 0 2 3 30 3 0 0 12 17 7 1 1 0 9 20 7 1 0 0 13 17 6 1 0 0 4 8 5 2 1 18 14 18 4 1 12 17 8 1 0 0 33 4 0 0 1 13 23 2 0 0 3 5 3 23 3 2 20 12 0 4 0 2 34 1 0 1 0 0 31 6 0 1 9 23 4 1 0 instr. name fixed from 'other name' grad

Data Processing

  • The data is then processed via python script to produce summary statistics for each class section per term

  • Key for calculations in the last column (celln corresponds to sub-answer of question)

hrs_per_week (cell1+4.5*cell2+8.5*cell3+13*cell4+16*cell5)/(cell1+cell2+cell3+cell4+cell5)
weight24531 (2*cell1+4*cell2+5*cell3+3*cell4+cell5)/(cell1+cell2+cell3+cell4+cell5)
weight532 (5*cell1+3*cell2+2*cell3)/(cell1+cell2+cell3)
weight54321 (5*cell1+4*cell2+3*cell3+2*cell4+cell5)/(cell1+cell2+cell3+cell4+cell5)
percent_classes (90*cell1+75*cell2+50*cell3+25*cell4)/100/(cell1+cell2+cell3+cell4)

Summary Statistics Per Term

Data is stored in postgres table ratings containing:

  • one record for each class section per term with calculations based on the data in raw_course:

col# column name sample1 sample2 description calculations (using col# and following key)
a course_id 34093 34094 common id for a course across terms from raw_course
b term_id 2005.1 2005.1 "standard" Waterloo format '1yym' where yy = year and m = month from raw_course
c instructor instr1 instr2 instructor in WatIAM format "Last, First M" from raw_course
d user_id instr1 instr2 instructor userid (limit 8 characters) from raw_course
e course_label ACTSCnnn/CSnnn ACTSCnnn course label(s) in alpha order seperated by / from raw_course
ee section 001 section number   from raw_course
f nresp 40 68 count of students responding to survey for this section from raw_course
g rresp 53% 78% fraction of student responding to any question from raw_course
h q1 3.23 3.98 weighted average weight54321
i q2 4.37 4.42   weight24531
j q3 3.93 3.73   weight54321
k q4 3.18 3.80   weight54321
l q5 3.45 3.89   weight54321
m q6 3.91 3.28   weight54321
n q7 3.16 3.45   weight532
o q8 3.45 3.84   weight54321
p q9 83% 86%   percent_classes
q q10 3.67 3.67   weight532
r q11 2.79 3.63   weight532
s q12 2.31 3.00   weight532
t q13 4.65 4.52   weight24531
u q14 4.97 4.55   weight24531
v q15 4.61 4.00   hrs_per_week
w prep 3.32 3.92 Average of Q1, Q10 (80-20 weighted in favour of Q1) reflect effort in preparation for teaching 0.8 * h + 0.2 * q
x delivery 3.77 3.83 Average of Q2-6 (equally weighted) reflect presentational and interaction techniques i + j + k + l + m / (count(i,j,k,l,m)
y effectiveness 3.39 3.76 Average of Q7, Q8 (20-80 weighted in favour of Q8) reflect impact made on students (perceived effectiveness) 0.8* o + 0.2 * n
z summary 3.47 3.82 25% prep + 25% delivery + 50% effec 0.25 * w +0.25 * x +0.5 * y
zz tag none grad may contain text to denote particular data source  

  • summary is also known as the "Tompa Score" for a particular course.
  • An instructor's overall summary for a term can be calculated by averaging the summary for all the courses they have taught.

Summary Statistics Aggregate

Data is stored in postgres table report_5year containing:

  • one record for each class section per term with calculations based on the data in ratings:

Five Year Report (15 terms), averages per-instructor and per-course:

Column description sample calc (using col# from above table)
course_id 9813 from source
term_id 2010.9 from source
instructor_name Doe, John from source
user_id jdoe from source
course_label MATH 123 2 from source
number_resp 113 from source
resp_rate 73% from source
percent_classes 85% p
hrs_per_week 5.5 v
prep 3.49 w
delivery 3.84 x
effectiveness 3.63 y
summary 3.65 z
course_avg_prep 4.12 for all matching courses, avg(prep)
course_avg_delivery 4.12 for all matching courses, avg(delivery)
course_avg_effectiveness 4.08 for all matching courses, avg(effect)
course_avg_summary 4.10 for all matching courses, avg(summary)
course_count 6 count of matching courses
instr_avg_prep 3.68 for all matching instr, avg(prep)
instr_avg_delivery 3.84 for all matching instr, avg(delivery)
instr_avg_effectiveness 3.67 for all matching instr, avg(effect)
instr_avg_summary 3.72 for all matching instr, avg(summary)
instr_count 4 count of matching instr
tag online tag marker for 'grad' and 'online' data sources

The above sample shows that in the Fall 2010 term, when John Doe taught section 2 of
MATH 123, 113 persons responded, accounting for 73% of the students;

they attended 85% of the lectures on average (N.B. the highest
possible value here is 90% since I [Frank] rate "x-y%" as if it were x%),
worked 5.5 hours per week outside of class;

they graded the professor at 3.49 for preparation, 3.84 for delivery,
3.63 for effectiveness (these 3 on a scale of 5=best to 1=worst), for
3.65 average score.

In 6 offerings of this course during the last 15 terms (up to last
term), the averages were 4.12, 4,12, 4.08, 4.10 for preparation,
delivery, effectiveness, and weighted average, respectively.

In 4 offerings by this instructor during the last 15 terms, the
averages were 3.68, 3.84, 3.67, 3.72 for preparation, delivery,
effectiveness, and weighted average, respectively.

Report output format (for pasting in instructions to faculty)

Column sample Description
course_id 9813 unique identifier matching the same course over all terms regardless of the cross-listings, as determined by the maintainers of Quest data
term_id 2010.9 Quest format. The first digit is 0 for years < 2000 and 1 for years > 2000. The next two digits are the last two digits of the year. The last digit is the month.
instructor_name Doe, John from source
user_id jdoe from source
course_label MATH 123 from source
section 001 from Quest, may be inaccurate
number_resp 113 number of responses
resp_rate 73% fraction of student responding to any question
percent_classes 85% average of Question 9: percent classes attended (highest possible: 90% due to lower bounds)
hrs_per_week 5.5 average for Question 15: hours per week outside class
prep 3.49 Preparation: 80% Question 1 + 20% Question 10
delivery 3.84 Delivery: average for Question 2 - Question 6
effectiveness 3.63 Effectiveness: 20% Question 7 + 80% Question 8
summary 3.65 25% prep + 25% delivery + 50% effectiveness
course_avg_prep 4.12 for all matching courses over the past 15 terms, avg(prep)
course_avg_delivery 4.12 for all matching courses, avg(delivery)
course_avg_effectiveness 4.08 for all matching courses, avg(effect)
course_avg_summary 4.10 for all matching courses, avg(summary)
course_count 6 count of matching courses over the past 15 terms
instr_avg_prep 3.68 for this instr over the past 15 terms, avg(prep)
instr_avg_delivery 3.84 for this instr, avg(delivery)
instr_avg_effectiveness 3.67 for this instr, avg(effect)
instr_avg_summary 3.72 for this instr, avg(summary)
instr_count 4 count of courses taught by this instr over the past 15 terms
tag online tag marker for 'grad' and 'online' data sources

-- DanielAllen - 2013-11-02

Edit | Attach | Watch | Print version | History: r16 | r14 < r13 < r12 < r11 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r12 - 2015-02-06 - DanielAllen
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback