Data Required to Produce CS Reports and Other Output

Deliverables in OFIS_dev for December 2009

Research Book / Webpage in OFIS

• Manually entered: webpage/webpage.php
  • Personal URL
  • photo URL - add database column for URL; user-entered data on webpage.php
    • data in http://www.cs.uwaterloo.ca/cscf/web/snaps
  • Biography
  • Research Group:
    • affiliations.php: Department Research Keyword
  • Major Awards:
    • awards_recognitions/awards_recognitions.php: Award Name / Description / Other information / Year of Award: [ for OFIS: for next version can we add an "include on webpage" option checkbox on the awards page? Or add an "Show Awards Section" checkbox on webpage.php? ]
  • Industrial and Sabbatical Experience: add new text-field on webpage.php
  • Representative Publications: contributions/contributions.php
    • publication type, status, peer-reviewed?, entry-as-appear, submission-year, acceptance-year, publication-year, display-on-webpage, [...]
    • [ for OFIS: add "bibtex-id" field to de-duplicate bibtex-supplied records - added to _dev by tduberst ]
  • Students Supervised / Present Positions (addition at request of Peter Forsyth):
    • supervisory_experience/details.php: add "include on webpage" option checkbox [ for OFIS: perhaps for next version ]
    • Name; Type of Training, Status; Years supervised; Title/Thesis; Present Position
  • Teaching Interests: [ for OFIS: add a new CS-specific tab on left-hand side so this can go on CV and webpage? or, add new text field to expertise/expertise.php? ]

Output not In OFIS

The Research Book web output includes a front-page including: introduction, research areas, faculty members, and an index. It links to institutes, lab directory, technical reports, and PhD theses.

OFIS's natural scope is around information associated with faculty members, including their research area(s). OFIS doesn't have the concept of groups/research-area properties (URL, contact person, description). It does have the concept of faculty group membership to produce the list of group members; and does have faculty specialties which can be used to produce the index.

Properties of Research Groups might be recorded in a CS-specific database that we set up separately. However, there is a compelling argument to keep the existing process and text database for group-specific properties (described in ST#70072). We will proceed with that basis for now.

The Research Book front page could be merged using a template of ofis-supplied information (list of faculty linking to ofis-supplied faculty pages; list of research areas linking to external research-group pages; list of keywords for the index) and text-database information (introduction, links to external data for institutes, lab directory, technical reports, PhD theses).

Front Page Section	Source	Details
Introduction	text db	as current
Affiliated Institutes and Centres
Other Information
Research Areas	OFIS	select d.dept_keyword from dept_keywords d where d.dept_id = 2950 - maps to expertise.dept_keyword_id (expertise/expertise.php)
Individual Research		[something like] select names, uwdir where dept_id = 2950
Index		expertise_keywords from expertise/expertise.php

Research Book LaTeX document

Output for the Research Book includes a LaTeX document with complete data for profs and research groups.

Currently, it is driven by the perl code in: services108.cs:/.software/regional/wwwdata_cs.uwaterloo.ca/data/vhosts/cs/research/directory-2008/source/bin/rd-latex

Given our group data in text format, and our user data in OFIS, we will use a process of generating user-specific intermediate text files to mirror the old text format, and generate the book using the generated source files. If this turns out to be unworkable, or not meet all our needs, we can reassess how to generate the book for some time after the start of 2010.

OCGS Report Section on CVs

While this won't be needed for a number of years, Peter Forsyth says it would be an attractive point for professors that it's entirely automated. Low priority for December 2009 as time allows.

• Automatically entered from Research Office and Grad Studies Office.
  • Missing Research Office automatic data older than Fall 2008 for Math/CS, requiring manual input for below.
  • employment/history.php
    • Position Held, Employer Name, Department, YYMM/MM Start, YYYY/MM End
  • supervisory_experience/details.php
    • as above with CS Annual Report Automatic/Manual
  • Graduate Courses past 7 years: teaching/teaching.php
    • Term, Subject, Course# [ D lookup: CS lists course title and cross-listings- necessary? - will ask Lynn Judge or Cathy Jardine in grad office for most recent OCGS template ]
  • External Research past 7 years: research_support/research_support.php
    • Year, Source, Type, $/yr, Purpose [ D lookup: is this "program?" ]

• Manual Input [ for OFIS: yes? ]
  • academic_background/academic_background.php
    • Degree, Institution, Country, Date graduated, Field, Display-on-Web
  • awards_recognitions/awards_recognitions.php
    • Award Name, Description, Other information, Year
  • service/professional.php
    • Service Title, Description, Period, Time Commitment, time interval
  • supervisory_experience/details.php as above in "automatic"
  • supervisory_experience/lifetime.php
    • Undergrad number, Masters number, PhD number
    • NOT merged with manual/automatic list of individuals; need to hand-edit lifetime totals.
  • contributions/life_time.php
    • Books auth, books edited, chapters in books, papers refereed journals, papers refereed conference, technical reports, abstracts, patents, others
    • NOT merged with list of contributions; need to hand-edit lifetime totals.
  • contributions/contributions.php
    • publication type, status, peer-reviewed?, entry-as-appear, submission-year, acceptance-year, publication-year, display-on-webpage, bibtex-id, [...]

Student Evaluations

• Data displayed in OFIS interface: Term, Subject, Course Number, Course Type (LEC, L01, L02), N Enrolled, % Responses, AVG Q1-9, Q10
• CS has a fairly complex process to get to a somewhat similar form of output.

• Data as should be supplied for OFIS according to OfisNotes08Apr2009:

uwdir

Lastname

Firstname

Middlename

Course (CS/CE/..)

Course#

Section

Term in numeric form

Q1

Q2

...

Qn

Calculated: Preparation

Delivery

Effectiveness

Overall

CS Processing

Modified from OfisNotes15May2009:

Lori Suess [MFCF Cient Support] ([and/or] Jim Johnston [MFCF Software Specialist/manager]) receive two data-sources each term: excel from Math Undergrad Office; and a floppy disk of data from IST. She sent me a sample of each. These are merged with a canned output-template to produce the evaluation pages, in .ps, text, and ([t]roff) format. That output is sent to Frank Tompa (and the student math office; for the book). It is not archived by Math.

The floppy from ist contains a text file with individual coded student responses

517000001001041907001   5363 #0001    N 51001                
517000002001041907001   5363 #0001    N      232316211223332 
517000003001041907001   5363 #0001    N      132216221222321 
517000004001041907001   5363 #0001    N      131 26211132331 
517000005001041907001   5363 #0001    N      231116311223322 
517000006001041907001   5363 #0001    N      232116211222332 
517000007001041907001   5363 #0001    N      122116221223313 
517000008001041907001   5363 #0001    N      232216211232211 

[...]

517000083001041907001   5363 #0001    N      231233 22222323 
517000084001041907001   5363 #0001    N      231236221222331 
517000085001041907001   5363 #0001    N 51002                
517000086001041907001   5363 #0001    N      523446253222232 

[...]

First row denotes a class; following rows are individual student responses, up until the non-indented delimiter "51002".

• delimiter-line: [card number from student] [garbage data; always the same, ending "N" ] [course number]
  • course number 51001: 5 means it's cs.
• data lines: course-number is replaced by student answers to each question in order.

Excel data from Math Undergraduate Office

Index #	Subject	Catalog	Sect	Last Name	First Name	Middle	Tot Enrl	Dept.
12311	ACTSC	231	001	Chisholm	Diana	Katherine	157	MC 6136
12321	ACTSC	232	001	Chisholm	Diana	Katherine	92	MC 6136
13311	ACTSC	331	001	Cai	Jun		105	MC 6136
13711	ACTSC	371	001	Kim	Hyun Tae		126	MC 6136
13721	ACTSC	372	001	Hardy	Mary	R	155	MC 6136
14331	ACTSC433/ACTSC833		001	Li	Siu Hang		61	MC 6136

[...]

• index# corresponds to the 51001 in the other file.

How does this turn into the free-form text?

• Lori runs a checksum program, she believes written by Bill Ince [Associate Director, CSCF]; it has been updated by Jim J any time Math Undergrad Office changes their data-stream; but they aren't very fond of updating it. Ray Butterworth has written a script in .csh to try and do the same thing; but it isn't in production.
• David Mcinnon, Math chair for undergrad studies, also was working on excel data-merging tool, but it was last year and he hasn't gotten back to them about getting the real data. The perl script she emailed me appears to be similar to capo:/fsys2/source/course-evaluation/ceqsum/ceqsum

bash-3.00$ ls -la /fsys2/source/course-evaluation/ceqsum/ceqsum ceqsum-test.pl
-r-xr-xr-x   1 root     none       23004 Apr 13  2005 /fsys2/source/course-evaluation/ceqsum/ceqsum
-rw-r--r--   1 drallen  other      24923 May 14 13:43 ceqsum-test.pl
bash-3.00$ diff -w /fsys2/source/course-evaluation/ceqsum/ceqsum ceqsum-test.pl |wc  
     126     525    3819

The diff appears to include input-data differences as well as output-formatting. As listed at the header of both files, capo:/fsys2/source/course-evaluation/ceqsum/ceqsum.perl is (lightly) commented.

Lori suggested that she would be happy to email us a copy of the .txt and excel data each term. They would be pleased to work with me/us to find some solution that feels less clunky than the current system.

Proposed Process for CS Data

1. Math Dean's Office sends floppy and .xls data to CSCF
2. CSCF runs a simple merge script turning it into a single data-source; filling in uwdir and reporting errors for any names we can't determine uwdir
3. CSCF sends output to Marlon for import; or submits it to OFIS directly.
   • OFIS data includes calculated fields.
4. (potentially) CSCF sends merged data to Frank Tompa; or produces a report to match his output as below.
5. (potentially) CSCF sends merged data to MFCF in a standard unchanging format, to make their processing easier as well.

CS Output Data

As currently produced by scripts written and maintained by Frank Tompa, this is the output currently generated. [ for Peter Forsyth, Frank Tompa; is there other output data we should produce? ]

   From:    fwtompa@uwaterloo.ca
   Subject:    Re: Course Evaluation data for OFIS
   Date:    April 24, 2009 5:21:59 PM GMT-04:00
   To:    drallen@cs.uwaterloo.ca

Following is the CS and SE data from Fall 2008. First, here is the key to decoding:

Format of output lines:

Surname Forenames <yy.y>course(<F>v)*
where
 yy = year
 t = 1 if Winter, 2 if Spring, 3 if Fall
 and for each field F, the value v represents:
   N = the number of responses
   R = the response percentage
   L = the avg for Q9 (using lower bounds)
   W = the avg for Q15 (using avgs)
   P = 80% Q1 + 20% Q10
   D = the avg for Q2-Q6
   E = 20% Q7 + 80% Q8

Notes: <P> Q1,10 together reflect effort in preparation for teaching
       (heavily weighted in favour of Q1)
   <D> Q2-6 together reflect presentational and interaction
        techniques
   <E> Q7,8 reflect impact made on students (perceived
       effectiveness).
       (heavily weighted in favour of Q8)
   values of -1 indicate no data available

e.g.:

Doe Jo Lee <08.3>CS 101 03<N>28<R>31.82<L>80<W>10<P>3.78<D>3.87<E>3.07

shows that for Jo Lee Doe's offering of CS 101 (section 3) in Fall 2008, 28 persons responded, 
accounting for 32% of the students; they attended 80% of the lectures on average (N.B. the highest 
possible value here is 90% since I rate "x-y%" as if it were x%), worked 10 hours per week outside 
of class, and graded the professor at 3.78 for preparation, 3.87 for delivery, 3.07 for 
effectiveness (these 3 on a scale of 5=best to 1=worst).

Good luck. Frank

From follow-up discussion in December 2009:

From:    fwtompa@uwaterloo.ca
Subject:    student evals
Date:    December 1, 2009 4:58:09 PM GMT-05:00
To:    drallen@cs.uwaterloo.ca

1. Here is a sample of the input format I now receive from MFCF:

<I>Xxxxx Xxxxx  <C>ACTSC221 001
<Q>1<Q'> 12 10 1 2 0 0
<Q>2<Q'> 1 4 20 0 0 0
<Q>3<Q'> 13 5 7 0 0 0
<Q>4<Q'> 9 8 6 1 1 0
<Q>5<Q'> 8 12 5 0 0 0
<Q>6<Q'> 7 5 4 0 0 9
<Q>7<Q'> 10 11 3 0
<Q>8<Q'> 10 11 4 0 0 0
<Q>9<Q'> 21 3 1 0 0
<Q>10<Q'> 7 18 0 0 0
<Q>11<Q'> 3 6 0 13 2
<Q>12<Q'> 14 8 2 1 0
<Q>13<Q'> 1 3 21 0 0 0
<Q>14<Q'> 1 2 18 4 0 0
<Q>15<Q'> 10 9 2 2 1
<N>25<R>39.69
<I>Xxxxx Xxxxx    <C>ACTSC231 001
<Q>1<Q'> 34 29 6 0 0 0
<Q>2<Q'> 3 10 52 4 0 0
<Q>3<Q'> 23 23 14 2 1 6
<Q>4<Q'> 37 21 8 3 0 0
<Q>5<Q'> 18 35 13 2 1 0
<Q>6<Q'> 8 10 10 2 0 39
<Q>7<Q'> 20 39 5 2
<Q>8<Q'> 30 30 7 1 1 0
<Q>9<Q'> 53 12 3 1 0
<Q>10<Q'> 16 40 11 0 1
<Q>11<Q'> 5 7 3 48 6
<Q>12<Q'> 13 34 12 1 9
<Q>13<Q'> 7 6 55 0 0 0
<Q>14<Q'> 6 12 48 0 2 1
<Q>15<Q'> 13 44 10 0 1
<N>69<R>45.10

etc. for each course section offered in the term.

2. Here is the sort of report I prepare for a tenure and promotion case:

name [course] <N R L W> P/aP D/aD E/aD [#]
Xxxxx Xxxxx 
[STAT206 (2004.3)] <30 30.00 80 2> 2.49/3.40 3.17/3.95 2.80/3.62 [5]
[STAT361 (2004.3)] <31 46.27 79 2> 3.75/3.75 4.00/4.00 3.65/3.65 [1]
[STAT331/STAT361/SYDE334 (2005.1)] <60 49 82 4> 3.25/3.57 3.76/3.92 3.33/3.61 [8]
[STAT441/STAT841/CM463 (2005.3)] <18 53 88 3> 3.22/3.72 3.56/3.87 3.14/3.75 [5]
[STAT331/STAT361/SYDE334 (2006.1)] <31 38 79 3> 3.15/3.57 3.59/3.92 3.25/3.61 [8]
[STAT331/STAT361/SYDE334 (2006.1)] <36 56 83 4> 2.83/3.57 3.20/3.92 2.89/3.61 [8]
[STAT220 (2006.3)] <5 14 87 2> 4.20/4.28 4.40/4.34 4.20/4.12 [5]
[STAT441/STAT841/CM463 (2006.3)] <10 67 84 3> 3.80/3.72 3.75/3.87 3.56/3.75 [5]
[STAT444/STAT844/CM464 (2007.1)] <13 76 86 3> 4.05/3.92 4.31/4.16 3.92/3.98 [3]
[STAT230 (2007.1)] <25 27 87 3> 3.03/3.90 3.54/4.01 2.80/3.81 [57]
[STAT221 (2008.1)] <15 42 83 3> 3.72/4.06 3.70/3.99 3.57/3.77 [5]
[STAT444/STAT844/CM464 (2008.1)] <13 68 88 4> 3.52/3.92 3.85/4.16 3.73/3.98 [3]
[STAT341/CM361 (2008.3)] <14 88 76 3> 2.76/3.89 3.63/4.12 3.03/4.08 [7]
[STAT440/STAT840/CM461 (2009.1)] <19 61.29 86 3> 3.19/3.76 3.44/3.85 3.13/3.76 [5]
[STAT444/STAT844/CM464 (2009.1)] <21 53.85 82 4> 4.18/3.92 4.33/4.16 4.28/3.98 [3]
(n=15 @51%)    (attend 83% +  3 hr/wk) prep: 3.41  del: 3.75  eff: 3.42

3. Here is a sample of historical numbering for courses:

AMATH 473 01
AMATH/PH 473/673/454 001
AMATH/PHYS 473/673 001
AMATH / PHYS 473 / 673/454
AMATH/PHYS 473/673/454 1
AMATH/PHYS 473/673/454 1
AMATH/AMATH/Phys 473/673/454 1
AMATH473/AMATH673/PHYS454 1

CM/CS 473/673 1
CS/CS/CM 473/673/473 1
CS473/CM473/CS673 001

CO/CS/PHYS 481/681/467 0
CO/ CS / PHYS 481 / 681 / 467 / 667 / 467/ 767
CO/CS/PHYS 481/681/467/667/467/767 1
CO/CS/PHYS 481/467 1
CO/CS/PHY 481/467/467 1
CO481/CS467/PHYS467 001

Notice the variance in spacing, section numbering, ordering of courses, dept abbreviations, what's between the slashes, etc.

Frank

CEAB Accreditation Report for Software Engineering

Software Engineering needs to submit regular CEAB reports. Charles Clarke, Director of Software Engineering, has confirmed that the output from OFIS as it currently exists, meets their reporting requirements, as long as they have the necessary data input.

He is personally interested in further entering data into OFIS this December.

Anticipated Future Required Data for CS

Deliverables 2010.

CS Annual Report (listed as "Merit Report" in OFIS)

Lower priority than student evaluations, as directed by Peter Forsyth (10 November, 2009). The information is highly variable year-to-year, so much of it is one-off. Compared to Engineering Merit Report, much less amenable to being efficient at keeping accurate.

• Data automatically entered from Quest
  • teaching/teaching.php
    • Term, Subject, Course Number, Enrolled, R/S/F/D
      • PF says: RSFD: assume 'R', for all.
• Automatically entered from Graduate Studies Office
  • supervisory_experience/details.php
    • Name, University, Status (FT/PT), type of HQP training (undergrad/Masters/PhD/etc), undergrad type, other type, status of degree, type of training, co-supervised with, YYYY/MM Start, YYYY/MM End, Project Title, Present Position
    • Term supervision start/end: need to convert from month to F/S/W
    • MMath Thesis/Essay [ for OFIS: for next version, shall we add "if type is Masters of Math, is this an essay or thesis?" ]
• Manual Input
  • Courses Developed in this year, Courses coordinated in this year
    • could be handled as a new category in contributions.php / Service: Internal -or- since these seem to be one-offs for this report and for each year, probably we can fill in "None" and let them edit the doc.
  • employment/sabbatical.php
    • automatic in Engr, from Dean's Office; manual in Math for now. Anticipated automatic feed from HR, at some future time.
    • Start YYYY/MM, Length, Organization, Location
  • Thesis Committee Membership: service/internal.php and service/professional.php [ for OFIS: also "External professional-PhD Examiner / Reviewer" category? ]
    • Name [ Service Description ], Service Title, Service Period
  • Research Area: expertise/expertise.php
    • Research Keywords
    • [ for OFIS: the rest of expertise.php is CEAB-specific? should the page say "skip the below unless filling in CAEB? ]
  • Publications: contributions/contributions.php data:
    • ordered by: 1) refereed papers, journal articles, conference proceedings, book contributions with publication date 2008 2) accepted in 2008 for publication 3) other articles submitted 4) software distribution [ for OFIS: I think we need a new contribution type: Software. otherwise has similar/same format as other contrib. ]
  • Research Seminars: contributions/contributions.php "conference presentations"
  • Refereeing/Reviewing Activities: service/professional.php as above in OCGS
  • Research Funding: research_support/research_support.php
    • PI and co-applicants, Title, amount awarded, years held.
  • Centre/Institute Service: [ probably ] service/professional.php
    • [ for OFIS: new title category: "External professional - Centre/Institute" ], Description, Start YYYY/MM, End YYYY/MM
  • Consulting Activities: service/professional.php:
    • [ for OFIS: new title category: "External professional - Consulting" ? ], Description, Start YYYY/MM, End YYYY/MM, time spent [ for PF: hours? form asks for days? ]
  • Technology Transfer: ip_disclosure/ip_disclosure.php [ assuming all data ]
  • Administrative and committee service: service/internal.php [ assuming most or all department/faculty/university service-types ]:
    • Title, Description, Start YYYY/MM, End YYYY/MM, Time-committment.
    • Using Start/End months to break into Winter/Spring/Fall
    • [ for OFIS: possibly add a new category to committment hours: "hours per term" ]
  • External Service: service/professional.php and service/community.php :
    • Title, Description, Start YYYY/MM, End YYYY/MM, Time-committment.
  • Additional Information: comments/comments.php :
    • [ for OFIS: CS Requires simpler page; only the "additional comments" field at the bottom of the comments.php, for the "additional comments" area at the bottom of the Annual report. ]

CV (dump of all OFIS data)

Deemed low-priority for 2009 in conversation with Peter Forsyth.

NSERC Form 100 part 2

• List of research contributions.
  • From OfisNotes10Jul2009 would be easy, except requires:
    • identifying the grad student(s) involved and bolding them in the output
    • identifying the source of funding
  • Peter Forsyth recommends against it; his opinion is every prof has ideosyncratic ways to fill in their information.

Statistics on data for admins

• such as within Engr Dean's Office. Currently not a priority; unknown needs if it becomes a priority.

Data Not Required for CS

Accreditation Report

• for CS other than Software Engineering

-- DanielAllen - 03 Nov 2009