Claimer: The published outline at https://outline.uwaterloo.ca/viewer/view/n5h6by gives all the details about the course that are not expected to change during the entire term, such as the personnel; an overall description of the course; general, term-independent instructions; course, School, Faculty, and University policies; and the due date of the final report. The professor's and students' lecture schedules as they happened, the lecture slides, URLs to newly discovered relevant materials, and the archive of all e-mail sent by the professor to the students are found here.

To Go Directly to the Lecture Schedule

To Go Directly to the Student Lecture Schedule

To Go Directly to the Topics List

"E", "em", and "er" are gender non-specific third-person singular pronouns in subjective, objective, and possessive forms, respectively.

Courses and Sections:

CS846 is Advanced Topics in Software Engineering, a graduate seminar. Section 2 is Empirical Studies of People Doing Software Engineering or Advanced Topics in People Empirical Software Engineering.

We have 24 2-hour meeting slots during which we need to meet 36 hours, where an hour is defined as 55 minutes (because we meet from 12:00-13:50, not 14:00). We did not meet on 4 September. So, we have 23 2-hour meeting slots in which to distribute 18 meetings, totaling 36 hours. Thus, we can miss 5 more meeting slots. I know that I will not be able to meet on Tuesday 23 September and on Thursday 2 October. So we have 21 slots, allowing us to miss 3 more slots. Probably, those 3 slots will be to give a 1.5-week break between my last talk, and your first talk.

Instructor:

The reason I have no telephone is that I am nearly deaf. I do not sign, but I do read lips. So, I cannot use a voice-only telephone. I can use a video communication medium if the bandwidth of the connection is high enough that the image gets updated at the frequency of television or movies and thus, the lip movement is smooth enough to be decipherable.

I prefer to meet in person. There's never a bandwidth problem that prevents lipreading. For the safety of both of us, I am vaccinated to the hilt.
If necessary, we can meet via Zoom. Please send to me e-mail that you want to meet with some possible times, and I will reply with one of those times or an alternative proposal. When we agree on a time, I will send you a Zoom invitation.

Evaluation of Instructor at the End of the Term

Highest Level Course Outline:

Course Times and Locations:

Tuesdays and Thursdays: 12:00pm--1:50pm in DC 2585

Readings:

No textbook. Notes are provided at this Web site or are provided at vault.cs.uwaterloo.ca at a URL sent by e-mail to all that are registered in the course.

Workload:

1. Do one of
   • do some project, e.g., a pilot experiment with volunteering subjects (It is not clear how this would work while we are online!),
   • thoroughly explore the research on some topic involving the way people do SE,
   • conduct some pilot empirical study, e.g., with an anonymous survey or with interviews, of the way people do SE, e.g., at a company for which you have worked or in a more general setting, or
   • explore empirical studies of the way people do work in your own area particularly if this work leads to software's being developed,
2. Write a conference-sized paper, about 10 pages long, about the project or research.
3. Give a 30-minute talk based on the paper so that the whole class benefits from it, and the prof has one less lecture to do ☺.

Please get your project or report topic approved by the prof before going too far into it, to make sure that it will be acceptable. Please see the short description of the course for more details.

The due date for the project or report is Wednesday 3 December by 5:00pm

Archive of E-mail Sent to Class:

Click Here for an archive of the e-mail sent to the entire class

Possible Topics for Students:

Here is a list of possible topics for students. This list will be growing as I think of things. In addition to these, you may choose any topic from the topics list about which the prof is not talking or about which he does not talk about everything. Of course, if you have a relevant topic that excites you, propose it in e-mail to the prof.

Student Lecture Schedule

Lecture Schedule

The list below represents the sequence of lectures as we see they will be given. As time goes on, we may change the order. Also as time goes on, we will see how they divide themselves up into dates.

If a topic has hot links, then the slides for the topic are available for downloading. If there are no hot links on a topic, the slides are not ready yet, and will be later, we hope, at least one day before the lecture.

The title itself is a hot link to a copy of its slides in Acrobat form (.pdf). These slides may not be exactly what we are showing on the screen during the lecture. The lecture may have material that we do not have the legal right to distribute multiple copies of. It may have also an exercise that we want to do alive in class with your help. We do not want you to be able to see such material until we have finished.

Underneath the header Additional Materials or in additional rows, you will find some additional reading or viewing material.

            

Date	Day & Lecturer OR Topic & Slides	Additional Materials
4 September	Thursday NO LECTURE: Professor Out of Town
9 September	Tuesday Lecture: Daniel Berry
	Tradeoffs in Experiments of Software Methods
11 September	Thursday Lecture: Daniel Berry	Thanks to Bella Cheng and Helen Chen for recording the whiteboard
	Whole Methods	Replication
16 September	Tuesday Lecture: Daniel Berry
Empirical Evaluation of Tools for Hairy Tasks	Failed Mission
	Empirical Evaluation of Tools for Hairy Requirements Engineering Tasks Tool evaluation problem as motivation to read EMSE paper of the same title	Requirements for Requirements Engineering Tools that Require Understanding Requirement Semantics Requirements for tools that lead to using recall, precision, and summarization to evaluate tools
18 September	Thursday Lecture: Daniel Berry
	Requirements for Requirements Engineering Tools that Require Understanding Requirement Semantics Requirements for tools that lead to using recall, precision, and summarization to evaluate tools	Critique of evaluation in "Do Information Retrieval Algorithms for Automated Traceability Perform Effectively on Issue Tracking System Data?" by Thorsten Merten et al.
23 September	Tuesday NO LECTURE: Professor On Holiday!!!
25 September	Thursday Lecture: Daniel Berry
	Requirements for Requirements Engineering Tools that Require Understanding Requirement Semantics Requirements for tools that lead to using recall, precision, and summarization to evaluate tools	Critique of evaluation in "On the automatic classification of app reviews" by Maalej et al.
	Analysis of Output vs Inpuut Sizes as a function of Lambda and Precision	Plots of Lambda over P for Each Lambda
	Berry, D.M. (2025). Empirical Evaluation of Tools for Hairy Natural Language Requirements Engineering Tasks. In: Ferrari, A., Ginde, G. (eds) Handbook on Natural Language Processing for Requirements Engineering. Springer Book Chapter that focuses on the procedure to evaluate a tool. It is an update of one section in the full paper, described to the right.	Berry, D.M. Empirical evaluation of tools for hairy requirements engineering tasks. Empir Software Eng 26, 111 (2021) Full paper that argues at lenght for and then describes the procedure to evaluate tools for hairy RE tasks, the description of which is updated in the book chapter described to the left.
30 September	Tuesday Lecture: Daniel Berry
	Berry, D.M. (2025). Empirical Evaluation of Tools for Hairy Natural Language Requirements Engineering Tasks. In: Ferrari, A., Ginde, G. (eds) Handbook on Natural Language Processing for Requirements Engineering. Springer Book Chapter that focuses on the procedure to evaluate a tool. It is an update of one section in the full paper, described to the right.	Berry, D.M. Empirical evaluation of tools for hairy requirements engineering tasks. Empir Software Eng 26, 111 (2021) Full paper that argues at lenght for and then describes the procedure to evaluate tools for hairy RE tasks, the description of which is updated in the book chapter described to the left.
	Berry, D.M., Kamsties, E., Ribeiro, C., Tjong, S.F. (2025). Detecting Defects in Natural Language Requirements Specifications. In: Ferrari, A., Ginde, G. (eds) Handbook on Natural Language Processing for Requirements Engineering. Springer Book chapter that describes defects that tools have been built to search for and how well tools help. It describes also some questions about software defect detection that no one has answered.
What is a Specification for an AI?	Berry, D.M. (2022). Requirements Engineering for Artificial Intelligence: What Is a Requirements Specification for an Artificial Intelligence?. In: Gervasi, V., Vogelsang, A. (eds) Requirements Engineering: Foundation for Software Quality. REFSQ 2022. Lecture Notes in Computer Science, vol 13216. Springer	Origins of RE for AI
	RE for AI: What is an RS for an AI? (Short Version)	RE for AI: What is an RS for an AI? (Long Version)
2 October	Thursday NO LECTURE: Professor On Holiday!!!
7 October	Tuesday Lecture: Daniel Berry
	Origins of RE for AI	RE for AI: What is an RS for an AI? (Short Version)
	RE for AI: What is an RS for an AI? (Long Version)
Why do people think that ChatGPT is intelligent?	Berry, D.M., "Why Large Language Models Appear to Be Intelligent and Creative: Because They Generate Bullsh*t!", IEEE Computer 58;6, pp. 91--96, 2025	Why Large Language Models Appear to Be Intelligent and Creative: Because They Generate Bullshit!
9 October	Thursday Lecture: Daniel Berry
On a Particular (Failed) Experiment to Show the Effectiveness of Formal Methods in Software Development, Experiment and Analysis	"Formal Methods Application: An Empirical Tale of Software Development", Ann E. Kelley Sobel and Michael R. Clarkson, IEEE Transactions on Software Engineering, SE-28:3, pp. 308-320, 2002	The relationship between the "Comments..." below and to the left and the "Response..." below and to the right is that S & C were shown B & T's draft comments. Then B & T were shown S & C's draft response. Each pair of authors iterated to make sure that its draft covered all the points about the other pair's draft that each pair felt should be covered. So B & T's answers to S & C's points are in fact already in B & T's article, and vice versa.
	"Comments on 'Formal Methods Application: an Empirical Tale of Software Development'", Berry, D.M. and Tichy, W.F., IEEE Transactions on Software Engineering, SE-29:6, pp. 567--571, 2003	'Response to "Comments on 'Formal methods application: an empirical tale of software development'"', A.E.K. Sobel ; M.R. Clarkson, IEEE Transactions on Software Engineering, SE-29:6, pp. 572--575, 2003
	Supplemental Slides: Berry & Tichy on Sobel & Clarkson
	Supplemental Slides: Importance of Ignorance in RE subslides	Supplemental Slides: Success Stories of Formal Methods subslides
21 October	Tuesday Lecture: Daniel Berry
Importance of Domain Ignorance for Creative Requirements Idea Generation, Experiments, Case Studies	"The Impact of Domain Knowledge on the Effectiveness of Requirements Engineering Activities", Niknafs, A. and Berry, D.M., Empirical Software Engineering, 22:1, pp. 80--133, 2017	Main Slides: Overview of Importance of Ignorance and Empirical Studies About It
23 October	Thursday Lecture: Daniel Berry
	Main Slides: Overview of Importance of Ignorance and Empirical Studies About It	Supplemental Slides: Experiment 1 Plus Experiment 2
	Supplemental Slides: Corroborating, Industrial Case Study See Slides 14--41.
28 October	Tuesday Lecture: Daniel Berry
Benefitting from Newbies' Domain Ignorance, Questionnaire, Datamining, Case Study	"How to Benefit from Newbies' Domain Ignorance in Software Development Projects" Gaurav Mehrotra and Daniel M. Berry, Science of Computer Programming, 204, p. 102593, 2021	Main Slides: Benefitting from Newbies' Domain Ignorance, Full Slides
30 October	Thursday Lecture: Daniel Berry
4 November	Tuesday Lecture: Daniel Berry
6 November	Thursday Lecture: Daniel Berry
11 November	Tuesday NO LECTURE: used up available time
13 November	Thursday NO LECTURE: used up available time
18 November	Tuesday NO LECTURE: used up available time
20 November	Thursday Lecture: up to 3 of 11 Students
25 November	Tuesday Lecture: up to 3 of 11 Students
27 November	Thursday Lecture: up to 3 of 11 Students
3 December	Tuesday Lecture: up to 3 of 11 Students

Topics List

The URLs, particularly of published journal articles, conference articles, and books, should be exercised from within the University of Waterloo Library, where you can sign on with your watIam credentials. Then, you should not have to pay for a download.

Also, being on campus or in a VPN that makes you appear to be on campus works, as well.

           

Journals and Conferences	Journals and Conferences focusing on empirical software engineering as at least one of its subject of interest.	Note that you will find many papers describing empirical studies also in other software engineering journals and conferences, e.g., IEEE Transactions on Software Engineering, ACM Transactions on Software Engineering and Methodology, ICSE (International Confereence of Software Engineering), and more specialized venues.
	Empirical Software Engineering	Empirical Software Engineering grouped by Topics
	Science of Computer Programming	International Symposium on Empirical Software Engineering, renamed at one point to Empirical Software Engineering and Measurement
Software Engineering Experimentation	Read one of the following:
	Read Chapters 1, 2, 3, and 6:	Experimentation in Software Engineering, Wohlin, Claes, et al, 2012
	Read Chapters 1, 2, 3, and 4:	Basics of Software Engineering Experimentation, Juristo, Natalia and Moreno, Ana M., 2001
	Supplemental Reading: Read Chapter 11 ("Selecting Empirical Methods for Software Engineering Research") as an overview and whatever of the other chapters that interest you. In particular, there are chapters about many kinds of empirical studies, not just experimentation.	Guide to Advanced Empirical Software Engineering Shull, Forrest, et al, 2008
	Supplemental Reading: A discussion of issues in Empirical Software Engineering	Empirical Software Engineering Issues. Critical Assessment and Future Directions; Victor R. Basili, Kurt Schneider, Dietmar Pfahl, Dieter Rombach, Barbara Kitchenham, Richard W. Selby (Eds.); LNCS 4336, Springer
	Supplemental Reading: Semi-satirical paper about conducting randomized controlled trials with half of subjects receiving placebos	"Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials"
	Main Slides: Tradeoffs in Experiments of Software Methods
Empirical Software Engineering?	The first paper by Walter Tichy admonishing the Software Engineering Field for its failure to experimentally validate its work, a follow on about evaluating papers about empirical work, and a status report:
	"Should computer scientists experiment more?"; Walter F Tichy; IEEE Computer; Volume 31, Issue 5, Pages 32-40, 1998	"Hints for reviewing empirical work in software engineering"; Walter F Tichy; Empirical Software Engineering; Volume 5, Issue 4, Pages 309-212, 2000
	"Status of empirical research in software engineering"; Andreas Höfer, Walter F Tichy; Pages 10-19, 2007

Resources List

This table is a list of resources that you might find useful for your topics. Some of these may become lectures in the schedule, in which case, they will be moved to the Lecture Schedule table.

                     

Grounded Analysis for Developing Grounded Theory	Read the following for links to other work:
	"The Use of Grounded Theory in Empirical Software Engineering"; Jeffrey Carver; Page 42, 2007	"Building Theories in Software Engineering"; Dag I. K. Sjoberg, Tore Dybo, Bente C. D. Anda, Jo E. Hannay; Guide to Advanced Empirical Software Engineering Shull, Forrest, et al, 2008
Systematic Literature Reviews	A particular genre of reviews of the literature that you might want to use for your project:	First is a textbook chapter about performing SLRs. Then come studies of SLRs in SE that provide links to other SLRs and literature on how to conduct SLRs.
	Read Chapter 4:	Experimentation in Software Engineering, Wohlin, Claes, et al, 2012
	"Systematic literature reviews in software engineering – A systematic literature review"; Barbara Kitchenham, O. Pearl Brereton, David Budgen, Mark Turner, John Bailey, Stephen Linkman; Information and Software Technology; Volume 51, Issue 1, January 2009, Pages 7-15	"Systematic literature reviews in software engineering – A tertiary study"; Barbara Kitchenham, Rialette Pretorius, David Budgen, O. Pearl Brereton, Mark Turner, Mahmood Niazi, Stephen Linkman; Information and Software Technology; Volume 52, Issue 8, August 2010, Pages 792-805
	"Systematic literature reviews in software engineering: Preliminary results from interviews with researchers"; Muhammad Ali Babar, He Zhang; 3rd International Symposium on Empirical Software Engineering and Measurement, 2009
Inspections, which have been thoroughly empirically studied, an example of a SE process whose real-life enactment is experiment sized	Advances in Computers, Volume 42, 1996, Pages 39-76. Here is a prepublication draft.	Adam Porter's Publication List: Prior to 2002, there are lots of empirical studies of inspections.
	"What do we know about defect detection methods? [software testing]", P. Runeson, C. Andersson, T. Thelin, A. Andrews, T. Berling, IEEE Software, 23:3	"Survey of Software Inspection Research", Sami Kollanus and Jussi Koskinen, The Open Software Engineering Journal, 2009, 3, 15-34
	"The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data", Chris F. Kemerer and Mark C. Paulk, IEEE Transactions on Software Engineering, SE-35:4, 2009	"An Empirical Study of the Impact of Modern Code Review Practices on Software Quality, Shane McIntosh, Yasutaka Kamei, Bram Adams & Ahmed E. Hassan, Empirical Software Engineering, 21:5, pp. 2146-2189, 2016
	"Code Reviews, Software Inspections, and Code Walkthroughs: Systematic Mapping Study of Research Topics", Ilenia Fronza, Arto Hellas, Petri Ihantola, and Tommi Mikkonen, International Conference on Software Quality, pp. 121--133, 2020 The link brings you to a proceedings. Search for the first author's name, and then click on the link that is the title.
Caveats	"How Do I Know Whether to Trust a Research Result?", Shepperd, Martin, IEEE SW 32:1, pp. 106-109, 2015
Effectiveness of EPMcreate Variations as a Technique for Creative Requirements Idea Generation, Experiments, Surveys	"Applying a Pragmatics-Based Creativity-Fostering Technique to Requirements Elicitation", Mich, L., Anesi, C., and Berry, D.M., Requirements Engineering Journal, 10:4, pp. 262--275, 2005	"The Effectiveness of an Optimized EPMcreate as a Creativity Enhancement Technique for Website Requirements Elicitation", Sakhnini, V., Mich, L., and Berry, D.M., Requirements Engineering Journal, 17:3, pp. 171--186, 2012
	"Requirements Elicitation (ReqElic) in My Company: Preliminary Results of a Questionnaire", Mich, L., Sakhnini, V., Berry, D.M., Requirements Engineering Magazine, 2015:3, 2015	"Group Versus Individual Use of Power-Only EPMcreate as a Creativity Enhancement Technique for Requirements Elicitation", Victoria Sakhnini, Luisa Mich, & Daniel M. Berry, Empirical Software Engineering, 22:4, pp. 2001--2049, 2017
	"Downsizing EPMcreate to Lighter Creativity Techniques for Requirements Elicitation", Mich, L., Sakhnini, V., and Berry, D.M., Workshop on Creativity in Requirements Engineering (CreaRE) at REFSQ'2017, paper 2, 2017	"Creativity Techniques for Requirements Elicitation: Comparing Four-Step EPMcreate-Based Processes", Herrmann, A., Mich, L., and Berry, D.M., Workshop on Empirical Requirements Engineering (EmpiRE), pp. 1--7, 2018
Existence of Ambiguities in Requirements Specifications, Case Studies	"Ambiguity in Natural Language Software Requirements: A Case Study", Fabian de Bruijn and Hans L. Dekkers, Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ), pp. 223--247, 2010	"Requirement Ambiguity Not as Important as Expected --- Results of an Empirical Evaluation", Philippo, E.J., Heijstek, W., Kruiswijk, B., Chaudron, M.R.V., and Berry, D.M., Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ), pp. 65--79, 2013
	"The Prevalence and Severity of Persistent Ambiguity in Software Requirements Specifications: Is a Special Effort Needed to Find Them?", Ribeiro, C. and Berry, D.M., Science of Computer Programming, online first, pp. 1--35, 2020
	Main Slides: Results: The Prevalence and Severity of Persistent Ambiguity in Software Requirements Specifications	Supplemental Slides: Background: The Prevalence and Severity of Persistent Ambiguity in Software Requirements Specifications
	Supplemental Slides: Ambiguity in Natural Language Requirements Specifications	Supplemental Slides: Some "only" Exercises for Class
	Google search for "I eat only"	Google search for "I only eat"
	Google search for "All eat"	Google search for "Both eat"
	Google search for "Each eats"
Managing Inconsistencies, Interviews	"The Inconsistency Between Theory and Practice in Managing Inconsistency in Requirements Engineering", Hadar, I. Zamansky, A., and Berry, D.M., Empirical Software Engineering, 24:6, pp. 3972--4005, 2019	Daniel Kahneman, Nobel Prize Lecture, 2002
	Of 2 Minds: How Fast and Slow Thinking Shape Perception and Choice [Excerpt], Daniel Kahneman, 2012	Daniel Kahneman Explains The Machinery of Thought, fs blog This blog contains a link to his book Thinking, Fast and Slow
	Main Slides: Inconsistency Between Theory and Practice in Managing Inconsistency
Feasibility and Effectiveness af Requirements Reuse, Case Study with Data Mining	"Reuse of Requirements Reduced Time to Market at One Industrial Shop: A Case Study", Goldin, L. and Berry, D.M., Requirements Engineering Journal, 20:1, pp. 23--44, 2015
Requirements Engineering in the Software Development Lifecycle, Case Studies	Main Slides: Developers Want Requirements	"Developers Want Requirements, but Their Project Manager Doesn't; and a Possibly Transcendent Hawthorne Effect", Isaacs, D. and Berry, D.M., International Workshop on Empirical Requirements Engineering, pp. 37--44, 2011
	Main Slides: Requirements Determination is Unstoppable	"Requirements Determination is Unstoppable: An Experience Report", Berry, D.M., Czarnecki, K., Antkiewicz, M., and AbdElRazik, M., International Requirements Engineering Conference (RE), pp. 311--316, 2010 "The Problem of the Lack of Benefit of a Document to its Producer (PotLoBoaDtiP)", Berry, D.M., Czarnecki, K., Antkiewicz, M., and AbdElRazik, M., IEEE International Conference on Software --- Science, Technology & Engineering (SwSTE), pp. 37--42, 2016
	Main Slides: Quantifying the Impact of Requirements Definition and Management Process Maturity	"Quantifying the Impact of Requirements Definition and Management Process Maturity on Project Outcome in Business Application Development", Ellis, K. and Berry, D.M., Requirements Engineering Journal, 18:3, pp. 223--249, 2013
	Main Slides: Requirements Engineering for Two Consecutive Versions of a Product	"Experiences of Requirements Engineering for Two Consecutive Versions of a Product at VLSC", So, J., Berry, D.M., International Requirements Engineering Conference (RE), pp. 209-218, 2006
	Main Slides: Requirements Engineering for a Synagogue Kitchen	"Lessons Learned from and for Requirements Engineering and Building Construction: A Case Study of Requirements Engineering for a Synagogue Kitchen with Use Cases and Scenarios", Mauger, C. Berry, D.M., IEEE International Conference on Software --- Science, Technology & Engineering (SwSTE), pp. 67--76, 2014
Identification of Hedonic Qualities in Requirements Specifications, Experiments and Interviews	"Improving the Identification of Hedonic Quality in User Requirements --- A Controlled Experiment", Maier, A. and Berry, D.M., IEEE International Requirements Engineering Conference (RE), pp. 205--214, 2017	"Improving the Identification of Hedonic Quality in User Requirements: a Second Controlled Experiment", Maier, A. and Berry, D.M., Requirements Engineering Journal, 23, pp. 401--24, 2018

This page is at https://cs.uwaterloo.ca/~dberry/ATES/index.shtml