Master’s Thesis Presentation • Software Engineering • Evaluating and Comparing Generative-based Chatbots Based on Process Requirements

Please note: This master’s thesis presentation will take place in DC 3317.

Luis Fernando Lins dos Santos, Master’s candidate
David R. Cheriton School of Computer Science

Supervisor: Professor Paulo Alencar

Business processes refer to the sequences of tasks and information flows needed to achieve a specific goal. Such processes are used in multiple sectors, such as healthcare, manufacturing, banking, among others. They can be represented using diverse notations such as Event-driven Process Chain (EPC) and Yet Another Workflow Language (YAWL). A popular standard notation for modelling business processes is the Business Process Model and Notation (BPMN), due to its breadth of features and intuitiveness. Despite organizations increasingly turning to automated processes to enhance efficiency and precision, business processes might still contain human-dependent tasks, which present several challenges, such as complex task orders, task dependencies, and the need for contextual adaptability. These challenges often make it difficult for process participants to understand process execution, and it is common for them to get lost in the process they are trying to execute.

Since the advent of process modelling languages, there have been multiple strategies to help users execute business processes. Most recently, chatbots, programs that allow users to interact with a machine using natural language, have been increasingly used for process execution support. A recent category of chatbots worth mentioning is generative-based chatbots, powered by Large Language Models (LLMs), which are trained on billions of parameters and support conversational intelligence.

However, several challenges remain unaddressed: (i) how to incorporate process knowledge into generative-based chatbots; (ii) how to evaluate if a generative-based chatbot is meeting the requirements of BPMN constructs for process execution support; (iii) how different generative-based chatbot models compare in terms of meeting the requirements of BPMN constructs. To address these challenges, this thesis presents an exploratory approach to evaluate and compare generative-based chatbots in terms of meeting the requirements of BPMN constructs. This research comprises three distinct phases. The first phase presents a literature review that examines how chatbots are used in conjunction with business processes. In the second phase, the focus shifts to retrieval-based chatbots and their potential for process execution support when maintaining direct communication with a process engine system, in an attempt to achieve a gold standard of process-aware chatbots. Finally, the third phase delves into the realm of using generative-based models as process-aware chatbots to enhance business process execution support, including an evaluation of how two distinct generative models (GPT and PaLM) perform when helping in the execution of two business processes (Trip Planning and Wedding Planning).

This thesis makes several contributions. First, it offers a comprehensive exploration of the challenges and gaps in the existing literature regarding the development of process-aware chatbots. Second, it provides an exploratory study about the use of generative-based chatbots for process execution support that meets the requirements of BPMN constructs. Finally, through comparative qualitative and quantitative evaluations, it sheds light on the performance of prominent generative models, GPT and PaLM, in the context of process execution support, contributing valuable insights for future research and development in this evolving field.